Attendance

Attendees:

• Louise Davies
• Silvia Sottini
• Kevin Phipps
• Malik Almohammad
• Patrick Austin
• Alex de Maria
• Andy Götz

Agenda

Site Updates

SESAME

Not much to update. Cybersecurity attack - recovering quickly. No ICAT updates, just stopped everything.

AG: was ICAT a weak link?

MA: ICAT wasn't affected, services related to ICAT were clear. Clusters were affected. Odd, lots of other SESAME public services were attacked, but ICAT & related services (e.g. DOIs) not.

AG: getting ICAT back up?

MA: yes, good job we have backups. doing well all things considered.

AG: good job on getting things back up quickly!

ESRF

ESRF emailed Kevin re: ICAT security, but he was on leave. Is that resolved?

AG: security audit, internal hackers to check for vulnerabilities. Discovered way to become admin from user session. Contacted Patrick after, identified issue - issue was fixed last year, but we weren't running latest versions of ICAT (5.0.3 or 6) with the fix. Tried to run on production, error re: indexing DatasetParameter table. Fine on test instance, which points to same DB - puzzling?

AdM: 2 machines running icat in production. One machine updated to 5.0.3 fine, got error after trying to upgrade second machine.

KP: don't know why that's happening

AG: need to do deeper analysis when Marjolaine is back from leave next week

KP: not had follow up re: indexing error?

AG: not yet

AdM: brute force to get active admin session ID. Our ICAT is not exposed outside, and admin user is read-only. So limited scope for exploitation.

AG: lots of vulnerabilities coming up recently (PHP, brotli)

LD: re: ICAT upgrade, when we upgrade, we don't do that? We don't upgrade one and then the second, would we not upgrade both at the same time? Do one, wait, do the second. We take both down then both back up.

KP: Yes, but not exactly the same time...

LD: Is one kept running one the DB, or are both turned off? If we... turn all the components off, upgrade, bring back then that's a difference.

KP: ICAT has to be running to upgrade it. If you have two pointing the the database, then at least one will point. But it's a short period.

AdM: Tried the same day but it failed?

AG: No. She was confident she could do the other the next day. Scheduled maintainence for the second.

AdM: Sometimes the schema changes.

LD: These were patch changes so should never have schema changes

PA: was only a patch, but it did change the persistence context

AdM: Sometimes, you configure then you install. When you have two, you have to have them running. First one then the other. You cannot shut them all down. Is that correct?

KP: Yes.

AG: Should be more info in the log file?

KP: Depends on the log level. Some is in ICAT logs, some goes to Payara logs. Depends on the error. Look in both places.

LD: If it's startup, sounds more like a Payara error.

KP: Yes, table warnings go there.

AG: Checked timings and it was Wednesday and Friday.

KP: let us know if there are still issues & share the logs

 

AG: OAI-PMH, panosc node. Data respository to be seen by EOSC node. Needs to be in provide server. Want to have a service classified as data, need to have it indexed by openaire. Need to register OAI-PMH endpoint with openaire. Failed with OAI-PMH, didn't have openaire schema/profile. Our endpoint need to provide profile. Should be trivial, conforms to datacite schema, missing profile.

LD: need to create mapping file?

AdM: yes, maybe just need to change mapping to something openaire expects.

AG: look into it, maybe need Rolf

LD: Yes installed for ISIS. At some point there was a website (maybe a PaNOSC thing) that was trying to build. ISIS failed as it wasn't performant enough. A lot of time ISIS breaks things by having the most open data. Potentially it's just a question of scale. Rolf made it flexible as every facility uses ICAT differently. I just know it exists, not how to use it. Either Santosh or Will will have set it up. Likely forgotten.

AG: Want to get our repositories officially recognised.

AdM: Related to that. Harvesters. The problem is that we have beamlines are dependent on ICAT, if ICAT is slow or crashes they cannot acquire data. How do we protect against that? We open to provide public access but it has an impact. Playing with idea of duplicated infrastructure for internal/external use.

LD: Similar concerns for DLS tape archive. Separating open data downloads and attaching them to ORCID's so you know who's requesting. Also it goes via a "queue" which can de-prioritise recalls relative to internal recalls. But this is data not metadata. For certain things, like OAI-PMH, you could throttle or block but I wouldn't know.

KP: have thought about having separate ICAT for open data users, but still connected to same DB... Unless you have separate DB...

SS: DLS things are different, managed by STFC aka separate from beamline operations. LIMS separate to ICAT.

KP: bringing ispyb into ICAT ties both acquisition and retrieval together

PA: duplicate info across multiple DBs. Alan keen to put search index in elastic - could expose that to read queries and it doesn't impact DB operations. Still duplicating, but separate technologies to improve.

AG: accessing a copy?

PA: yes, metadata in different format. idea behind PaNFinder, scaping & re-indexing.

LD: Before the AI use case, they want it decentralised. Needed for LLMs which was the motivation.

PA: yes, but still idea of a different use case

AG: what we do elastic search is similar right?

AdM: yes, but users using AI. Using API directly. Not bad, but has an impact. DLS - ispyb has same issues?

AG: random queries, searching for terms we don't even have experiments on. may need to consider turning off anonymous access.

PA: had these issues for a while.

MB: session table growing in size, not cleaning automatically.

PA: should be cleared up

KP: think it was fixed recently, maybe not fixed in 5 but definitely 6.

ISIS

Lots of DOIs still on DataCite schema 2 which is very old. Can't do instrument PIDs until then. Also going to have a chat with ISIS about architecture. Some discussion in the past about archive architecture for Extreme Photonics Application Centre (EPAC). Will they need tape? Do they want to sunset their windows servers? We don't know. Will follow through on it. ~ 2 weeks time so might have a report by next collaboration meeting.

DLS

Still working on DOI minting & open data. Currently internal DLS user testing before moving to production - collecting feedback at the moment. Few tweaks, but hopefully end of next month go on production. Idea is to be ready for April as that's when needed policies come into force. NeXus files issue - federal IDs recorded in there (internal user IDs). Have to change beamlines to stop doing that.

AG: urgently need DOI, processed data. Cater these requests?

SS: piece of work to enable users to upload processed files. Separate storage location than tape. Not done yet, but would like to do. No requests for minting DOIs, but not rolled it out yet. Not had any questions from users yet.

AG: lots of users don't care about data DOIs. More journals now being stricter on data citations.

KP: long term road map to allow upload. Only have permission to put DOIs on raw data, but vision is to allow it in the future.

SS: plan for DLS 2 to open processed data. Need legal team to look at it first!

 

SS: AI queries. Open data or users doing it for their own data?

AdM: Both. Looks like scanning for everything.

SS: users with big number of visits? or scanning all data they can access

AdM: Normally ICAT prevents large downloads/requests - entity limits. So they scan. Users have been coming to ESRF for 10 years. Have a lot of data.

AG: Support staff also have access, and want to train AI by recalling all the data for a given technique.

SS: beamline scientists have access to only their own beamline.

AG: internal staff have access - from data policy

AdM: beamline staff can only see their own, but also some staff admin level access.

LD: same as us.

SS: yes, but no one doing it.

LD: Just because you can, doesn't mean you should. Once submitted a lot of requests, and was asked to stop. Can prevent internal staff from causing load issues. Question of behaviour. Users have access, data policy allows it, but doesn't mean you should. Sounds like people are being naive about this. They're being selfish.

SS: Sure software people in Diamond have access to data on Diamond systems. Do analysis people have access to ICAT? Are they allowed to download all the data? I don't think so.

PA/LD: depends on your configuration

SS: they can't see the data when it's restored

PA: doesn't mean they can't see it in ICAT/access it via non-DLS download

AdM: naive, but once we analysed, we can see that many users and of lots of types are doing it/using AI/creating load. Helpdesk, issue report from logbook, AI generated report with reproduction & fix for the issue. And it was correct!

Component Updates

None

AOB

ispyb meeting

Next month @ ESRF. Discussion on ICAT ispyb solution. Evangelising the use of ICAT

AdM: also talk about other issues, ontologies, etc. Will distribute agenda soon

AG: ontologies for samples

SS/LD: yes, sample ontology is of a lot of interest.

SS: talking to Elizabeth Newbold - campus wide sample vocabulary

d3A

Direct DOI data download. aria2 CLI download tool. Can download multiple files from IDS via aria2 from DOIs. Working in test, going into production soon.