Attendance

Attendees:

• Rolf Krahl
• Chris Prosser
• Louise Davies
• Silvia Sottini
• Marjolaine Bodin
• Kevin Phipps
• Malik Almohammad
• Patrick Austin
• Santhosh Anandarama
• Rodrigo Cabezas Quirós
• Alan Kyffin
• Andy Götz

Agenda

Site Updates

HZB

Data Publication landing pages public: https://codebase.helmholtz.cloud/hzb/research_data_management/python-hzb-rdm-landingpage

D3A, activity from LEAPS. Not yet integrated into landing pages. Separate package for creating landing pages, first step. D3A work is to add metalink via content negotiation.

KP: Something with the IDS as well? It doesn't, not able to restart downloads from a certain byte? Looking for changes to that?

RK: Not directly related to D3A. Bug reported by ESRF. not worked on (yet). Is something else D3A to do with IDS - automatic staging from disk. IDS reports temporary error & triggers staging, so just need to trigger same request. 2 issues, behaviour not standardised - dedicated HTTP status code. IDS currently reports 503.

SESAME

No major update. Working with other groups to create DOIs based on abstracts. Huge amount of data from beamlines, issues with storage.

ESRF

SUbmitted renew core trust seal certification. Slowness issues with DataPortal - unknown cause. Unknown if ICAT related. Issue with updating investigation - emailed mailing list about it. Motivation, near embargo email people, have requests on changing release date, wanted to add automated mechanism for this but encountered issues.

KP: might have an update later re: this issue

ISIS

Migrating DOI service to use DataCite directly, and to link Instrument DOIs to data. Other small internal tasks.

DLS

Continuing DOI minting project. Legal clearance WIP, agreeing on data policy changes. Already worked out the policy changes, going through steps of approval. Intent to roll out to staff in autumn and users next spring.

Issues with tape library, stability issues. New hardware put in recently and hope to improve stability. Currently fine, hope it stays fine.

KP: other day, DLS went over 90 PB

CP: yes, a couple weeks ago

ALBA

Testing new ingestion system, PACER. In production on some beamlines (MX will take more time?). Check out the source code, share link: https://github.com/ALBA-Synchrotron/icat-pacer, https://icat-pacer.cells.es/pacer/

Component Updates

icat.server

Marjolaine's issue

PA: not sure if I have further updates than what I emailed on mailing list. Rather than fixing bugs piecemeal, should instead consider whether SOAP and REST should share underlying logic.

AK: agree, move towards one code path works for both interfaces. Issue is breaking changes if something relies on the different behaviour.

PA: SOAP one "better"? Yes, would be a breaking change.

RK: not using REST interface, was surprise that update and create are the same call. Interesting design!

PA: anyone else using REST?

MB: ingest, using SOAP. ICAT+, use REST.

LD: Not necessarily removing SOAP, just make the behaviour match SOAP. But if you were willing to migrate, any breaking change would be less significant than that.

KP: way forward, evaluate how big a job it is to make them share the same logic, then work on it if it's not too bad. Meanwhile, use SOAP if you're encountering these issues. And people will need to be aware of breaking changes when it's released.

PA: if this would be ICAT 8? when is ICAT 7?

RK: still need to do a little bit of work to submit PRs for ICAT 7. Won't be able to do before summer, maybe September? Hesitant to promise though!

PA: we can do the work, and it shouldn't affect each other

RK: no need to have 2 separate major versions. Can see if it makes sense to wait for the other feature, otherwise 2 majors.

ids.server, python-icat

No

AOB

Umbrella

Connected to eduTEAMS (built on eduGAIN). Integrating can mean accepting more people. Replace anonymous access with asking to auth.

KP: plan is to do similar, can see DOIs anon. Access to data is via ORCiD

AG: umbrella accepts ORCiD

RK: how many interested in adopting umbrella? HZB WIP on identity management. If push towards umbrella, could push internally. Add umbrella ID to it.

AG: renewed interest, adopted by larger org.

AG: advantage cross border

Performance

Tools for analysing performance? Alex identifying bottlenecks. 20 sec wait, then suddenly goes away.

KP: Payara, configure to log queries over a certain time. Have used to debug DLS queries, initially to log 60s, then 30s, now 10s. Occasional over 10s. That's identifying DB issues, within ICAT unknown.

AG: made it too popular!

KP: yes, if users very reliant, need service to be very solid. Let me know if you need pointers to which settings

RK: internal knowledge base for Payara settings. Large downloads fail, payara timeout. (See this commit for the change in our Payara settings that we applied in response.)

LD: Also got to consider role of reverse proxy, if using, as that will have its own timeout.

AG: slow download, block anyone else?

RK: no, IDS spawns new thread per download. Ideally you don't want infinite threads, but in practice it doesn't matter.

AK: similar problem with site firewall, used for internal connections as well, can cut off connections after ~30mins.

RK: Had a 28GB dataset, that was the first one of that size that's been tried. Noticed I could not do it from my office. But could to the same server rack. Whether the 28GB can transfer in the given timeframe.

PA: open telemetry. Other projects exploring. Long term should explore?

AK: Payara integrates some open telemetry wrappers for you (logging and metrics), but not traces. Going to be added in future version

Next meeting

Having July meeting? Yes, in the calendar for 30th July. ESRF most likely won't be here - peak holiday time.