Attendance

Attendees:

• Rolf Krahl
• Louise Davies
• Kevin Phipps
• Malik Almohammad
• Patrick Austin
• Alex de Maria
• Santhosh Anandarama
• Alan Kyffin
• Viktor Bozhinov

Agenda

Site Updates

SESAME

No issues. User from Germany wants to download huge amount of data, using ICAT+ can't download more than 1-2GB. Investigated and will share something we found with Alex. Update nginx to increase timeout values. Increase IDS nginx proxy timeout.

LD: Have had to do similar in the past for the DLS IDS. Increase timeouts as an hour might not be enough if large.

KP: there are timeouts in Payara itself but also Apache or nginx. Sounds like you found and increased both.

HZB

Servers currently running on 2 bare metal machines, EOL already & OS old. Replacing - looking into solutions.

Once that's fixed want to deploy first instance of SEPIA (sample database) which is closely tied to ICAT.

ESRF

Working on processing plan - how to declare samples/parameters before experiment etc. Has anyone already looked into this? We have an archive of all the machine configuration for all historical data. Useful to connecting into ICAT? e.g. beam lost, can warn users that their data/sample was affected

RK: For registering samples before the experiment you can do that if you have the investigation from your proposal system, you can create the sample already (without datasets). With proposed schema changes you would be able to register samples before investigation even, but currently schema requires a sample is 1:1 to an Investigation.

ISIS

Nothing major to report, converting accounts from simple to DB auth. Removing unused queues e.g. on backups. Evaluating public cloud - perhaps moving servers to public cloud depending on this.

KP: you said you're investigation moving off the STFC cloud and onto public cload? Due to the recent outages?

SA: Yes.

LD: Proposals system is currently hosted on the [STFC] cloud. Maybe if everything else gets migrated then ICAT (which is not on STFC cloud) will too but depends on a cost:benefit analysis.

KP: We have had problems with our in house cloud recently.

DLS

KP: Kirsty is leaving DLS, another month left. Putting transition plans in place whilst waiting for recruitment. Working on whole visit restore for DLS - deployed and initial user testing. Also implemented endpoint to supply list of file paths.

LD: Also got DB migration next week but it's not really relevant to ICAT specifically.

Component Updates

icat.server

KP: Something from Alex on the size of location fields? Rolf suggested we discuss?

RK: Few schema changes in general are queued. Some are compatible with existing installations and could go into a minor release. I would add Alex's request onto that, as a 4th change. We have a student working on these changes, we would be able to provide pull requests for all 4 schema changes that could go into v6.2 but it would be helpful to have a snapshot release for icat.server and icat.client ASAP. Chicken and egg problem as you need a snapshot of icat.server to generate icat.client to write tests for the SOAP interface on icat.server. Produce PRs in icat.server without these tests. Could deliver these tests if we had an icat.client snapshot. First question is therefore what is the rlease plan? quick release of v6.2 or not?

LD: Makes sense easier upgrade path, facilities keen on content.

RK: Also PRs from STFC?

AK: No content, just code cleanup and upgrade to junit5 (needed that as reuqired by Quarkus testing component). Another thing, now the Jakartae spec is finalised expect a Payara7 release sooner rather than later. Tested icat-ansible with the latest Payara7 alpha. So expect this is an easy change and not like Payara6.

RK: Apart from schema changes for v6.2, 2 or not compatible. These would need to be icat 7. Student is available until mid - August so could deliver then. Then it would make sense to wait for Payara7.

AK: Don't think it will require any changes to ICAT to be compatible.

PK: We won't have to create icat7 for Payara7.

AK: No it should just work. Should we mention the issues? v6.2 are:

• entityType subject
• Start date end date instrument.
• Internal id to datapublication.
• location field length increase

v7 are:

• SampleType changes (changes uniqueness constraint)
• Adding pid property to SampleType.

KP: what's the status of those? have they got code which has been reviewed?

RK: coding is not complex. One is submitted and positively reviewed. The other v6.2 changes are not very difficult. For the extension of the location attribute. It would not be required but useful to distribute upgrade scripts. I would need assistance to test an Oracle script. For incompatible (v7?) coding is not compicated but needs an upgrade strategy. Relation between Sample and Investigation depends on the content. Would need to populate the pid as it is the new uniqueness constraint and it needs to have unique values. Could provide Python script. In the old can populate the pid attribute, and test it for uniqueness, then upgrade is not problematic.

KP: Sounds good. If it's in a Python script it will work with Oracle and MariaDB?

RK: suggest that formally we should provide an upgrade script. But the python makes it likely they will run without issue.

AdM: You are synchronising the Sample information from wherever user submits proposals?

RK: No we are creating a saple database which has the sample information and populates the table in ICAT.

AdM: in our case the user portal cannot handle one sample linked to multiple investigations. We cannot use this feature. Or can your system handle one sample with multiple investigations?

RK: We don't have any sample information from the user protal in the first place. What happens if a sample happens in more than one investigation. You register twice because that's what the schema makes you do.

AdM: User portal is the source of information. It records 2 different samples even if they are the same.

RK: If the soruce tells you there are two then not much you can do. Maybe you can de-duplicate after the fact, delete one isntance and create relations to the one you wish to keep.

LD: The new schema requires a new PID. So a system with multiple samples may not generate unique pids. But the ids doesn't have to be...

RK: If a sample is a sub-entity of the Investigation then pid will not be meaningful. Then you do not need to care if it is unique.

KP: how to proceed?

RK: Phillip will produce the PRs

AK: Will review.

RK: as soon as acceptable merge and snapshot. Would ask if someone can help with the Oracle script update. No way of testing if it works.

KP: have done this before so can test that.

LD: version 7 once the PRs are made for that everyone who's concerned has to play with the snapshot. As that's a larger schema change.

KP: may be running on a snapshot version.

AK: yeah, probably need a 6.1.1 release.

ids.server & related

No activity

python-icat

Will likely get minor release when icat.server 6.2 comes out

icat.lucene

Sample relation change will likely have knock on effects to icat.lucene indexing.

RK: old PR for the sample changes stuff, needs updating but after can ping Patrick to see if any lucene changes required

AOB

F2F

Potentially Oct this year if STFC hosts?

RK: international data week in Oct so need to avoid. 3rd october & next 2 weeks from then.

KP: end of October fine?

RK: as long as it doesn't clash it's fine.

LD: this is assuming STFC is hosting - unless anyone else volunteers! we haven't hosted in a while

KP: we'll send something out when we have something firmer but we are working on it