ICAT Project


Meeting 93 – 28 January 2016


  • Actions from meeting 92 (Steve)
  • Component development
    • ICAT (Steve)
    • ICAT Manager
    • Python-icat (Rolf)
    • IDS (Steve)
      • FUSE (Alex)
      • Globus transfer (Frazer)
      • SmartClient (Steve)
      • WebDAV (Kevin)
    • TopCAT (Jody) – with demo
    • IJP (Brian R)
    • ICAT Dashboard (Alex)
    • Face to Face meeting for 2016 (Steve)
  • Issues from ICAT production deployments (Any)
  • AOB (Any)

Present: Frazer, Jody, Jianguo, Steve, Brian R, Kevin, Tom,  Rolf, Alex dM and Shelley

Apologizes: Alex PD.

Action 92.0 Provide feedback on the ICAT enhancements paper.  Some have been provided this can be marked as done.

Action 92.1Provide feedback on the idea of adding a DOI attribute to the DatafileCollection. There have been no objections so it will happen in ICAT 4.7

Action 92.3 Rolf has produced a python-icat survey but has only three responses so far.

Action 91.0 Peter has added his Puppet modules to contrib and included a link to Travis

Action 85.3 Migrate ICAT Manager to the ICAT project GitHub. The action was transferred from Andy to Alex de Maria.

  • Steve reported on progress with ICAT 4.6 which is almost ready to go out with the following new features:
    • “getProperties()” restful call to find out what can be revealed about a server without a sessionId. This includes maxEntities and information about the available authenticators. Requires updated authenticators
    • New calls added to the restful interface for lucene queries.
    • Documentation for the restful interface is now generated by MireDot.
    • Changes to properties files to work with container independent setup script.
    • Change from log4j to slf4j with logback
  • Steve also reported that ICAT 4.7 is in a good state with the following features:
    • A restful select can mention multiple quantities to be returned
    • A major re-write of the authorization code should make it more efficient in most cases as all the work is carried out by the database unless the INCLUDE mechanism is used.
    • COUNT queries will not fail because they are are counting more than maxEntities values
    • A couple of bugs in eclipse link result in incorrect information being returned. One of these bugs was trapped previously but with the generality of the queries now supported it has become impractical to trap.
    • A restful delete(Many) call has been added. Authz is applied to all entities involved
    • A restful write call to do both create(Many) and update(Many) has been written. Notifications are sent for all objects created and/or updated. Authz is also applied to all new or updated entries.
  • This was followed by a discussion about Rolf’s pull request to add a SOAP call to be able to find out about the authenticators (which is currently only available in the SOAP call). Steve wants to support the SOAP web service while it is needed – which will of course include bug fixes – but not to enhance it. He considers that the extra efficiency of the RESTful interface means that new application code should avoid the SOAP API. Shelley said she was happy to have only RESTful and Kevin emphasised the need to maintain the SOAP interface while it is needed by existing code.
  • Alex dM is starting to familiarise himself with the ICAT manager. He plans to visit RAL soon.
  • A small change is being made to the JMS messages that the Dashboard can pick up from the IDS. Rolf has found a possible bug and is investigating more possible race conditions.
  • Jody then gave a demo of TopCAT. The new interface was generally appreciated. Alex dM suggests that Jody make a screencast of his demo.  Shelley asked whether it would be possible for users to annotate metadata to help with re-running auto-reduction of their data.  This is not possible at present, but may be considered in future (along with other features such as the ability to mint DOIs).  Steve asked whether there are known performance issues with large volumes of data. There is no problem with ISIS but the Diamond data has not been indexed yet.
  • Steve reported that  from Alex PD had reported that a sprint review would be held mid February. It was suggested to give a demo at the next ICAT meeting.
  • Concerning the F2F meeting which we hope to co-locate with NOBUGS, Steve has not received a reply yet from the organisers.
  • Rolf reported they have a new person, “Michael Hellmig”.
  • Tom reported that there are now 1.1 billion files in the Diamond ICAT catalogue. They are not yet indexed – with an estimated four months to build the lucene index. (Note that at the time of writing these minutes the estimate is now four and a half days.)
  • Shelley said they have a second facility and are considering using the same ICAT instance for both. Kevin pointed out that for CLF separate ICAT instances are used for each laser, as their non-overlapping downtimes would otherwise make it difficult to schedule upgrades.
  • Alex dM asked whether TopCAT would be able to cope with different versions of ICAT and IDS. The general view was that this should be possible except for very old



Item Description Assigned
93.0 Make screencast of TopCAT demo Jody
93.1 Demo of dashboard Alex PD
92.2 Provide information on LSF’s use and potential future use of ICAT components. Jianguo
85.3 Migrate ICAT Manager to the ICAT project GitHub Alex dM
74.0 Look into JDBC connector resilience with Wildfly Steve