You might remember us starting this “instrument control software migration” project a while back, where we were slowly experimenting with moving bits and bobs into EPICS. Reasons for this migration include lack of support, retirement of the main (and only?) developer Gerry Swislow, and limitations in SPEC itself that prevented us from running more complex experiments and integrating a wider range of devices. EPICS, eventually with the optional BlueSky experiment orchestration on top, seems to solve many of these issues.
After a power outage in the lab 1.5 weeks ago, we decided to not spend the considerable time needed to get SPEC back up and running again (which would’ve required recalibrating some motors). Instead we wanted to spend that time to finally finish this control software migration to EPICS.
Progress on this has been going in steps over the last year or so, with intermediate resting periods as we continued measuring mostly using the old methods and scripts. Every now and then, an event would spur us into improving another bit, until the day when the final improvement would be done that makes SPEC redundant. That day is now.
Final changes needed:
Of course, there are always a few things that aren’t quite ready yet. So this week, we’ve been working hard on:
1. A new logbook
You might be aware that we are keen on organizing our measurements with the use of an electronic logbook – viz. an Excel sheet that defines what we want to measure and how we want to measure it. While the old logbook has proven useful and has over 50k lines in it, the current move meant that we could take the opportunity to refactor the logbook to simplify and generalize it and make it more flexible.
The resulting logbook contains much less information to fill out, removes a few places where information was entered that was already available somewhere else (in particular sample owner and sample name, which are available in the project/proposal sheets filled out by each of our collaborator), and allows for each line to fill in a measurement template and some flexible keyword-value combinations for that template.
The measurement template is a python script with the exact commands that runs the experiment. That means we have a standard template now with the way we normally run experiments, i.e. measure the sample for n repetitions with beam and transmission measurements before shifting configurations. For in-situ or operando experiments, we can now have bespoke templates that address the additional hardware and readouts as appropriate, potentially including python loops as well to make our life brief and easy. You can even use these templates to measure in completely different geometries such as grazing incidence with no change to the logbook.
The generalizations also mean that the same logbook could be used for other instruments too with little to no modification.
2. A new scripting engine
That new logbook requires a new piece of Python software that chains the planned measurements together. That is not a large piece of code, but needs to be robust in the way it handles the excel sheet as excel allows far too much flexibility, sometimes even for machine operators. That code is available here (and comes with the example logbook), with many thanks to Anja for taking the initial proof-of-concept and modifying it into actually useful code.
3. Connecting up the Arduino Pro-based multi-function device (MFD)
To provide various I/O for, for example, switching valves, driving external equipment and reading out (temperature) sensors, I built a network-connected Arduino Pro Portenta Machine Control-based box a while back. These Arduinos combine a lot of what you need in an industrial setting (24V pins with some protection, etc.), and the enclosure I made mainly just breaks out these pins in appropriate connectors (For the compressed air valve, however, I did use a spare solid-state relay I happened to have lying around to reduce the current through the board).
With a bit of Arduino code and a custom caproto-based EPICS IOC, all of these pins are now addressable over the EPICS channel-access communications bus.
4. Automated collimation optimization
There will be a separate post about this particular bit of coolness, but Anja adapted a piece of software from Martin Radtke at BAMLine that uses a Bayesian methods for motor optimization. The adaptation makes it possible to automatically optimize our three slit sets to attain a given beam shape. To do this, an appropriate cost function must be specified, and Anja had to spend some time to tune this to get the desired performance. The current function multiplies two orthogonal top-hat functions with error function sides (to create a smooth transition and more closely approximate the actual collimated beam shape), to get a rectangular-ish target beam profile.
Now that it works, you can specify the beam dimensions you desire and let it optimize. For GISAXS, for example, you’d want a flat but wide beam, for transmission-SAXS you’d want a reasonable beam, and for capillaries you’d want to restrict the dimensions a bit to avoid hitting the capillary walls. Optimization takes up to a few hours for the longest configurations, but is automated. The same optimization can be used to position the beamstop in the most effective location.
Interestingly, we also figured out just in time that we’d need to run these optimisations keeping the beam close to the same place for all sample-to-detector distances. That also meant measuring and taking the detector “wobble” into account, i.e. the problem caused by the rails not being perfectly straight along the beam tube.
5. EPICS IOC management
Every device or set of devices that is connected to the machine network comes with its own EPICS IOC: a small piece of software that translates whatever protocol the devices talk to a standardized format. That means that, depending on the experiments we run, we might have five to ten (or maybe even more) of these IOCs running on the machine server at a given time. Tracking which of these is running and (re)launching the necessary ones needs to be easy.
At the moment, we have at least eight IOCs running, to control:
- the sample stage (two to five motors, depending on the stage)
- the remaining 18 motors of the MOUSE instrument
- the two x-ray sources, which gives us shutter control and parameter readouts.
- the Arduino Portenta IOC for digital and analog I/O as well as temperature probe readout
- one for “parroting” extra experiment metadata
- one for reading out the pressure gauge
- one that can process statistics of detector images (beam positions, intensities and transmission factors). We don’t have AreaDetector running for our Eiger R 1M yet, so this is a bridging solution
- and several other small ones (some under development) for running syringe pumps, peristaltic pumps and hotplates.
The choice for managing these IOCs fell between abusing RunDeck (which we have been using to make our data processing easier) or using the Linux systemd services for this. Since the second one is actually meant for maintaining continuously running daemons, it seemed the more appropriate. Combined with the open source “Cockpit” web-based UI, this is now also manageable for dummies like me.
The recommendation from the people who’ve been down this road already, however, is to eventually containerize everything to make deployment easier. We will think about that in the context of other automation projects at BAM too.
6. Greasing slit spindles
We had to swap out a block of slits a few months ago for two reasons. Firstly, the previously “scatterless” downstream slits had started scattering as a sample evaporated in the vacuum and condensed on the slit blades. While this could be temporarily resolved with an aluminium tube (that blocked the diffraction ring now appearing), the next problem was that a slit motor started misbehaving. These are Schneider Electric MDrive motors (now under the Novanta IMS brand), and the controller was causing errors. After the replacement, the slits are mostly working fine again, even though some motors seem to be configured wrong. Despite that, the automated alignment works with whatever is given so we don’t really mind.
Recently, however, a spindle on another block of slits started squeaking. So it was time to take the tubes out of the collimation, open the slit boxes up and put some of our (expensive!) Braycote high-vacuum grease on the spindles. The squeak is gone, and hopefully it will stay gone for a while now. It did set us back a little for the realignment, however.
Back in business?
So after two weeks of hard work, we think we are almost ready to start measuring samples again. We’ll start easy and measure some more powder samples for a colleague from the University of Namibia (a collaboration I’m quite excited about). After that, we’ll have to rejig the data processing methods to work with the modified data formats: most of what we are doing now is the same as before, but the folder naming slightly changed as well as the metadata that is supplied, and I’d like to finally start putting the new HDF5Translator framework I’ve built into production. So it’ll be a few more weeks until we get our data processed, but we can at least continue measuring the raw data until then.
Next steps?
Once you’ve taken the final step, there’s nowhere else to go but further, I suppose. We’re going to be doing a few things:
- Upgrade some of the IOCs so they play nice, in particular the Trinamics one is almost-but-not-quite behaving like an EPICS Motor, and I’d like to get them compatible
- Look at building some visualisations / engineering dashboards of the MOUSE instrument to help us with operations. We’ve been shown CS Studio for this, which looks like it has a low barrier of entry and does a lot of what we need, but there are also other options out there.
- Building up measurement plans for more common experiments, such as for measuring liquids. The integration of the syringe injector should help with this.
- Prepare for some cool upcoming experiments including more work on battery materials and lipids.
Up next in May is another exciting step in this saga: we’re putting a band back together to write the modular data corrections into a Python library. It should work for any monochromatic X-ray as well as neutron experiment, and hopefully will find a home in many laboratories and beamlines. More on that later. For now, I wish everyone happy and relaxing holidays, and I hope to see you all again next year!