After about 1.5 months of struggles with the collapse of the veritable domino-stack of hardware that comprises the MOUSE, the system is now functional to such a degree that we can measure again. It is a bit of a hodgepodge at the moment with plenty of work remaining, but we can breathe a bit easier, knowing that measurements can happen over Christmas. Read on for the saucy deets on how we’re rapidly moving away from SPEC, and how our new Horizon Europe post-doc, Dr. Anja Hörmann, has helped get basic scanning functionality back.
Why the stress though?
Firstly, you might be wondering why we are stressing so much to get a lab machine working again. There are three reasons for this, the first being that I feel somewhat personally responsible for the state of the machine. I and my friends and colleagues have spent several years modifying it to measure in a completely different way than originally intended. After spending years modifying an instrument inside and out, you do build a connection with it, derive a certain amount of pride from it, and it just won’t do to have it not functional. Like our last breakdown mid-2021, we hope that the TLC added in this period will, in the end, improve it.
Secondly, people are depending on it to work. Anja, for example, needs this instrument to work so she can dive in to grazing incidence work for her project (a more detailed introduction will follow in the near future). Likewise, we have master’s students and other colleagues who are waiting for measurement data, there are interesting and important experiments to be performed, and I still have over 700 automatically-synthesised MOF samples that need to be measured. It is not a fast machine, but it is a stable workhorse, and we need it.
Thirdly, even in semi-off mode, the machine requires energy to run. Every week it spends not measuring is a waste of our -now very finite- planetary resources. Wasting time is not an option, both politically as well as scientifically. This is another reason we spent so much time on automating this piece of kit, so we can progress effectively with large amounts of reliable data.
Where are we now?
At the moment, the Motor controller is built, electrically approved, and connected. My new Trinamic EPICS IOC is programmed and functional enough to do moves on the various axes (with backlash), translate between the different coordinate systems and handle axis calibrations, etc.. The code repository is here.
After a plethora of bugs were squashed, and the motors were moving reliably, it was time to write scan methods. Motor scans are essential to alignment and calibrations of any instrument. As the new axes are no longer recognized by the original SPEC control system, we had to resort to other ways. We can move the axes to particular positions from SPEC by poking it with an EPICS command from within, but scans work differently and would require recognized motors.
So Anja, who also needs these scans for her GI alignments, got to work on a Python script to do this from outside SPEC. By controlling the motor via EPICS from there on the one hand, and reading out the detector with their API calls, scans are now finally possible again for the sample motors. Cue a huge sigh of relief as the first scans came in. We’re back in business in a jury-rigged fashion: we can run the measurements using our regular scripts used in SPEC, and can align the samples and sample holders with the Python scripts. There’s even a live dashboard using DASH+Plotly on the scans!
What remains to be done?
Getting it working is one thing, getting it working nice is a whole other project, but one that can be done with a bit more consideration, contemplation, and, hopefully, collaboration. Now in this process, we came across issues that need resolving in the near future.
- Firstly, and unrelatedly, our APC UPS battery overheated, so we took the UPS out of the loop altogether for now. They appear to have swollen as well. Make sure to regularly check your battery systems!
- We also discovered to our horror that we seem to have nonfunctioning limit switches in most of our stages. I might be wrong about this, and we’re troubleshooting with the (commendably responsive and helpful) manufacturer to find out what’s going on and what and how need replacing. So for now, our motor stages are un-homed, and run without end stop limits.
- Now that we have the sample stage motors on EPICS, we will move forward and do the same for the remaining motor stages in the MOUSE, most of which are controlled using Schneider Electric’s MForce or MDrive controllers. An EPICS IOC exists for this in the synApps collection, so we’re hoping to get those to run.
- Once that’s going, we will try installing ADEiger to get our detector onto the EPICS network. I have never worked with EPICS’ AreaDetector devices, so it may work or fail gloriously. In any case, we can control the detector from Python directly if needed.
- Then there’s a few more odds and ends to add to the network, after which we’ll have to take a look and see if we should bundle our hardware using Ophyd and control them via Bluesky, or perhaps look towards alternatives such as the French BLISS.
All in all, plenty of work ahead in the conversion, but as said, we can take our time on this. Simultaneously, there is a small underground lab community forming that wants to do the same to their machines, so here’s hoping that we can effectively bundle forces to achieve similar goals. If you want in on the action, feel free to drop me a line!