Last time, I showed the rapid development of the replacement motor controller hardware, and pointed at the development of the driver software (an EPICS IOC for directing the motor controller from the machine network). That programming task was a bit complex, and involved about a week of testing and debugging to get it to work reliably with most of the necessary features. Today, I will tell you about how we added this to the MOUSE instrument, and how this helps us migrate away from the SPEC control system currently providing hardware control of the MOUSE.
At this point, we are assuming that the control software steering the MOUSE consists of the SPEC version *with* support for EPICS motors. This will be tested in a few paragraphs.
According to the SPEC documentation, any EPICS-controlled motor should support about 20 specific EPICS fields, so I made sure that my IOC did most of these. Check out the IOC here if you have a Trinamic motor control board that you would like to address over EPICS CA, as it should support most of their boards. Given the structure of the code, implementing many extra fields should not be a problem. Contributions to the code are – as always – very welcome.
Before hooking up the motors again, we need to install the hardware in a reasonable location. Thanks to a generous person, we now have three 19″ racks (and a few empty NIM bins to boot). Since almost everything I build is in a 19″ enclosure, this will fill up nicely soon enough. So we quickly moved our EPICS server into the rack, alongside the electrical safety interlock and the new motor controller, and it’s already starting to look like it belongs there.
Following that, it was time to try and figure out if SPEC wants to talk to this thing. It turns out that I did not implement a few fields, and that makes SPEC claim the motor controller is unresponsive. We do have a small trick up our sleeve, i.e. that we can tell SPEC to send messages to the EPICS network directly, but that would preclude the use of motor-reliant functions like axis scans. So, I’ve to update the IOC to include these extra fields, most notably the “SET” field (which is something a bit more complicated) and the mystery “DISP” field (which I still don’t know what it should do). It also gives another impetus to migrate away from SPEC…
But when we started testing the motor stage motions even without SPEC, we noticed some oddness in the limit switches. The MKS/Newport stages we have, use optical gates for limit switches, and, after some systematic troubleshooting, it now looks like these have failed on most of the stages that were in the MOUSE; the stage I used for testing worked just fine when placed in the MOUSE. Now that I have been informed how much current these motor stage limit switch circuits use, I realise that the original motor controllers were likely unable to provide sufficient power for them, which may have caused their demise over the long term, kicking off trouble further down the line. Although this is not a certainty as it could just as easily have been the extended exposure of these optical gates to vacuum. Here’s hoping I can get some replacement limit switch boards for the stages and restore them to full working order.
For now, We’ve a multi-pronged approach to resolving some of these issues. For one, we’ll use un-homed, limit-switch-free operation for the sample stages. This means the sample stage will be more difficult to use as it is not position-aware and thus needs its position to be manually calibrated. The second solution is to directly send epics commands from SPEC to get the motors to move to their final position, which is not ideal but workable. The third solution is to try implementing the remaining EPICS fields in the IOC so that, hopefully, SPEC will recognise them as fully functional motors. Lastly, I’ll be trying to get some spare components to fix the motor stage limit switches or send them in for servicing.
If these efforts fail to make it work properly again with SPEC, it might just be time to expedite the migration from SPEC to something else.. maybe pure Python or Jupyter notebooks, with an eventual transition to Bluesky. Since we already have ways of addressing the detector via a Python library, and since the motor controller responds just fine in principle, there isn’t that much more we would need to bypass SPEC altogether, hypothetically. But it would take more programming to get such a solution to work reliably, and I’m sure there’ll be plenty of integration hell if one would try.
For now I hope we’ll be back in “normal” operation soon as there are many people in need of measurements for their respective projects. Once operation has resumed, I’m really hoping to get rid of this guilt towards them for having a broken machine post haste.
And I think back fondly to my naive self, but a few weeks ago, who thought all the MOUSE needed was a new motor controller… What a journey this has turned out to have become.
To be continued…