{"id":4053,"date":"2024-02-23T14:22:52","date_gmt":"2024-02-23T13:22:52","guid":{"rendered":"https:\/\/lookingatnothing.com\/?p=4053"},"modified":"2024-02-23T14:24:28","modified_gmt":"2024-02-23T13:24:28","slug":"steps-for-liberating-the-mouse-from-its-original-control-system","status":"publish","type":"post","link":"https:\/\/lookingatnothing.com\/index.php\/archives\/4053","title":{"rendered":"Steps for &#8220;liberating&#8221; the MOUSE from its original control system."},"content":{"rendered":"\n<p>Regular readers will know we are in the process of migrating away from the last vestiges of the original SPEC-based control system on our (already highly customized) Xenocs Xeuss 2.0 instrument, i.e. the MOUSE. While we&#8217;ve been doing this so far in a rather unorganised whatever-comes-first manner, the next month we will have a few visitors capable of helping us with this. So it&#8217;s time to make a list and a plan, which could be useful to those of you who also wish to, for lack of a better word, further &#8220;liberate&#8221; their machines. <\/p>\n\n\n\n<!--more-->\n\n\n<div id=\"ez-toc-container\" class=\"ez-toc-v2_0_71 counter-hierarchy ez-toc-counter ez-toc-grey ez-toc-container-direction\">\n<div class=\"ez-toc-title-container\">\n<p class=\"ez-toc-title\" style=\"cursor:inherit\">Table of Contents<\/p>\n<span class=\"ez-toc-title-toggle\"><a href=\"#\" class=\"ez-toc-pull-right ez-toc-btn ez-toc-btn-xs ez-toc-btn-default ez-toc-toggle\" aria-label=\"Toggle Table of Content\"><span class=\"ez-toc-js-icon-con\"><span class=\"\"><span class=\"eztoc-hide\" style=\"display:none;\">Toggle<\/span><span class=\"ez-toc-icon-toggle-span\"><svg style=\"fill: #999;color:#999\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" class=\"list-377408\" width=\"20px\" height=\"20px\" viewBox=\"0 0 24 24\" fill=\"none\"><path d=\"M6 6H4v2h2V6zm14 0H8v2h12V6zM4 11h2v2H4v-2zm16 0H8v2h12v-2zM4 16h2v2H4v-2zm16 0H8v2h12v-2z\" fill=\"currentColor\"><\/path><\/svg><svg style=\"fill: #999;color:#999\" class=\"arrow-unsorted-368013\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"10px\" height=\"10px\" viewBox=\"0 0 24 24\" version=\"1.2\" baseProfile=\"tiny\"><path d=\"M18.2 9.3l-6.2-6.3-6.2 6.3c-.2.2-.3.4-.3.7s.1.5.3.7c.2.2.4.3.7.3h11c.3 0 .5-.1.7-.3.2-.2.3-.5.3-.7s-.1-.5-.3-.7zM5.8 14.7l6.2 6.3 6.2-6.3c.2-.2.3-.5.3-.7s-.1-.5-.3-.7c-.2-.2-.4-.3-.7-.3h-11c-.3 0-.5.1-.7.3-.2.2-.3.5-.3.7s.1.5.3.7z\"\/><\/svg><\/span><\/span><\/span><\/a><\/span><\/div>\n<nav><ul class='ez-toc-list ez-toc-list-level-1 ' ><li class='ez-toc-page-1 ez-toc-heading-level-1'><a class=\"ez-toc-link ez-toc-heading-1\" href=\"https:\/\/lookingatnothing.com\/index.php\/archives\/4053\/#The_backstory_skip_if_youve_heard_this_one\" title=\"The backstory (skip if you&#8217;ve heard this one!)\">The backstory (skip if you&#8217;ve heard this one!)<\/a><ul class='ez-toc-list-level-2' ><li class='ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-2\" href=\"https:\/\/lookingatnothing.com\/index.php\/archives\/4053\/#In_the_beginning_there_was%E2%80%A6\" title=\"In the beginning, there was&#8230;\">In the beginning, there was&#8230;<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-3\" href=\"https:\/\/lookingatnothing.com\/index.php\/archives\/4053\/#The_changes_we_made%E2%80%A6\" title=\"The changes we made&#8230;\">The changes we made&#8230;<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-4\" href=\"https:\/\/lookingatnothing.com\/index.php\/archives\/4053\/#What_direction_to_take\" title=\"What direction to take?\">What direction to take?<\/a><\/li><\/ul><\/li><li class='ez-toc-page-1 ez-toc-heading-level-1'><a class=\"ez-toc-link ez-toc-heading-5\" href=\"https:\/\/lookingatnothing.com\/index.php\/archives\/4053\/#The_upcoming_changes\" title=\"The upcoming changes\">The upcoming changes<\/a><ul class='ez-toc-list-level-3' ><li class='ez-toc-heading-level-3'><ul class='ez-toc-list-level-3' ><li class='ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-6\" href=\"https:\/\/lookingatnothing.com\/index.php\/archives\/4053\/#Adding_a_new_server\" title=\"Adding a new server\">Adding a new server<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-7\" href=\"https:\/\/lookingatnothing.com\/index.php\/archives\/4053\/#Controlling_Schneider_MDriveMForce_motors\" title=\"Controlling Schneider MDrive\/MForce motors\">Controlling Schneider MDrive\/MForce motors<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-8\" href=\"https:\/\/lookingatnothing.com\/index.php\/archives\/4053\/#Addressing_and_reading_out_the_detector_directly\" title=\"Addressing and reading out the detector directly\">Addressing and reading out the detector directly<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-9\" href=\"https:\/\/lookingatnothing.com\/index.php\/archives\/4053\/#Generator_control\" title=\"Generator control\">Generator control<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-10\" href=\"https:\/\/lookingatnothing.com\/index.php\/archives\/4053\/#Valves_control\" title=\"Valves control\">Valves control<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-11\" href=\"https:\/\/lookingatnothing.com\/index.php\/archives\/4053\/#A_UI_for_managing_microservices\" title=\"A UI for managing microservices\">A UI for managing microservices<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-12\" href=\"https:\/\/lookingatnothing.com\/index.php\/archives\/4053\/#An_IOC_for_tracking_metadata\" title=\"An IOC for tracking metadata\">An IOC for tracking metadata<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-13\" href=\"https:\/\/lookingatnothing.com\/index.php\/archives\/4053\/#Moving_the_generator_control_boxes_upstream\" title=\"Moving the generator control boxes upstream\">Moving the generator control boxes upstream<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-14\" href=\"https:\/\/lookingatnothing.com\/index.php\/archives\/4053\/#Changing_the_workflow_programs\" title=\"Changing the workflow programs.\">Changing the workflow programs.<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-15\" href=\"https:\/\/lookingatnothing.com\/index.php\/archives\/4053\/#Stretch_goal_1_An_ultimate_pure_pythonic_data_processing_pipeline\" title=\"Stretch goal 1: An ultimate, pure pythonic data processing pipeline\">Stretch goal 1: An ultimate, pure pythonic data processing pipeline<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-16\" href=\"https:\/\/lookingatnothing.com\/index.php\/archives\/4053\/#Stretch_goal_2_A_better_UI_for_loading_sample_information_and_measurement_settings\" title=\"Stretch goal 2: A better UI for loading sample information and measurement settings\">Stretch goal 2: A better UI for loading sample information and measurement settings<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-17\" href=\"https:\/\/lookingatnothing.com\/index.php\/archives\/4053\/#Stretch_goal_3_A_live_ops_dashboard\" title=\"Stretch goal 3: A live ops dashboard.\">Stretch goal 3: A live ops dashboard.<\/a><\/li><\/ul><\/li><\/ul><\/li><\/ul><\/nav><\/div>\n\n\n\n\n<figure class=\"wp-block-gallery has-nested-images columns-default is-cropped wp-block-gallery-2 is-layout-flex wp-block-gallery-is-layout-flex\">\n<figure class=\"wp-block-image size-large\"><a href=\"https:\/\/lookingatnothing.com\/wp-content\/uploads\/2024\/02\/image.png\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"749\" data-id=\"4058\" src=\"https:\/\/lookingatnothing.com\/wp-content\/uploads\/2024\/02\/image-1024x749.png\" alt=\"\" class=\"wp-image-4058\" srcset=\"https:\/\/lookingatnothing.com\/wp-content\/uploads\/2024\/02\/image-1024x749.png 1024w, https:\/\/lookingatnothing.com\/wp-content\/uploads\/2024\/02\/image-300x220.png 300w, https:\/\/lookingatnothing.com\/wp-content\/uploads\/2024\/02\/image-768x562.png 768w, https:\/\/lookingatnothing.com\/wp-content\/uploads\/2024\/02\/image-80x60.png 80w, https:\/\/lookingatnothing.com\/wp-content\/uploads\/2024\/02\/image.png 1536w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/a><figcaption class=\"wp-element-caption\">How it started (2018)<\/figcaption><\/figure>\n\n\n\n<figure class=\"wp-block-image size-large\"><a href=\"https:\/\/lookingatnothing.com\/wp-content\/uploads\/2024\/02\/image-1.png\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"765\" data-id=\"4059\" src=\"https:\/\/lookingatnothing.com\/wp-content\/uploads\/2024\/02\/image-1-1024x765.png\" alt=\"\" class=\"wp-image-4059\" srcset=\"https:\/\/lookingatnothing.com\/wp-content\/uploads\/2024\/02\/image-1-1024x765.png 1024w, https:\/\/lookingatnothing.com\/wp-content\/uploads\/2024\/02\/image-1-300x224.png 300w, https:\/\/lookingatnothing.com\/wp-content\/uploads\/2024\/02\/image-1-768x574.png 768w, https:\/\/lookingatnothing.com\/wp-content\/uploads\/2024\/02\/image-1-326x245.png 326w, https:\/\/lookingatnothing.com\/wp-content\/uploads\/2024\/02\/image-1-80x60.png 80w, https:\/\/lookingatnothing.com\/wp-content\/uploads\/2024\/02\/image-1.png 1496w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/a><figcaption class=\"wp-element-caption\">How it&#8217;s going (2021)<\/figcaption><\/figure>\n\n\n\n<figure class=\"wp-block-image size-large\"><a href=\"https:\/\/lookingatnothing.com\/wp-content\/uploads\/2024\/02\/image-2.png\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"664\" data-id=\"4060\" src=\"https:\/\/lookingatnothing.com\/wp-content\/uploads\/2024\/02\/image-2-1024x664.png\" alt=\"\" class=\"wp-image-4060\" srcset=\"https:\/\/lookingatnothing.com\/wp-content\/uploads\/2024\/02\/image-2-1024x664.png 1024w, https:\/\/lookingatnothing.com\/wp-content\/uploads\/2024\/02\/image-2-300x194.png 300w, https:\/\/lookingatnothing.com\/wp-content\/uploads\/2024\/02\/image-2-768x498.png 768w, https:\/\/lookingatnothing.com\/wp-content\/uploads\/2024\/02\/image-2-1536x996.png 1536w, https:\/\/lookingatnothing.com\/wp-content\/uploads\/2024\/02\/image-2.png 1728w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/a><figcaption class=\"wp-element-caption\">How it&#8217;s going further (Jan. 2024)<\/figcaption><\/figure>\n<figcaption class=\"blocks-gallery-caption wp-element-caption\">Progress of the hardware (and publication record) over time&#8230; <\/figcaption><\/figure>\n\n\n\n<h1 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"The_backstory_skip_if_youve_heard_this_one\"><\/span>The backstory (skip if you&#8217;ve heard this one!)<span class=\"ez-toc-section-end\"><\/span><\/h1>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"In_the_beginning_there_was%E2%80%A6\"><\/span>In the beginning, there was&#8230; <span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p>The original software that came with the instrument was okay, with a graphical user interface allowing for various standard measurements to be done using a moderately user-friendly interface. Back then, the <a href=\"https:\/\/www.synchrotron-soleil.fr\/en\/beamlines\/swing\">Foxtrot<\/a> software by the SWING team from Soleil was added to allow some data reduction to take place. For scripting of measurements, one could fall back to SPEC programming. Such a &#8216;human-in-the-middle&#8217; working style, while tedious and a bit fiddly, poses no major hurdles for the casual measurement. <\/p>\n\n\n\n<p>But it can make one feel constrained and limited: if you stick within the boundaries set by the manufacturer, you will never do more or better than they envisioned (and we can envision a lot!). The original set-up also precluded a drive towards more automation and tighter integration of equipment. Talking to some other instrument scientists &#8211; and not limited to SAXS &#8211; this feeling seems to be very common. <\/p>\n\n\n\n<p>We can see how even a few small improvements here and there which would greatly improve either the data quality or ease-of-use in our methodology, but we are unable to implement these without in-depth, documented access to the instrument controls and voiding warranty. Blocked by this, we simmer in our impotence to make these changes, devolving into a bunch of grumbling, middle-aged scientists; fondly &#8220;remembering&#8221; the age when machines would proudly come with electrical diagrams and communications protocols as standard. <\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"The_changes_we_made%E2%80%A6\"><\/span>The changes we made&#8230;<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p>So we almost immediately ditched the UI when we got our system, moving instead to an <a href=\"https:\/\/iopscience.iop.org\/article\/10.1088\/1748-0221\/16\/06\/P06034\">overarching methodology<\/a> involving an electronic logbook with automatic script generation, first with manual <a href=\"https:\/\/dawnsci.org\">DAWN<\/a> data processing, then automatically (headless), then with automatic calibrations, automatic thickness determination, eventually adding <a href=\"https:\/\/github.com\/scicatproject\">SciCat<\/a> to catalog these steps, and <a href=\"https:\/\/www.rundeck.com\">RunDeck<\/a> to manage this data flow. <\/p>\n\n\n\n<p>More recently we added automatic data merging (from measurements in different configurations) and even a casual automatic data fitting procedure with <a href=\"https:\/\/github.com\/bamresearch\/mcsas3\">McSAS3<\/a>. On the hardware side, we replaced the sample motor controllers and sample stages, and added <a href=\"https:\/\/lookingatnothing.com\/index.php\/archives\/4024\">temperature controllers<\/a>, environmental readouts, multi-position <a href=\"https:\/\/lookingatnothing.com\/index.php\/archives\/2920\">flow-through sample holders<\/a>, <a href=\"https:\/\/zenodo.org\/records\/7501030\">a heated flow-through cell<\/a>, a UPS (now retired), a new network switch, an EPICS server to handle some hardware, a prototype <a href=\"https:\/\/lookingatnothing.com\/index.php\/archives\/3230\">USAXS<\/a> setup, now also a <a href=\"https:\/\/lookingatnothing.com\/index.php\/archives\/3847\">GISAXS set-up<\/a>, and lots of feedthroughs for liquids, coolants and electrical connections. With all those changes, <a href=\"https:\/\/lookingatnothing.com\/index.php\/archives\/4015\">instrument performance and efficiency is through the roof<\/a>, despite our understaffed laboratory. <\/p>\n\n\n\n<p>But some of the core code still is <a href=\"https:\/\/certif.com\/spec.html\">SPEC<\/a>, and after the <a href=\"https:\/\/lookingatnothing.com\/index.php\/archives\/3934\">motor<\/a> <a href=\"https:\/\/lookingatnothing.com\/index.php\/archives\/3961\">controller<\/a> <a href=\"https:\/\/lookingatnothing.com\/index.php\/archives\/3975\">incident<\/a> leading to new sample motor controllers, we are hitting issues with SPEC. It is perfectly possible to call upon the OG programmer of SPEC, Gerry Swislow, to help us once more. Honestly, though, that would only be a stopgap solution until the next issue. And so it was as good a time as any to try moving away from the remnants. So, what do we need to do?<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"What_direction_to_take\"><\/span>What direction to take?<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p>So, after looking around at some of the alternatives (<a href=\"https:\/\/www.esrf.fr\/BLISS\">Bliss<\/a> and <a href=\"https:\/\/www.tango-controls.org\">Tango<\/a>, to name but two, gleefully ignoring LabView), we decided to commit to an <a href=\"https:\/\/epics-controls.org\">EPICS<\/a> base. I&#8217;ve talked about this before, but it forms the basis of many synchrotrons, neutron sources as well as their instruments and therefore will be around for a while. It has the added advantage that it is open source, and you can put <a href=\"https:\/\/blueskyproject.io\/ophyd\/\">Ophyd<\/a> + <a href=\"https:\/\/blueskyproject.io\/bluesky\/\">BlueSky<\/a> on top, which form a modern user interface that is suitable for close integration and interaction with both instruments as well as their data pipelines. Both are programmed and supported by a large user base, and already contain drivers for <a href=\"https:\/\/epics-controls.org\/resources-and-support\/modules\/hardware-support\/?listpage=11&amp;instance=1\">much of the hardware we need to interact with<\/a>. The only downside is that they are usually installed and maintained by dedicated teams at large facilities, not one or two people at small laboratories. Can we successfully install and adapt the bits we need?<\/p>\n\n\n\n<h1 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"The_upcoming_changes\"><\/span>The upcoming changes<span class=\"ez-toc-section-end\"><\/span><\/h1>\n\n\n\n<p>We need to make the following changes: <\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Adding_a_new_server\"><\/span>Adding a new server<span class=\"ez-toc-section-end\"><\/span><\/h3>\n\n\n\n<p>This one was relatively easy, our IT supplied a modern server with two network cards, modest memory (32GB) and an SSD. This one has had Ubuntu LTS installed on it as a base, on which we will run the many IOCs, some dashboards and perhaps RunDeck. <\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Controlling_Schneider_MDriveMForce_motors\"><\/span>Controlling Schneider MDrive\/MForce motors<span class=\"ez-toc-section-end\"><\/span><\/h3>\n\n\n\n<p>Besides the sample motors, we must also be able to drive slit motors and detector motors. These are driven by variants of what is now Schneider Electric&#8217;s MDrive\/MForce style motors over an RS485 bus. Fortunately for us, these motor drivers are supported by the enormous ANL-developed <a href=\"https:\/\/github.com\/EPICS-synApps\">SynApps<\/a> package. Unfortunately for us, installation of these outside of ANL is not a quick fire-and-forget operation, and so my colleague Ingo is spending some spare moments in an effort to try and get these running on normal installations. Compilation has (as of yesterday) succeeded, so now we have to find out how to configure these and get them to run. <\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Addressing_and_reading_out_the_detector_directly\"><\/span>Addressing and reading out the detector directly<span class=\"ez-toc-section-end\"><\/span><\/h3>\n\n\n\n<p>For our new GISAXS experiments, Anja has been interacting with the Dectris detector directly, stealing control away from the Xenocs system when needed. This is going reasonably fine, and we are collecting the nicely informative <a href=\"https:\/\/www.nexusformat.org\">NeXus<\/a> files directly from the detector for further processing, be it immediate for scanning functionality, and directly storing these on our networked server for detailed analysis. Extending this to also take over the measurements for our standard transmission measurements might be easy. Alternatively, probably at a later stage when BlueSky comes into the picture, we can use the EPICS\/SynApps drivers for the detector too. <\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Generator_control\"><\/span>Generator control<span class=\"ez-toc-section-end\"><\/span><\/h3>\n\n\n\n<p>We do not need generator control as such, but only need to be able to read out its status and flip the shutter open and closed. For this, Xenocs has been kind enough to provide sufficient information to me, so that I could write a small EPICS IOC using Caproto to expose this functionality to the machine network. I&#8217;m still struggling with (a perhaps unnecessary) asynchronous handling library, but other than that, it should run fine soon. <\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Valves_control\"><\/span>Valves control<span class=\"ez-toc-section-end\"><\/span><\/h3>\n\n\n\n<p>The vacuum and compressed air system is controlled using a few valves. These valves are driven by a small ethernet-attached relay interface, which would be no problem to reuse. However, I&#8217;ve become enamoured by the new <a href=\"https:\/\/www.arduino.cc\/pro\/portenta-machine-control\">Arduino Portenta Machine Control (PMC)<\/a>, and so I am planning to use this instead. I have written an EPICS IOC using Caproto to expose the PMC functionality to the EPICS network, simultaneously reading out some environmental parameters (temperature, pressure, humidity, dewpoint and volatile chemicals). This IOC is in beta, but available <a href=\"https:\/\/github.com\/bamresearch\/NetworkedPortenta\">here<\/a> for you to use and (hopefully) contribute. More on this Arduino in a separate post once it has its pretty 19&#8243; box around it that the workshop is working on right now. <\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"A_UI_for_managing_microservices\"><\/span>A UI for managing microservices<span class=\"ez-toc-section-end\"><\/span><\/h3>\n\n\n\n<p>All this talk about IOCs must make you wonder how we want to manage all these little server programs. They launch with nothing more than a command-line command splattered with a few options, and in principle run in a screen without issue. However, when you have a few of them, it is easy to lose track of which is doing what. So I wrote a small Dash dashboard to launch and keep track of these, and at the same time show a little graph on the system and network resources load. That dashboard is available <a href=\"https:\/\/github.com\/bamresearch\/IOCLauncher\">here<\/a> for you to use and (hopefully) contribute (it bears repeating). <\/p>\n\n\n\n<p>On second thought, we might just run another RunDeck instance on this server to manage these (the same software we use for managing our data pipeline), since that does 90% of the above, and is the more pragmatic solution. Too bad about the time spent writing my own, but I learned a few more things in the process of writing it. <\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"An_IOC_for_tracking_metadata\"><\/span>An IOC for tracking metadata<span class=\"ez-toc-section-end\"><\/span><\/h3>\n\n\n\n<p>So there is some information the machine never knows. This includes information such as who is running the measurements, who owns the samples, what sample is being measured, what its composition is (with SLDs and absorption coefficients), what proposal this is for, and many more. To make it easier for us to record this later in the NeXus files, and to help us eventually build a dashboard showing the current status of the instrument complete with names and sample information (see below), we need this information exposed. So a little, easily extensible &#8220;Parrot&#8221; IOC needs to be built that does nothing more than keep track of a few strings and numbers. <\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Moving_the_generator_control_boxes_upstream\"><\/span>Moving the generator control boxes upstream<span class=\"ez-toc-section-end\"><\/span><\/h3>\n\n\n\n<p>This is pretty much completely unnecessary, but I want to do it anyways. Underneath the MOUSE, the cabling is a bit of a mess. Part of the reason for this is that most electronics boxes are centrally positioned, some with thick cables, leading to this mess. I like tidy cables and tidy interiors, so this irks me. One way to tidy this up is to move the control boxes sideways, next to where the X-ray sources themselves are, letting us compartmentalise the cabling a little. In my mind, it would also reduce the risk in case of a large liquid spill on top of the optical table, but there is still plenty of hardware underneath to make that problematic if it were to happen. One step further, though. <\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Changing_the_workflow_programs\"><\/span>Changing the workflow programs.<span class=\"ez-toc-section-end\"><\/span><\/h3>\n\n\n\n<p>At the end, the scripting engine needs to be reworked so it can make measurement scripts for the new system. Likewise, the program that did the NeXus-to-NeXus translation from the Xenocs and Dectris base files into a nice comprehensive NeXus file DAWN could work with, needs to be modified to work with the new information sources. Likewise, there are probably a few other steps in the workflow that need some adjustment before it works as smoothly as it did before (and hopefully even more smoothly). This story will have a bit of a tail.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Stretch_goal_1_An_ultimate_pure_pythonic_data_processing_pipeline\"><\/span>Stretch goal 1: An ultimate, pure pythonic data processing pipeline<span class=\"ez-toc-section-end\"><\/span><\/h3>\n\n\n\n<p>Before the end of last year was so rudely interrupted with a hardware failure, I had started working on an upgraded pythonic dataclass (a data carrier) to use with a comprehensive data processing pipeline (<a href=\"https:\/\/lookingatnothing.com\/index.php\/archives\/3975\">we talked about this before<\/a>). That would be useful not just for data correction, but also for any subsequent steps you want to tack on at the end such as in-line data analysis steps or ML-based categorisation, classification and tagging steps. This is a large project for which we&#8217;ll need to get a few heads together. More on this at a later stage. <\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Stretch_goal_2_A_better_UI_for_loading_sample_information_and_measurement_settings\"><\/span>Stretch goal 2: A better UI for loading sample information and measurement settings<span class=\"ez-toc-section-end\"><\/span><\/h3>\n\n\n\n<p>While Excel has its strengths (user familiarity, flexibility, good at making series and sequences), it also has many drawbacks. This is exemplified by the information we get from users, who have no problems inserting &#8220;~2.3 to 2.8&#8221; in nominally machine-readable density fields, copy-pasting images of ChemDraw structures (with lots of unknown <sub>n<\/sub>&#8216;s and <sub>x<\/sub>&#8216;s) in another machine-readable atomic composition field, or just ignoring the whole structure and writing a detailed text description of their sample. This is not their fault; Excel allows you to do these things. But it does make life hard for those of us who want to automate certain aspects such as X-ray absorption and SLD calculations. <\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Stretch_goal_3_A_live_ops_dashboard\"><\/span>Stretch goal 3: A live ops dashboard. <span class=\"ez-toc-section-end\"><\/span><\/h3>\n\n\n\n<p>Lastly, wouldn&#8217;t it be cool to see the latest frames pop up on a dashboard in the coffee room and office, with the sample information and other live statistics coming up? Stay tuned for that. <\/p>\n","protected":false},"excerpt":{"rendered":"<div class=\"mh-excerpt\"><p>Regular readers will know we are in the process of migrating away from the last vestiges of the original SPEC-based control system on our (already <a class=\"mh-excerpt-more\" href=\"https:\/\/lookingatnothing.com\/index.php\/archives\/4053\" title=\"Steps for &#8220;liberating&#8221; the MOUSE from its original control system.\">[&#8230;]<\/a><\/p>\n<\/div>","protected":false},"author":2,"featured_media":4060,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"activitypub_content_warning":"","activitypub_content_visibility":"","activitypub_max_image_attachments":4,"activitypub_interaction_policy_quote":"anyone","activitypub_status":"","footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2}},"categories":[1],"tags":[],"class_list":["post-4053","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-uncategorized"],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"https:\/\/lookingatnothing.com\/wp-content\/uploads\/2024\/02\/image-2.png","jetpack_shortlink":"https:\/\/wp.me\/p1gZ2v-13n","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/lookingatnothing.com\/index.php\/wp-json\/wp\/v2\/posts\/4053","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/lookingatnothing.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/lookingatnothing.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/lookingatnothing.com\/index.php\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/lookingatnothing.com\/index.php\/wp-json\/wp\/v2\/comments?post=4053"}],"version-history":[{"count":6,"href":"https:\/\/lookingatnothing.com\/index.php\/wp-json\/wp\/v2\/posts\/4053\/revisions"}],"predecessor-version":[{"id":4062,"href":"https:\/\/lookingatnothing.com\/index.php\/wp-json\/wp\/v2\/posts\/4053\/revisions\/4062"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/lookingatnothing.com\/index.php\/wp-json\/wp\/v2\/media\/4060"}],"wp:attachment":[{"href":"https:\/\/lookingatnothing.com\/index.php\/wp-json\/wp\/v2\/media?parent=4053"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/lookingatnothing.com\/index.php\/wp-json\/wp\/v2\/categories?post=4053"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/lookingatnothing.com\/index.php\/wp-json\/wp\/v2\/tags?post=4053"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}