Over the last two decades, I’ve worked with and visited even more scattering instruments and synchrotron beam lines. For Reasons (™), I have been putting some thought into how I would set up and organise a highly automated, holistic scattering beamline, if I were given free rein. Find out more below…
The issues worth addressing
The way scattering beamlines operate at the moment obviously works to some degree. Some even fell into the argument ad antiquitam fallacy of “we’ve always done it this way, and it’s always been shit”, not realising that rules, traditions and methods once laid down by ancestors should actually be revamped. Nonetheless, in most there are large inefficiencies in the process and problems that result in poor science. Some of the ones I noticed over the last 20 years include:
- Preparation and sample selection: Poor technique understanding and preparation means that users come with incomplete sets or insufficient samples. Deciding on the fly which samples to run next is also not ideal with a tired or inexperienced crew. Backgrounds and calibration measurements are forgotten or not taken at sufficient intervals. Lastly, if the sample preparation has not been done or documented sufficiently well, it’s not even worth continuing the experiment at this point as the resulting science will be sub-par. There needs to be an additional go/no-go decision point after the project and sample information has been received to avoid wasting everyone’s time.
- Unnecessary Training: The (re-)training of users takes resources but is not necessarily worth it for standard samples that could be run in a standard manner (capillaries/liquids, solids, mapping experiments, and a decent fraction of in-situ and operando studies). What would take 15 minutes to set up by experienced staff in dedicated production time now takes a day of handholding. Lack of experience also increases the risk of improper operation leading to bad science (or a waste of time or equipment).
- Disconnect by SAC and PRP vs. structured beamline development: Depending on their level of involvement in the beamline and understanding of the technique, the Science Advisory Committee (SAC) and Peer Review Panel (PRP) may not be able to select proposals that structurally advance the beamline development. Put simply: you cannot make in-depth improvements to a beamline if you are continually asked to support too diverse a range of disconnected experiments. Beamline staff should be able to indicate which direction they want to go long-term, and rank proposals on their contribution to this goal.
- Different access methods: Fortunately, many synchrotrons already support different levels of access, requiring different gradations of proposals. Mail-in, proof-of-principle, and short (or standard) measurements should require much less scrutiny than long, in-depth projects.
- Insufficient ROI: The chance that a beamtime results in a scientific finding remains too low. I’m not talking about the problems of instrument breakdowns and quietly ignored negative scientific findings, but data not being corrected and exploited to the fullest, if at all. We cannot expect the current cadre of multidisciplinary users to be intimately familiar with the pitfalls of our science. And so, my final point:
- From facilitator to partner: We specialists need to get more involved in the entire project, offering more guidance and education before the experiment, consulting on whether or not to continue and to what degree, advising on proof-of-principle measurements, doing the measurements and data corrections, advising on and testing analysis strategies, and even helping by providing tools for data interpretation (i.e. interpretation of all sample series data connected to the metadata on the samples). This takes time, effort, and mostly people.
Freeing up time via a “production” scattering beamline
Using highly adaptable beamlines for standard experiments is both a waste of capacity and leads to imperfect data, as you can’t expect a flexible beamline to also be highly optimised to produce the best data. So I’m arguing for a highly automated “production” beamline, that can lighten the pressure on the flexible beamlines by processing the large fraction of standard samples. That leaves the flexible beamlines free to focus on their strength, while the production beamline can gradually build up a vast library of high-quality measurements for both types to compare against.
Goals:
The goal of the production beamline would be to carry out reference-quality holistic scattering experiments with partners (the materials scientists, geologists and biologists, formerly known as users). We know this can be done, as we duo it in on a smaller scale in the MOUSE lab now. In other words, we, as a beamline, would get involved in the entire experiment for a select set of high-throughput projects and would guarantee high-quality scientific outcomes through fully-exploited, high-quality data. That implies:
- Education of scientists on the use and utility of the technique for their materials, via courses, materials and video calls.
- Careful selection of projects to partner with, with input from (but not necessarily solely decided by) a peer-review panel for larger-scope projects.
- Assisting the users with sample preparation (either through guidance or with the help of a small automated materials preparation platform), ensuring unparalleled metadata collection during the preparation
- Automated measurements with on-line quality checks
- Automated in-line data corrections (of course!), resulting in corrected, n-dimensional scattering data on absolute scale with uncertainty estimates
- Automated in-line preliminary analysis (which can be fed back to the measurement orchestrator software), and classification with ML systems to match to similar experiments and -results.
- Automated reporting of the results to the partners as they come in from the beamline, and storage of the results in a growing database of measurements.
- Aids for interpretation of the analyses by advanced visualisation tools of the results from the project measurements
Automation means limiting to what you do well
This automation comes at a bit of a price. We have been automating our instrument, which means we carry out experiments along a more or less fixed scheme. All samples are measured in all configurations, with standard exposure settings (for now) which we know work well for almost all samples. We have developed some standard sample holders and will use other ones only in case of a unique need.
My ideal beamline would do the same. We would offer a choice between a set of predefined, pre-calibrated instrument configurations for which everything has been set up. We offer a standard measurement scheme that automatically produces reliable data. Nonstandard configurations would be possible, but would need a solid, well-founded justification, and would come with fewer guarantees. Essentially: “Our way guarantees a decent outcome. Deviations void this warranty”.
It also means gradually building up capabilities over time. Boring ones for starters, and more exotic as time goes on.
On attending the experiments
I don’t think it makes much sense anymore for nearly every experiment to be accompanied by partners in person. Not every Ph.D. student tangentially using scattering for their materials science, biology or geology investigations, needs to learn how to use a beamline.
The high degree of automation and the lack of required user input means that most experiments can be run well without partners, or even staff present. The addition of a capable experiment orchestrator such as bluesky will even allow adaptive experiments to be run autonomously, and to deal with foreseeable issues. This means we can use our time more efficiently supporting the handful of experiments that actually do require some expert presence.
Sample and materials preparation
For these projects, we would perform near-autonomous measurements of a wide range of well-documented samples and experiments. I’d strive for >80% of mail-in and automatically synthesised samples, including a limited selection of well-prepared in-situ and operando samples. The remaining 20% of samples would be projects that develop new beamline capabilities and would, therefore, demand attendance by on-site users.
Samples not synthesised in our automated materials preparation lab, must be accompanied by detailed metadata. Automatic quality checks on the sample metadata have to be performed to ensure all the information is present for the post-measurement automated data correction, analysis and interpretation steps. Sample metadata can be uploaded via an API by the project partners.
Beamline ramp-up during the initial phase
In the first year of operation, I’d focus on measuring large quantities of static samples in transmission and grazing incidence, to make sure the organizational and data processing pipeline is working robustly.
Gradually, I would introduce optional mapping, and broadening the scope to include a limited range of (highly automatable) in-situ and operando experiments, such as thermal cycling, flow-through and electrochemistry experiments.
The instruments
The beamline should come with a materials preparation laboratory that can be used to automatically generate well-documented, consistent sample series. That is a separate but interlinked project to be run alongside the development of the production beamline.
Naturally, the automation puts some requirements on the instrument itself. Assuming we have a decent optics hutch with a monochromatic, parallel beam coming out of it, I’d add the following to the beamline:
- A meticulously minimized background to dramatically improve sensitivity and reduce the detrimental effect of background subtraction on the data uncertainties. this means all-vacuum flight paths, and, if windows must be introduced, that they be kept small in diameter and thus can remain as thin as possible.
- A (gated, interlocked) vacuum sample chamber can be installed for windowless operation with absolutely zero background.
- five or more nominally scatterless motorised slit sets with encoders
- with a movable channel-cut crystal or orthogonal channel-cut crystals for USAXS between the last two downstream slits (with the last slit set on motor stages inside the vacuum chamber)
- A robot arm as a sample loader for racks, orthogonal long-range magnetic drives for the sample stages for rapid (fly-)mapping and sample switching.
- Our M-sized bases for stages and S-sized bases (Thorlabs KB75/M) for quick-swap sample holders
- Three to four smallish, staggered, static L-shaped detector banks for a wide 2D Q-range
- An additional backscattering fluorescence detector for sample composition validation
- A USAXS analyser crystal and detector between the first (wide-angle) and second detector banks for further Q-range expansion on isotropically scattering samples. It has to be here because of the beam shift by the upstream crystals.
The infrastructure
Being a production beamline, this beamline requires a higher level of infrastructural support for project and data management, processing, simulations, analysis and interpretation/visualisation.
Crucially, a (set of) database(s) is required from the start to meticulously document the projects, samples, measurements, simulations and analyses associated with a project. The measurement database also should be used to build up a library of measurements for the machine learning system to feed from.
The machine learning system should make associations between the current and previous measurements to draw links to similar past experiments, so that suggestions on analyses pathways can be suggested and attempted. Initially necessarily simulation-heavy, this should eventually become a trove of past knowledge to more rapidly exploit the new data. At the same time, we must remain aware that new insights sometimes come from outside the envelope of previously measured samples, and so a fresh human insight should always be applied. As time goes on, this ML system should allow gradual transfer of experience from the beamline staff to a more longer-lasting repository, so less knowledge is lost when staff retires or moves.
There also needs to be better sharing and a tighter integration between the information originating from the project partners and the beamline. This necessitates that the barriers to information entry are eliminated as much as possible. That means that sample provenance, information and metadata must be shared in an unrestricted fashion and uploaded to the fullest extent in an automated manner. This will not happen overnight, but a best-effort start can be implemented using databases like OpenBIS and Scicat already.
Further computational infrastructure is needed for automated data collection, processing and analysis, as well as for visualisation and correlation. This is not a light task, but can grow gradually. Rome was not built in a day either.
Lastly, the data collection should contain sufficient provisions for feedback processing, so that measurements can be optimised as they progress, and a robust fault-handling is in place. An externally facing interface is needed for the partners and beamline staff to easily mark and rank measurements as they come in, which can then also be fed back to the experiment orchestrator to further optimise the running experiment (in case of mapping experiments for example).
The people
Most importantly, this dream beamline should have — and indeed hinges on — having the right people. I would need a diverse team of tech-savvy, modern thinkers with solid experience in automation and organisation. In particular, you need the following (and at minimum two of each to avoid lock-in):
- backend engineers for the database and operational back-end
- Instrument engineers for the design and integration of the automated components
- frontend and API engineers for the interacting side of the software
- A small team of project consultants to run the partner-support side of things
- Technique specialists to ensure the data and analysis quality matches the needs of good science, and who can advocate the beamline capabilities to potential new partners
- An operations manager to keep the team well-funded and replenished with a steady stream of regular, new candidates (avoiding an atrophied team long-term).
- A PRP and SAC who understand the aim of the beamline with continuous, active, bidirectional communication to ensure we’re working together towards the same goal (ideally that being science of unparalleled quality)
- A continuous stream of master’s students, Ph.D. students and post-docs to prepare the next generation for the continuation and expansion of the beamline
The verdict
… and all I’m asking for is 100 million Euros …
I’m a realist, I know this is never going to happen in the current environment. While the technical bits could be gradually developed and introduced, the required organisational and mindset changes might be a step too far. It would require an adjustment of the way the SAC and PRP are integrated in the process, with much more interaction with the beamline staff to achieve a collaborative agreement on the intended direction and a tuning of the projects to match that direction.
From the beamline side, we need a change from measurement facilitator to project partner, more in-depth involvement in the proposal selection process (i.e. not just a technical feasibility assessment), and a shift to automated and automatable workflows. A tighter integration with the infrastructure (IT, machine learning, website, data management, and eScience departments in particular) is required to turn the required information flows into a well-oiled machine.
I suspect we have become too stagnant and risk-averse, too comfortable with the status quo, too resistant to change. The changes needed would need to be radical, approached with a fresh mindset, without too many preconceptions on how things should be. Learn from the past but don’t let it dictate your actions.
Nevertheless, I hope it was a fun read and perhaps it will spark an idea or two. It is not inconceivable that we can, together perhaps, implement one or two of these ideas and make the future a little brighter.