Testing, testing, testing: How researchers make sure the LSST Camera is the best it can be

Building the world's largest digital camera ever made for astronomy, the Vera C. Rubin Observatory's Legacy Survey of Space and Time Camera, is no simple task – that much is obvious.

The camera features a 3,200 megapixel sensor array, some of the largest lenses ever built, and complex electronics meant to take an ocean of astrophysical data off the camera and send it out into the world.

What may be less obvious is how much work goes into making sure the camera, built at the Department of Energy's SLAC National Accelerator Laboratory, works. The system was, after all, custom designed to peer wider and deeper into our universe than any camera has before And, in the process, drive the effort to understand dark matter and dark energy. Turning such ambitious plans and designs into realities is going to come with some trial and error and a whole lot of calibration and testing.

Here, three members of the team responsible for all that calibration and testing talk about what's gone into making the LSST Camera the best it can be.

Making stunning images even better

One of the central challenges, says LSST Camera scientist Yousuke Utsumi, is "converting images to scientific knowledge." After all, the camera is not designed just to take pretty pictures, but rather to create a precise map of the universe, and that requires taking detailed, accurate images of distant galaxies. "We want to measure galaxies precisely to understand the nature of dark matter."

Doing so requires more than just specially designed lenses and sensors, Utsumi says, because no matter how well designed and built those components are, there will be imperfections. For instance, consider an image taken by an ordinary camera: There will always be some distortions in shape and color near the edges. There will also be slight distortions in the digital sensors as well, and similar effects will hold true for the LSST Camera. "We need to understand what's going on there so we can correct for it."

Utsumi and his team took thousands of images over three months with the LSST Camera sensors of all sorts of shapes and patterns. They then compared the camera's images with the originals to understand how to correct for any distortions or errors. The team has also worked on how to correct other issues, such as the fact that brighter objects appear larger than they actually are, as well as "ghosts," or images of an object that appear because of electronic crosstalk between sensors within the camera.

"We know a lot about the camera now, so it will be exciting to see how it works on the telescope," Utsumi says.

Building a more foolproof camera

Although Utsumi's work is central to making the camera work as best as it can, the sensors and lenses are only two sets of  components in a camera the size of a small SUV. The camera has a refrigeration and vacuum system, multiple on-board computers and an array of other electronics that monitor and control the camera's operation.

Stuart Marshall, the LSST Camera's operations physicist, is in charge of making sure all of those systems function properly. "Once everything is working correctly, we can sit there taking data, and there's a small army of people to look at what comes out and do science," he says. "I've concentrated on making sure that everything works to make that happen."

Getting there means a lot of behind-the-scenes work on the camera infrastructure. "If you work backward from the sensors, for them to work, they have to be cold. They have to be at minus 100 degrees Celsius, or -148 degrees Fahrenheit, and you can't be at minus 100 degrees unless you're in a vacuum, and we have to have power and communication and the data has to flow."

At this point, that means a lot of testing and, in case something is wrong, trying out different ideas to identify the cause of a problem and find a solution. For instance, Marshall says, he's spent a lot of time in the last year updating the vacuum system to improve its reliability. As a result, the camera team has changed some valves and updated software to make the system more foolproof. "If you're on top of a mountain at 9,000 feet in the middle of the telescope dome, it's easier to make a mistake ," since there's less oxygen at altitude and more things moving around compared to the clean room at SLAC, Marshall says. "So we're trying to make sure the system can catch errors before any damage is done. There's an awful lot of that built into the whole camera system." 

Preparing camera controls for crunch time

A perhaps subtler challenge, says senior scientist Tony Johnson, is making sure all the camera software is working as well as it can. Johnson works on the camera control software, which turns it on and off, reacts to abnormal conditions, adjusts camera parameters as needed and shuts it down if something goes particularly  wrong. He also works with the data acquisition system, which takes data off the camera sensors and sends it out into the world.

"At this stage, everything is mostly finished, but myriad things can be improved," Johnson says. "For instance, can we reliably write an image from the data acquisition system within two seconds every single time, or does it sometimes take a bit longer, and sometimes that causes a problem?"

So, Johnson says, he and his team work to track down issues like those, which may involve software or hardware, and make sure all the pieces work together as expected.

Another issue Johnson works on: Making sure the camera will work as expected once it's made the journey to Chile, where it will sit atop the Simonyi Survey Telescope at Rubin Observatory and begin its work.

"One aspect of this is the camera has been built by a fairly small group of people, and there are a fairly small group of people who are expert in each part of the camera," Johnson says. "What we need to transition to the specialists who will operate the observatory day and night, so we have to do  a fair amount of knowledge transfer." Partly that's a matter of documentation, but it also means working with the scientists in Chile to identify potential problems, continue to improve software, and generally make the system more reliable.

"It's a challenge, but most of the time it's an exciting challenge," Johnson says. "I think most of us who are building the camera are not just building it because we like building hardware or we like building software, although we may do those things. We're building it because we see the end goal of getting new science out of it."

For questions or comments, contact the SLAC Office of Communications at communications@slac.stanford.edu.