Invoking pharaoh’s name

“For a single device there may be as many as millions of finite elements to model, so to explore the physics of the materials, to learn how the device changes if it is hit with radiation or some other effect, requires extremely high-fidelity, which is what Charon provides,” explains Hoekstra.

With it “we develop a detailed physics understanding of what the radiation effects are,” he said. “We need to go to a lower-fidelity tool when we want to simulate the full electrical system, because we can’t simulate every device in the electrical circuit at the Charon fidelity: it wouldn’t fit on the largest computers. So we take what we learned with the higher-fidelity Charon and use that to inform the lower-fidelity Xyce, which simulates the full system.”

Going Commercial

While addressing DOE’s nuclear weapons verification needs, Hoekstra’s group is also working with commercial vendors to make Xyce technology available to them. “It will give them the ability to do very large-scale simulation of circuits which is needed as integrated circuits get bigger and bigger,” he says. The companies involved are major electrical design and automation tool companies that develop software and manufacturing processes for integrated circuits. They in turn supply semiconductor manufacturers.

“There are two features of our tools that differentiate them from the commercial tools,” said Hoekstra. “One is that we can go to greater fidelity physics in the models. What goes in hand with that is our ability to scale up to the really large scale computing platforms of ASC (Advanced Simulation and Computing). Our code is the only version that is capable of massively parallel execution, which means we can run dramatically larger problems,” he said.

What makes that possible is the supercomputing power made available through ASC, a DOE program created in 1995, aimed at developing supercomputer capability to simulate the performance, safety and reliability of nuclear weapons and to certify their functionality. It involves the collaboration of three DOE laboratories: Sandia, Los Alamos, and Lawrence Livermore, in conjunction with numerous university researchers.

For the higher-fidelity Charon code Hoekstra’s team has run large-scale calculations using Livermore’s Purple, an ASC supercomputer made up of 12,544 processors. Utilizing over 8,000 of these processors, it takes some two weeks for Charon to model 250 million variables. So far the group has used Purple for approximately 12 weeks during the past year. Because Xyce, powerful as it is, does not require such massive computing capacity, it is being run on a few hundred processors at Sandia on high-capacity computing Linux clusters, which themselves can consist of thousands of processors.

Until recently the suite of codes involved in the complex QASPR process were referred to individually by names such as NuGET, Cascade, and GRASP. Xyce and Charon are at the end of this chain of codes, and actually model the responses of electronic devices to hostile environments. To more tightly integrate the codes for effectiveness and efficiency they have now been coupled under a single umbrella term standing for Radiation Analysis Modeling and Simulation for Electrical Systems, or RAMSES.

Pushing the Limits

When QASPR will be ready for “real world” electronic component qualification is in large measure dictated by the needs of the Electrical and Microsystems Modeling group’s customers — namely the military, who specify the requirements for refurbished nuclear weapons systems, and the DOE scientists and engineers who design the new components. Hoekstra expects that the software to model the devices’ durability will be ready when the components themselves are ready for testing.

In the meantime, much needs to be done. “We have to improve our understanding of the physics and our ability to model that physics,” said Hoekstra. For example, he noted, “The radiation effects are only partially understood, so we are trying to increase our knowledge of those effects and get that knowledge into the models. We are doing tightly coupled experiments and modeling to understand the physics.”

Beyond that there is the issue of scale. “We are really pushing the limits of computing to model these very high-fidelity models. We need to improve our models and increase our computing horsepower,” he said.

Another challenge is to convince those responsible for nuclear weapons qualification that they can rely on the QASPR methodology to accurately depict the response of electronic components to hostile environments. Doing so requires a two-step process known as verification and validation. Verification is more mathematical than empirical in that it verifies that the computer model gives the answer it is supposed to give in mathematical terms.

Validation answers the question, “Do the results represent reality?” To validate RAMSES results, physical tests are run on electronic components in radiation facilities other than the now defunct Sandia Pulsed Reactor. Though the conditions they create are not as close to an actual hostile environment as was achievable with the SPR, “We still have facilities that give us gamma and neutron irradiation. We put the devices and circuits into those test facilities and use the results to validate our models,” says Hoekstra.

Two DOE Computational Science Graduate Fellows (CSGF) have been involved in the development of the suite of codes used by the QASPR group. Judith Hill, who did her DOE CSGF practicum at Sandia in 2000, worked on Charon when she returned to Sandia as a postdoctoral fellow. Similarly, David Ropp, who did his DOE CSGF practicum at Los Alamos in 1993, contributed to the ASC program while a postdoctoral fellow at Sandia from 2000 to 2004. During that period Ropp worked on Trilinos, a package of algorithms designed to run on the large-scale ASC computers such as Purple.

For its work, the Xyce team recently won a prestigious 2008 R&D 100 Award, given by R&D magazine. The magazines’ Web site states that the “Award provides a mark of excellence known to industry, government and academia as proof that the product is one of the most innovative ideas of the year.”

This article originally appeared in Deixis: The CSGF Annual, 2008-09.