Filling in the blanks

A typical simulation run on today’s high-performance computers, such as Hopper, the National Energy Research Scientific Computing Center’s 1.3-petaflops Cray XE6, analyzes the behavior of 69 billion particles in three dimensions and generates about 20 terabytes of data per time step, Nugent says. A reasonably large data storage allocation at the computing center is about 100 terabytes, so at most it’s possible to read out five time steps.

Weber uses topological analysis, a technique that will initially help reduce the amount of data written to disk and will eventually help scientists determine where and when interesting behavior is occurring in the simulation. The embedded topology analysis studies the data in real time so that  seemingly separate and distinct behavior can be understood as connected, and therefore is analyzed as such.

With embedded topology analytics, Nugent and the research team hope to run about 100 time steps and then read out a summary that highlights data of interest. The analytics are expected to let the team compress 20 terabytes to a few megabytes.

Currently, the simulation software can be instructed to concentrate resolution on interesting areas of space where dense clusters of matter are forming, based on a rather simple density measurement. To perform a more sophisticated analysis at a higher level of physics, Weber and Nugent are designing the embedded analytics to perform without a research team intervening. Nugent says the embedded analytics will not only summarize terabytes of data but also guide the simulation.

The first step – fully embedding the analytics into NYX – is nearly finished, Weber says. “By being forced to integrate the data analysis into the simulations, these results become available during the simulation, and there are very interesting possibilities to use the results to affect the calculations. We are more than 90 percent there.”

Besides selecting time points for analysis, the Berkeley Lab team – which includes David Trebotich, a computational scientist in the applied numerical algorithms group; Daniel Ushizima, a research scientist studying carbon sequestration; and post-doctoral scientist Dmitriy Morozov – is studying whether analytics can  annotate data sets. The goal is to set up simulations that calculate full physics on only a subset of the data, allowing the analytics to substitute a representation for some of the forces.

For example, in carbon sequestration simulations, researchers would like to simulate movement of gas through rock at a small-pore scale. But to look more realistically than that requires simulation of various sizes of pores and voids. The analytics and visualization allow extraction and visualization of representative summary data in the form of a Reeb graph, a three-dimensional, symbolic mathematical representation that doesn’t require a full download of the entire simulation.

“We’ve been talking about using this Reeb graph to extract information at the small scale and then using that result in another simulation that looks at larger-scale behavior,” Weber says. The detailed simulation would then provide the summary information of the full physics model at the pore level. In turn, that summary information would serve as a stand-in for the next run, making it possible to do simulations on a larger scale.

Both the carbon sequestration code and the NYX code are designed to be portable to the new processing, memory and interconnect technologies exascale computing requires. Indeed, NYX is an extension of the BoxLib framework that is being explicitly designed to port to exascale computing.

Says Nugent, “We are hopeful that in five to eight years’ time, we will be running on exascale machines.”