In astrophysics, dark matter is akin to the idiomatic 800-pound gorilla – a dominating influence that throws its weight around, dictating how stars and galaxies move. But unlike gorillas, dark matter is invisible and its nature is elusive. Physicists can only surmise its existence from its effects on visible matter.
“If you didn’t have dark matter the stars would not be able to move at the velocity they’re moving at. They would fly apart,” says Michael Kuhlen, a postdoctoral researcher at the University of California, Berkeley. Physicists theorize that “halos” of invisible matter host galaxies, providing the gravity that holds them together.
Kuhlen is part of a research team using Jaguar, Oak Ridge National Laboratory’s world-leading Cray XT computer, to help understand dark matter distribution. Their Via Lactea II model first gained attention when it elucidated the “lumpy” nature of the dark matter halo enfolding a galaxy like the Milky Way.
Via Lactea II was made possible with a grant of 1.5 million processor hours on Jaguar, provided through INCITE, the Innovative and Novel Computational Impact on Theory and Experiment program, supported by the Department of Energy’s Office of Science. The researchers, led by Piero Madau of the University of California, Santa Cruz (UCSC), now have a 5 million processor-hour INCITE grant to run their next, more detailed dark matter simulation.
There’s a lot to detail. As much as 83 percent of the matter comprising the universe is dark matter, physicists say. Researchers have offered several theories describing it, but the one gaining the most acceptance casts the mysterious material as a low-temperature fundamental particle that interacts only weakly with ordinary matter – except through gravity.
Via Lactea II is based on this cold dark matter representation. Although its name is Latin for Milky Way, it’s not designed to precisely simulate evolution of our galaxy but of one similar to it, Kuhlen says. “By similar, we mean it has the right mass, the right rotation curve and roughly the correct accretion history” – the process that formed the galaxy.
The latest INCITE grant will enable more precise simulations. Instead of resolving structures on a scale of hundreds of parsecs (a parsec is equal to about 3.26 light-years, or about 19 trillion miles), the group’s model could have a resolution in the tens of parsecs. It’s like switching from a microscope capable of enlarging objects by 10 times to one capable of enlarging objects by 100 times.
“We could actually resolve scales in these dark matter halos that also are accessible to observational astronomy. We can get a correlation between what observers see and what we can predict,” potentially providing further data on dark matter’s nature, Kuhlen says.
For instance, astronomers measure the motion of stars at the center of nearby dwarf galaxies to find clues about the dark matter holding them together. By increasing their simulation’s level of detail, the Via Lactea researchers could give astronomers predictions to compare against their observations or could suggest what to look for.
Kuhlen and others in the group also seek ways to include the effects of visible matter in their simulation. “Up to this point we have taken the approach to ignore all the baryonic physics – all the normal matter. That will only work up to a certain point. As you increase resolution further and further you get into regimes where it’s important.”
This will be Via Lactea’s third iteration. The first came to life in spring 2006 as an application to test the then-new Columbia supercomputer at NASA’s Ames Research Center in California. “Since that worked and immediately produced interesting results, we tried to push it on higher and higher” in resolution, says Kuhlen, who at the time was a UCSC doctoral student under Madau.
The results helped earn the team its first INCITE grant. In November 2007, PKDGRAV2, the code that powers Via Lactea II, went through more than 1 million hours on 3,000 processor cores.
The ease with which the code ran on Jaguar is a tribute to Oak Ridge’s system, says Doug Potter, a member of the research team and a postdoctoral researcher at the University of Zurich’s Institute for Theoretical Physics. He and others on the team attended a workshop at the lab on how to use the computer effectively. After that, “It was a simple matter to get our codes running. Jaguar is set up and managed so well there wasn’t really any need for technical support.”
Tracking each dark matter particle obviously is impossible, even with the most powerful computer. So to model evolution of the dark matter halo around a galaxy similar to the Milky Way, Via Lactea II followed the interactions of about 1.1 billion particles, each with mass equivalent to about 4,000 suns.
The simulation began 20 million years after the Big Bang and followed the particles –with a combined mass of 1.7 trillion suns, representing the total dark matter in the system – over about 13.7 billion years. Gradually gravity draws the particles together from across a vast expanse of space, pulling them into small lumps that merge to become successively bigger lumps and clumps. Eventually the clumps come together in a “halo” massive enough to provide the gravity holding visible matter together in a galaxy.
In a Nature paper published in August 2008, the researchers said Via Lactea II showed that small dark matter lumps retain their structure even after merging into bigger clumps. Dark matter may be like Russian matryoshka dolls, they said, with small shapes inside successively bigger but similar shapes.
“We have high enough resolution that we can look at the internal structure of the lumps and they have little lumps inside them – substructures and sub-sub-structures and sub-sub-sub-structures, like nested dolls,” Kuhlen says. The simulations also show streams of dark matter flowing through the galactic halo.
The model indicates that hundreds of concentrated dark matter clumps survive even near our solar system. Earlier simulations with coarser resolution – larger representative particles – portrayed dark matter in this area as smooth.
Via Lactea and other simulations are helping make the case for the cold dark matter model, Kuhlen says. If dark matter was hotter, its thermal velocity would make for a smooth structure, rather than clumpy. Apart from that, Via Lactea says little about just what kind of particles comprise dark matter, because it’s designed to resolve dark matter’s gravitational effects.
“We’re concerned with how dark matter is laid out on a larger scale than one in which individual particle physics would matter. At the same time, it doesn’t mean (the simulation is) irrelevant for experiments that attempt to directly detect dark matter particles.”
In fact, the group’s findings could help researchers understand experiments that attempt to directly detect rare collisions between dark matter particles and heavy atomic nuclei, like iodine, xenon or tungsten.
Via Lactea II produced a huge amount of data: about 20 terabytes (20 trillion bytes) – enough to fill about 100 average-sized personal computer hard drives. That’s tiny, however, compared to the 500 terabytes expected to come out of the next version, tentatively named Silver River, the East Asian nickname for the Milky Way.
Instead of dark matter particles about 4,000 solar masses big, researchers plan to track particles just 100 solar masses in size, Kuhlen says. “It’s like skipping a generation. It’s also why it’s causing us trouble. It’s a very big jump.”
The researchers have struggled with establishing the simulation’s initial conditions – the parameters for its starting point. The obstacle kept the group from using the bulk of a 2009 INCITE grant of 5 million processor-hours on Jaguar. The grant has been extended through 2010.
Led by Jürg Diemand, Joachim Stadel and Potter, the team is continuing to develop PKDGRAV2 and tackle the initial conditions problem. “The sheer size and high resolution of the initial conditions presented us with several new problems,” Stadel says.
It’s tricky, because the researchers must zoom in on just a tiny region of the entire universe – an area producing a dark matter halo the size of the Milky Way, Stadel says. The simulation must take into account gravitational fields from around the universe but focuses most of its computing power on the halo itself, Stadel says.
The net result is a simulation that starts with about 10 billion dark-matter-halo particles, adds another 20 billion in the area immediately outside the halo and then throws in about 15 billion more massive, lower-resolution particles. Tracking such a huge number of particles as they interact over billions of years obviously is an enormous computational challenge.
Computing initial conditions, Stadel says, taxed the available computing power and memory and slowed communications between processors. “All of these problems were present to a lesser extent in earlier simulations, but were just below the level of being recognized as potentially serious problems for a simulation with 10 times the resolution.” The researchers have resolved the problems, however, and have run a series of lower-resolution simulations. They’re checking to see if they’re correct when evolved from the early universe to the present, Stadel says.
Many of the INCITE researchers also collaborated with the Zurich group on GHALO, a galactic dark matter simulation that tracked 2.1 billion particles of 1,000 solar masses each. Other researchers on the project include Ben Moore, director of the Institute for Theoretical Physics, and Marcel Zemp of the University of Michigan.
Diemand, Moore, Stadel, Potter and other researchers on the team specialize in computational astrophysics and related areas. Still others focus on astrophysics, astronomy and cosmology. The group also has collaborated with researchers working in particle physics and other subjects.
It’s invigorating, Kuhlen says. “We find ourselves at an intersection of a lot of different fields,” with experiments in gamma ray and particle detection and bigger and better computer simulations reaching fruition. “It’s a really interesting time because there are advances in physics and computers all coming together.”