SiCortex Newsletter Articles

From www.xconomy.com — Peddle Power: MIT Cyclocross Team Promotes Alternative Energy, Low-Power Computing

Fri, 29 Feb 2008 15:24:26 GMT

....the bikes were used to power machines made by SiCortex, of Maynard, MA, a venture-funded startup (investors include Flagship Ventures, Polaris Venture Partners, and Prism VentureWorks, along with Chevron and JK&B Capital) that specializes in low-powered supercomputers. To give you an idea of how low-powered, CEO John Mucci says the chip in his supercomputer, with six processors, uses about eight watts of power. The chip in my laptop, he told me, takes almost 100 watts. Ouch.

Read the full story at http://www.xconomy.com/2007/12/11/peddle-power-mit-
cyclocross-team-promotes-alternative-energy-low-power-computing/

Five Ways To Reduce the Footprint Of An HPC Application

Fri, 29 Feb 2008 15:18:37 GMT

0. Minimize Runtime (duh)

No HPC user needs to be reminded to do this. The thirst for faster runs is insatiable, and developers work hard to make their codes run faster. There remain, however, options at runtime. Computing to a higher-than-needed accuracy, for example, increases runtime, perhaps to the third power, and hence increases footprint.

1. Eschew Peak

It has long been noted that “peak is meaningless,” in that knowing the peak speed of a processor tells little about the sustained performance that an application will actually get. Peak is, however, a meaningful indicator of the amount of power that will be consumed during a program run. It is the sustained performance as a percent of peak that determines the footprint of an application. An application that runs at twice the percentage of peak on processors with half the peak has half the footprint. Or maybe a quarter the footprint, since power goes up more than linearly with clock speed.

2. Just Communicate

For the past ten years, Linux/MPI applications have been headed in the opposite direction, accepting additional arithmetic to avoid doing communications. A simple example: many applications recompute known values from scratch to avoid doing a lookup in a large table that requires communications. SiCortex systems can send individual values from tables as big as a terabyte in time comparable to a floating point operation. Applications that look up values have a correspondingly smaller footprint.

3. Practice Spin Control

Modern HPC applications deal with enormous amounts of data that must be shared amongst large numbers of processors. Traditionally, this has meant large arrays of disk drives, often hundreds of them. Keeping all those disks spinning is the worst of all worlds: low access times and high energy use. Clusters with large amounts of main memory and the right access methods can keep all that data cached, reducing time-to-completion and lowering the disk energy footprint to boot.

4. Keep the Offense On the Field

When an application is moving toward completion, it is playing offense. When it stops to do a checkpoint to protect itself against unreliable hardware, it is playing defense, and bloating its footprint.
Perversely, the more energy a processor chip uses, the hotter and therefore the less reliable it is, forcing more frequent checkpoints. It is not unheard of for applications to spend half their run time checkpointing the other half.
Applications that run on inherently reliable hardware can potentially halve their footprint, and their time-to-completion, by keeping the offense on the field the whole game.

The Payoff for Petaflops

Mon, 23 Jul 2007 19:58:19 GMT

Now comes the petaflops, and with it the high processor counts that make for a radically different computing milieu. Algorithms can no longer depend on their author’s knowing what individual processors are doing; there are simply too many of them. Adaptive approaches, ones that figure out on the fly how best to organize the work, are sure to become more important, as are algorithms inspired by biology. These algorithms, however, need extensive testing on a wide range of data sets to confirm that they perform appropriately.
Designers of these new approaches need a cost-effective and highly-transparent computing resource in which to develop their codes. The SiCortex combination of low-cost processors and a rich library of monitoring software makes it the natural birthplace for this new generation of petaflops software.

The Journey of a Thousand Teraflops

Mon, 23 Jul 2007 15:29:40 GMT

The coming generation of petaflops systems will strain this dichotomy to the breaking point. With their unprecedented processor counts, petaflops systems cry out for new applications approaches, but, with their even higher price tags, they price out the experimentation that is the path to those new approaches.
SiCortex can help. The SC5832, with its thousands of $300 Linux processors and enormous bisection bandwidth, is the ideal, affordable, launching pad for new peta-applications. Just as important as its high processor count is its rich array of open-source development tools and performance monitors.
SiCortex support for petaflops software development starts with pervasive instrumentation within each node chip. Each processor core is capable of recording its own instruction execution behavior, including cache hits/misses and stalls. At the node level, each chip monitors its shared L2 cache, PCI I/O transactions, and interprocessor communications. (For more details, see the Late Winter 2007 edition of 5832, available at www.SiCortex.com.) This hardware-level data is then made available to the SiCortex suite of development tools through the widely used PAPI subsystem.

The SiCortex tools strategy combines best-of-breed open-source tools with full interoperability. The tools work on unmodified codes and provide rapid characterization of hardware utilization, memory, I/O and communications usages, and thread/task load balancing. Advanced users can have full access to the performance monitoring hardware through pfmon, a tool that communicates directly with the Perfmon2 kernel subsystem.
Perfmon2, Libpfm, and PAPI: While not typically used directly by applications developers, Perfmon2, Libpfm, and PAPI provide a consistent interface to the SiCortex hardware counters. They offer first- and third-person semantics for thread-centric counting and sampling.

The SiCortex “-ex” Interface Architecture: In order to make the performance monitoring experience as accessible as possible, SiCortex has designed a consistent set of commands that layer on top of standard tools.
Papiex/PAPI: Papiex is used to provide summary information such as memory footprint, percent of time in I/O, and percent of time in MPI. A typical Papiex run will produce upwards of 30 top-level run statistics. In short, Papiex derives meaningful statistics from the wealth of performance data that the node chip provides, giving a high-level view of how time is being spent within the processors.

Mpiex/mpiP: Mpiex utilizes the LLNL mpiP package to characterize MPI load balance, MPI function profile, message size distribution, and call site information.
Ioex: Ioex, based on concepts from IOtrack, developed at PDC/KTH, characterizes the I/O behavior and performance of a high-processor count application.

Hpcex/HPCToolkit: Hpcex, based on the HPCtoolkit from Rice University, produces statistical profiles without the need for user-coded instrumentation. It can profile by load module, file, function, line, and even instruction.

Gptlex/GPTL: Gptlex controls the behavior of GPTL, developed at NCAR, and adds support for automatic compiler instrumentation in GCC and Fortran on SiCortex systems.

Tauex/TAU: Tauex provides a consistent Interface to TAU, the widely-adopted parallel performance profiling environment from the University of Oregon and Paratools, Inc. It supports parallel profiling, tracing, and high-level 2D and 3D visualization. (Detailed information about TAU is available at: www.cs.uoregon.edu/research/tau/home.php).

Vampir: Vampir is a powerful visualization tool for temporal performance data that scales to trace data volumes in excess of 40 GBytes. (Detailed information about Vampir is available at: www.vampir.eu)

The Payoff For Applications

Mon, 23 Jul 2007 12:32:39 GMT

Users with extreme performance requirements on dedicated applications can go further: investigating the possibilities for extending the standard SiCortex MPI library to provide more targeted support. For example, the ability of each DMA Engine to inject commands for execution by other DMA Engines is a natural fit for functions that many applications now carry out laboriously in user software.

Collective Wisdom

This newsletter has focused on the DMA Engine's role in node-to-node data transfers. As you might expect, it also has extensive facilities for accelerating so-called "collective" operations that involve large numbers of processors. Fast collectives are essential for high processor count computing. A future edition of the newsletter will discuss collectives in detail.

The SiCortex DMA Engines: Almost Like a Cluster Within a Cluster

Fri, 20 Jul 2007 20:19:29 GMT

The SiCortex DMA Engine

Each SiCortex node chip includes, in addition to its six 64-bit Linux cores, a DMA Engine that is capable of:

Cooperating with other DMA Engines to transfer data from one processor's virtual memory to another's.
Sending small amounts of data with very low latency and no operating system involvement.
Establishing connections and then sending or receiving large amounts of data without any application intervention (see diagram).
Doing all of this for multiple execution threads, and hence for multiple ongoing transfers, at a time, with separate state maintained for each.

In order to do all this, the DMA Engine executes its own microcoded instruction set that has been tuned on behalf of the MPI Library. This instruction set is quite general-purpose, and features the ability to queue operations on other DMA Engines. Because each DMA Engine executes its own instructions and can invoke the help of other DMA Engines, they really do act like a cluster inside of a cluster.

The DMA Engine Command Set

Three DMA Engine instructions are used to do the heavy MPI lifting. They are:

Send Event: Immediately transmit a packet, with up to 112 bytes of data, to another node's DMA engine, where it is available to the destination user-mode program.

Send Command: Transmit a command to a destination node where it will be executed by that node's DMA Engine.

Put Buffer: Send a sequence of packets to a destination node according to parameters contained within the DMA command. Put buffer is a zero-copy operation.

Notice that no separate "Get" command is needed. A DMA engine that wants data gets it by queuing a "Put" command in the DMA Engine of the node that has it.

The Payoff For Applications

Mon, 14 May 2007 21:52:13 GMT

In the early days of supercomputing there was a very direct and beneficial link of understanding between computational scientists and their hardware. The scientists knew their vector pipes and architects like Seymour Cray knew their inner loops. Each could help the other to maximize the answers per day from the machines.

One of the reasons that the efficiency of today’s scientific codes is so low (often a few per cent or less) is that this linkage of understanding has been broken. Chip designers do not understand floating point codes because that is not what they design for. Scientists have lost track of the innards of complex out-of-order nodes made up of chips from multiple vendors.

SiCortex is working to make its hardware more transparent and accessible to the scientists who rely on it. Where we can, we are eliminating the need to worry about what the hardware is doing. The Kautz graph fabric of the SC5832 and SC648, for example, has such a flat performance curve that there is rarely any reason to lay out data in any but the most natural way.

Where the behavior of the hardware does matter to the application, SiCortex is making it easy to monitor the inner workings, not just of the processors, but of the whole system. Often the true performance issues lurk in places other than the apparent ones.

In the past 10 years the transparency of software such as Linux has accelerated user involvement and interaction. Now SiCortex is expanding that transparency to hardware, making it easier to develop applications that run efficiently on the hardware, and easier to develop hardware that fits those applications.

"The unexamined program is not worth running."

Mon, 14 May 2007 21:46:09 GMT

Each node within a SiCortex system has its own central monitoring circuitry which can selectively sample more than 1200 activities around the chip. Because the data comes together in one place, it is possible to learn about dependencies as well as about individual event counts.

(Click the image above for more information.)

The Payoff For Applications

Mon, 14 May 2007 21:37:37 GMT

Three characteristics become obvious as you experience the Kautz graph visualizations:

1. Performance remains very high even as the number of nodes approaches a thousand.

2. The “response curve” of the network is essentially flat, meaning that the time it takes to get to the most distant node is almost the same as the time to the closest. (Each hop only costs 30ns.)

3. The Kautz graph’s enormous bisection bandwidth allows it to power through collective operations such as broadcast without degradation.

All three of these characteristics are essential to the high processor count applications that will dominate technical computing in the years to come.

The first key to increasing the performance of any Linux/MPI application is to run it on more processors, provided the network scales cost-effectively. The SiCortex Kautz graph achieves this goal by integrating the switching within processor nodes and putting the wires in a backplane.

A second key to increasing performance is often to use more sophisticated, and perhaps adaptive, data structures. That is why so many mainstream applications are now moving from fixed grids to dynamic ones. The flat response curve of the Kautz network assures that performance remains high even as the data moves around unpredictably in memory.

And it is well known among computational scientists that the performance of collectives is often the limiting factor in scalability. The Kautz graph addresses this issue by offering enormous bisection bandwidth and supporting microcode-accelerated collectives.

Ultimately, the payoff for applications is that you can henceforth forget about your SiCortex system’s Kautz graph and implement your application in the way that best fits the algorithm. The Kautz graph will perform well regardless.

Just Plane Fast

Mon, 14 May 2007 21:34:32 GMT

A small Kautz graph is easy to understand. Each node in a degree three graph has three wires out and three wires back in. There are obvious symmetries, albeit not the traditional ones. For example, the path from node A to node B is not the same as the path from node B back to node A. It is known that the number of nodes in a Kautz graph is exponential in the diameter, but for a true understanding, it helps enormously to be able to see what is going on.

Suggestions for further reading: Bermond, J.-C., and Peyrat, C., “de Bruijn and Kautz Networks: a competitor for the hypercube?” in Andre, F., and Verjus, J.P., eds., Hypercube and Distributed Computers, North Holland, 1989. Banerjee, Subrata, et al, “Regular Multihop Logical Topologies for Lightwave Networks”, in IEEE Communications Surveys, Vol 2, No 1, First Quarter 1999.

Now It Is Your Turn

The SiCortex SC5832 and SC648 are the first clusters to be designed from a clean sheet of silicon, but they could not have been designed on a clean sheet of paper. In a system with thousands of interconnected processors, there is too much going on. You can see that for yourself using the Kautz Graph Viewer on the accompanying CD. Take it all the way up to 972 nodes to see how quickly a Kautz graph can get from anywhere to anywhere. Click on any node to start a broadcast and see how fast it can get from anywhere to everywhere. Treat it as a one-dimensional row of processors or a two-dimensional box. The ability to visualize the backplane topology from a variety of points of view was crucial to the overall systems design, which involved integrating the behavior of the Kautz graph with the behavior of the nodes and the applications software running on those nodes. It was the partnership of experienced architect and computer visualization that made it possible to take this significant step forward in network design. Of course, when it came time to work out the physical wiring of the backplane, the architects had to stand aside. They could not compete with the efficiency of the computer’s genetic algorithm.

The Payoff for Applications

Mon, 14 May 2007 21:21:15 GMT

Every HPC application faces the temptation to employ heroic measures in order to eke out performance. A common example is the use of local disks on conventional cluster nodes. Used in parallel, they offer the potential of very high aggregate bandwidth, but at the expense of specialized programming and uncertain reliability.

FabriCache is better in every way. It is faster, because it operates at memory speed. It is more robust, because it takes advantage of the full error correction facilities built into Lustre. And it is simpler, because a FabriCache file is no different from any other file that the application uses. A simple way to test for disk file performance issues is to temporarily shift a file into FabriCache and see how overall applications performance changes.

SiCortex is committed to architecture that maximizes performance while minimizing special cases. FabriCache is just one example.

The SiCortex FabriCache™: Measure Its Abilities In Genomes/sec.

Fri, 11 May 2007 16:36:05 GMT

Storage systems and processing platforms have similar histories. Both started as single large monoliths, but both have moved strongly to parallelism to take advantage of the volume economies introduced by the personal computer industry. Large attached storage systems now have thousands of individual spindles; Top500 computing platforms increasingly have thousands of individual processors. The challenge for system architects, of course, is to maximize the parallelism between those thousands of spindles and those thousands of processors, whilst assuring reliability when one of either fails. Winning solutions maximize total transfer bandwidth while minimizing software complexity. Several successful approaches now exist, including Lustre, PVFS, and GPFS among others, but until now none of them could extend their reach to the other system resource that is present in the thousands: DIMMs. By reflecting the power of the Lustre parallel file system back onto terabytes of its own main memory, the SiCortex SC5832 FabriCache provides many of the benefits of a global shared memory without the enormous cost.

Parallel File Architecture

Since the SiCortex FabriCache takes advantage of the Lustre parallel file system, a few words about Lustre are in order. Produced by Cluster File Systems, Lustre has established itself as a standard in high performance computing, used by Cray and HP among others.

Like any parallel file system, Lustre starts by spreading, or “striping”, individual files across as many disk spindles as possible. Having a file spread across many disks allows it to be read back into an individual processor very fast. More importantly, it allows many processors to be getting at pieces of the same file simultaneously. For the physical scientist, parallel file systems allow all the processors in the system to be writing updated information about their individual grid points to the appropriate places in a single file at once. For the life scientist, parallel file systems allow every processor to be reading from single copies of genomes as they look for patterns of similarity.

Traditional parallel file systems coordinate transfers by means of extra processors added to the network. Instead of going straight to the disk spindles themselves, the cluster nodes send their read/write requests to these intermediate processors (Lustre calls them “Object Storage Servers”, or OSSs) which then do the actual disk transfers and pass the data back and forth to node memory. The Lustre OSSs have an additional function: assuring the integrity of the data. The OSSs deny write access to portions of a file that are already being written by another node and suspend read access to portions that are about to be written. In addition to the OSSs that coordinate transfers between nodes and established files, a Metadata Server node manages the efficient creation and deletion of files. If thousands of nodes simultaneously ask for a new temporary file apiece, it is the job of the Metadata Server to assure that they do not have to wait in line to get them.

Lustre on SiCortex is different because it is “inboard” rather than “outboard.” Instead of being an external file system in addition to the root file system used by the processing node kernels, SiCortex Lustre is the file system for the whole configuration. And rather than running on exogenous processors, SiCortex Linux uses SC5832 and SC1458 nodes directly as its OSSs. Most critically, instead of communicating with OSSs over an ethernet or similar enabling network, SiCortex uses its very high bandwidth Kautz graph.

Each SiCortex circuit module implements 27 nodes. (A SC5832 has 36 of these circuit boards; a SC1458 has nine.) Of the 27 nodes, three are linked to PCI EXPRESS® slots to which disk spindles can be interfaced. Each, or all, of these three can be Lustre OSSs, either on a dedicated basis or in addition to running applications.

Implementing FabriCache Via Lustre

On SiCortex systems, every node knows how to act as an OSS. The reason that only three of 27 act as OSSs to spindles is that they are the ones that have PCI EXPRESS slots. All 27, however, are connected to their own DIMMs, so all 27 can act as OSSs to portions of that memory set aside for FabriCache.

FabriCaches can be established in either of two ways. For intense interaction, a subset of SiCortex nodes can have all their available memory dedicated to FabriCache usage. The remaining compute nodes communicate with them on the Kautz graph fabric. A more interesting case is to set aside a portion of every node’s memory for FabriCache. Now all the nodes (972 for the SC5832, 243 for the SC1458) are part-time OSSs.

(Click the image above for more information.)

The power of FabriCache is that, either way, compute nodes address it as just another file with a single address space, because that is what it is. The Lustre logic that is already in every node’s kernel has the responsibility of determining where in the file a read or write goes, and which OSS is responsible for that portion of that file. It sends the request over the fabric to that OSS node, and exchanges the data the same way. If the data happens to be in the node’s own portion of the FabriCache file, it acts as its own OSS and the operation collapses to a memory copy. Of course, if another node has reserved that part of the file for writing, the operation may be held up for coherence reasons. In effect, the node acting as Lustre OSS pulls rank on the file acting as compute resource in order to assure integrity of the data in the FabriCache. The key point is that the node acting as FabriCache OSS is transparent to the node acting as compute resource. No special code is needed to access the data. It is just a file. And no special code is needed to get extreme performance. Lustre does that automatically.

(Click the image above for more information.)