SiCortex Knowledge Base

SC648 Update

Wed, 17 Sep 2008 18:26:05 GMT

We are pleased to inform you that we have replaced the SC648 with an equivalently priced SC1458, packaged in the same cabinet. As with the SC648, the SC1458 can now be configured with 2, 3 or 4 modules – and offers the ability to easily accommodate 9 modules.

Note that the SiCortex software will continue to support SC648 systems.

Software Release Notes Version 3.0 Field Test

Wed, 17 Sep 2008 17:34:02 GMT

This document describes, in this order:
• New Features and Other Changes
• Active Issues and their temporary workarounds
• Fixed Issues
• Tips and Tricks

Fix for ECC bug

Tue, 16 Sep 2008 13:23:10 GMT

During our Field Test, we have discovered an issue with the kernel code concerning Error Detection And Correction which can cause nodes to fail by reporting incorrect uncorrectable ECC errors. This is more often found on larger systems (due to the greater number of nodes), but can also affect the SC072-PDS.

To fix this error, download this file, and unpack it and install it according to the following directions:
# cd /opt/sicortex/kernel/linux
# tar xzf 5.0.0.88.60314-r5-all.tgz
# rm default
# ln -s 5.0.0.88.60314-r5 default

Installing the V3.0 Field Test Software

Thu, 11 Sep 2008 14:47:03 GMT

For instructions click SC072 V3.0-FT Software Installation Instructions.

Node NTP Synchronization Failure

Fri, 05 Sep 2008 19:08:44 GMT

Applies to the SC5832, SC1458, and SC648 only.

The nodes synchronize with the SSP, which synchronizes with an upstream NTP time server. If the SSP cannot synchronize to an official upstream NTP server, the nodes need to synchronize directly with the SSP to maintain time consistency. For that to happen, you need to edit the /etc/ntp.conf file and add the line:

 server 127.127.1.0

immediately before the line:

 fudge 127.127.1.0 stratum 10

Linking netcdf from Fortran

Fri, 05 Sep 2008 16:42:13 GMT

The C and Fortran 77 interfaces to netcdf work, but the Fortran 90 interface does not. Attempts to link applications with netcdf using Fortran 90 will fail with missing symbols reported.

Avoiding a Lustre Race Condition

Fri, 05 Sep 2008 13:51:48 GMT

A race condition can occur when the Lustre file system comes under heavy load. To avoid this problem, set the sys.timeout value for the file system to 300. You can do this in one of three ways. The following examples assume that the file system is named fubar, and the MDT device is /dev/sda:

During creation of the MDT

 mkfs.lustre - -mgs - -mdt - -param sys.timeout=300 .../dev/sda

On an existing, but unmounted, MDT

tunefs.lustre - -param sys.timeout=300 /dev/sda

On a running system, or via the mount script after the MDT has been mounted

 lctl conf_param fubar.sys.timeout=300

Linking the BLACS library

Thu, 04 Sep 2008 20:12:06 GMT

Three libraries - libblacs.a, libblacsCinit.a, and libblacsF77init.a - comprise the BLACS library. When you link the BLACS init routines, include either -libblacsCinit or -libblacsF77init, but not both, on the link line. Linking a Fortran program to the C entry point produces unexpected behavior.

MPI Sync Cycles %

Thu, 04 Sep 2008 19:51:01 GMT

The MPI Sync Cycles % derived metric in the papiex output files, job_summary.txt and task_summary.txt, always report 0. However, the raw values in the individual thread and task files are correct, so you can manually calculate the summary values using the raw MPI Sync Cycles % value from each of the individual thread and task files.

Peephole Optimization

Thu, 04 Sep 2008 19:33:04 GMT

Some of the PathScale compiler's peephole optimizations can generate incorrect code. If you encounter this problem, disable peephole optimization by including -CG:ebo_level=0 on the compile line.

Limiting access to cluster compute nodes (2.2b only)

Mon, 04 Aug 2008 14:41:37 GMT

RECONFIGURE VMWARE TO ADD NAT

Shutdown the VMWare SSP.
On Red Hat as root, run vmware-config.pl and pres return until you see the question:
"Would you like to skip networking setup and keep your old settings as they are?
(yes/no)". Enter no.
In response to the question: "Do you want networking for your virtual machines? (yes/no/help), enter yes.
In response to the question: "Would you prefer to modify your existing networking configuration using the
wizard or the editor? (wizard/editor/help)", enter wizard.
The system displays the current network configuration listing the bridged interfaces and the question, "Do you want to be able to use NAT networking in your virtual machines? (yes/no)." Enter yes.
The system displays something like, "The NAT network is currently configured to use the private subnet 192.168.185.0/255.255.255.0. Do you want to keep these settings?" Enter yes and note the network number and netmask.
In response to the question: "Do you wish to configure another NAT network?" enter no. The configure script builds the appropriate modules and restarts the services.

Reconfigure Red Hat

Edit /etc/hosts and add a line like: NET.2 ssp
Where NET is the subnet address from step 6 above and ".2" is the host number
Edit /etc/sysconfig/iptables and add the following three lines to the end of the file:
*nat
--append POSTROUTING --out-interface eth1 -j MASQUERADE
COMMIT
Run "/etc/init.d/iptables restart" to activate the new settings.

Reconfigure the VMWare SSP

Start the VMWare SSP
Correct the virtual network for Ethernet 2 by right clicking the button on the VMWare toolbar and selecting NAT from the drop-down menu.
Login to the VMWare SSP.
Edit /etc/conf.d/net and comment out the line: config_site=("dhcp").
Add the following lines (where NET refers to the subnet address from step 6 above and DNSSERVERS refers to the spece-separated IP addresses from the native Red Hat /etc/resolve.conf file):

config_site="NET.2 netmask 255.255.255.0"
gateway="NET.1"
routes_site=("default via NET.1")
dns_servers="DNSSERVERS"
dns_domain_site="your-domain"
Optional. Edit /opt/sicortex/dnsmasq-hosts.sca to change the IP address for ssp-ws to NET.1 (where NET is the subnet address from step 6 above and ".1" is the host number).

Now, when you reboot the VMWare SSP, you will notice several failures which are all safe to ignore:

Didn't start Apache2 (because there is no fully qualified domain name)
Failed to set clock (?)
No automount maps (safe to ignore)

SLURM compute partitions should contain nodes exclusively used for computations

Wed, 11 Jun 2008 19:32:29 GMT

The System ships with several sample partitions that are intended as examples only. The supplied partitions may not match your site’s I/O configuration and system administrative needs.

For example, on the SC5832, the sample partition scx-comp defines a compute partition that excludes the I/O nodes m[0,2,4,6,32,34]n6.

You can define your own partitions that match your site’s I/O and system administrative needs. For a compute partition, be sure to exclude all nodes used for I/O or system administrative tasks, such as head nodes, gateway nodes, file system servers, and so on.

For details on creating partitions, see Creating and Using Partitions in Chapter 5 of The SiCortex® System Administration Guide.

Marginal MGTnet links can cause boot failure

Wed, 11 Jun 2008 19:25:26 GMT

Typically, SiCortex systems are configured with one (SC648) or four (SC5832) gigabyte Ethernet ports that connect the SSP to management nodes in the system (m[0,2,4,6]n6 on the SC5832, and m0n6 on the SC648).

The SSP uses these links to serve the root file system to the nodes, either via NFS or by copying the file system image to the management nodes, which then serve the image to the other nodes via NBD.
When any of these Ethernet links are disconnected or otherwise flaky, the SSP will fail to deliver the file system image properly. When this happens, the other nodes distribute themselves across the available root file system servers.

For the SC5832, this may result in fewer root file system servers than is desirable. For the SC648, this may prevent the system from booting.

We recommend that system administrators do two things:

On the SSP — Make sure the GigE Ethernet cables are snugly connected to port 2 on the processor module(s) and to the appropriate port(s) on the SSP, then confirm that the links are working.

In ssp:/sbin/ifconfig, these links are named mgt[0,1,2,3]. You can use the Linux ethtool command to confirm that these links are up.

On the nodes — Check the console log files for the gateway nodes (typically sc1-m0n6 on the SC648, and scx-m[0,2,4,6]n6 on the SC5832). The log should show the interface is being brought up in the usual Linux way.

Compiler Overview

Tue, 03 Jun 2008 15:41:25 GMT

Compilers:

C/C++: GNU (gcc/g++) and Pathscale (pathcc/pathCC)
Fortran - pathf95.

The compilers build code by default for 64-bit MIPS 'n64'. The compilers can built for other ABIs, including a 32-bit 'n32', which can be selected using 'mabi' option to the compiler.

The standard options for gcc should include -mips64 -march=5kf. For both GNU and Pathscale compilers there are extensive tuning and optimization flags, with -O3 being a good starting point. For debugging, -g can be used, with a negligible performance penalty.

Standard compiler drivers such as mpicc, mpicxx, mpif77 and mpif90 exist, which automatically link with the MPI library.

Math libraries

The system ships with numerous standard math libraries, such as BLAS, FFT, etc. To use an optimized math library, which may use numerical approximations, link with 'libscm' /before/ linking with 'libm'. For dynamic executables already linked with libm, where you would like to use the faster libscm, do the following:

$ export LD_PRELOAD=libscm.so # for csh, use setenv
$
$ srun ....
$ unset LD_PRELOAD # for csh, use unsetenv

Performance Tools

The system ships with an excellent performance tools' suite. Most tools do not need recompilation of relinking. Here is an incomplete list. For details, refer to the Programming Guide or man pages.

papiex: Provides hardware performance metrics, such as cache misses, TLB misses, etc, for serial, threaded and MPI programs. This is the tool of choice to start a tuning exercise. It also provides time spent in I/O and MPI.

mpipex: Provides time spent in various MPI calls. Very low overhead, based on the Open Source mpiP library for LLNL.

ioex: Gives details on I/O calls and file access patterns exhibited by the application.

gptlex: Based on the open source GPTL library, provides a call-tree. Can cause significant overhead.

TAU: General purpose tuning tool, from University of Oregon. Requires re-compilation.

VAMPIR: MPI tuning tool with a nice GUI.

hpcex/hpcproftt/hpcviewer: Statistical line profiler that correlates user code (function/line of source code), against events such as cache misses. Derived from HPCToolkit from Rice University.

pfmon: Low-level tool for measuring performance metrics. Used system-wide performance measurements as well, such as for measuring OS noise.

DUMA: Memory debugger

Debuggers

gdb and Totalview.

Handling oom/malloc Failures

Mon, 19 May 2008 13:54:20 GMT

Because we run individual nodes without swap space, memory demands must be met locally. To do this set the Linux tuning parameters, overcommit_memory and overcommit_ratio, as follows. (For parameter details, see the man page.)

/proc/sys/vm/overcommit_memory = 2
/proc/sys/vm/overcommit_ratio = 90

which on our systems, prevents committing over 90% of physical RAM. Malloc will fail if your application exceeds this limit, even if it does not touch all of the memory malloced.

OOM Killer Basics

The oom killer tries to preserve the system by killing off applications, but it doesn’t always get the right one, and it may leave a node unusable. A node that becomes unusable does not affect the operation of the other nodes in the system because the unusable node’s fabric switch and links continue to operate normally.

Typically, when an oom occurs, the console log of an affected node contains messages, such as:

oom-killer: gfp_mask=0xd0, order=0

When a node becomes unresponsive, check the tail of the Linux console log file for messages. The console log file is located on the SSP in

/var/log//-n.console.

Overcommit Parameters

The overcommit_memory and overcommit_ratio parameters specify if and how to over commit physical memory.

• overcommit_memory = <0|1|2>

0 — Root allowed to heuristically over allocate memory slightly, but any obvious over commitment is refused.
1 — Always allow applications to over commit physical memory. Useful for some scientific applications, which allocate large amounts of memory, but don't actually touch all of the allocated pages.
2 — Never allow over commitment of memory. Refuse any request greater than overcommit_ratio = ## % of physical RAM. In these cases, malloc will fail.

• overcommit_ratio = <##>

The percentage of physical memory the application is allowed to commit when overcommit_memory = 2.

Setting the Overcommit Parameters

You can change the setting of the overcommit parameters two ways.

System-wide change — persistent through subsequent reboots

Edit the vm.overcommit_memory = and vm.overcommit_ratio = parameters in the /opt/sicortex/rootfs/default/etc/sysctl.conf file, then reboot the System.

System-wide change — in effect until next reboot

As root, reset the overcommit parameters at runtime, for example:
srun -p -N bash -c "echo <0|1|2> > /proc/sys/vm/overcommit_memory"
srun -p -N bash -c "echo > /proc/sys/vm/overcommit_ratio"

Lustre Considerations

If you allow over committing memory and see processes killed due to oom errors, umount then mount the Lustre file system to regain the file system space used by those processes.

Lustre — Mounting Multiple Sicortex File Systems

Wed, 09 Apr 2008 16:23:58 GMT

Collision occurs when two nodes in different SiCortex systems that have the same network id try to mount the external Lustre file system at the same time. Such collisions cause the mount to fail with the message transport endpoint shutdown, which will appear in dmesg on the external server, and in the SSP’s syslog and the node console logs.

There are two remedies:

Avoid having multiple SiCortex systems mount the same external Lustre file system at the same time.
Change the internal netblock on the SiCortex systems so that each node has a unique network id. For details, see Configuring I/O Nodes and Network Interfaces in Chapter 2, of T he SiCortex System Administration Guide.

Lustre — Reading and Writing the Same File from Two Different Nodes

Wed, 09 Apr 2008 16:18:51 GMT

Writing a file from one node while simultaneously reading it from another causes the file lock to ping-pong rapidly between the two, degrading performance.

On a Lustre file system, avoid simultaneously reading a file (such as doing tail -f) while it is being written to from another node.

Quick Start Guide Errata

Mon, 07 Apr 2008 17:58:24 GMT

How do I bring up a virtual console?

Mon, 07 Apr 2008 17:56:01 GMT

To bring up a virtual console, do the following:

Log in as user root, passwd sicortex
Enter scboot -p sca to start the SC072's CPUs

If the scboot command produces IOX or Gateway failures, type ctrl-C and re-issue the command. Resetting the MSP tmodule service processor) and nodes takes about two minutes.

Note that the /local directory is shared by all 72 nodes and the workstation.

How do I power on the system?

Mon, 07 Apr 2008 17:48:38 GMT

Pressing RESET (front left button) may be needed to produce the required boot/beep tone.
The beep comes 5 seconds after pressing reset and this assumes the rear power switch and front power (LED) are already switched on.
At power-on, errors about memory dump space and Eth1 can be ignored at this point; it will log a benign connection failure if no cable is present during boot time.
After several minutes of startup messages, you log into a Red Hat x86 board.
User: root passwd: sicortex