Forum: SC072-PDS Forum

Reply to: SC072-PDS Support

Mon, 29 Sep 2008 17:02:32 EDT

New software to ship in early Oct has the SSP native in a Red Hat terminal window. VMWare will no longer be used.

Reply to: Running the SC072 headless

Mon, 29 Sep 2008 16:56:32 EDT

The new OS (to be mailed out in early Oct) no longer uses VMWare. The SSP software is native in a Red Hat terminal window.

Reply to: Can't mount ssp-ws:/local on ssp (No route to host)

Thu, 26 Jun 2008 18:11:27 EDT

You need to allow nfs:tcp through the RedHat firewall.

On the ssp-ws, as root:

# system-config-securitylevel

then disable the firewall (drop-down menu "disable").

Re-try the mount and it should work. You can also try to leave the firewall enabled, and just allow the port nfs:tcp through. NFS is port 2049.

Post: Can't mount ssp-ws:/local on ssp (No route to host)

Thu, 26 Jun 2008 18:08:41 EDT

Even after setting up /etc/exports and running exportfs -a -v, you get an error message when, on the ssp, you say:

# mount -t nfs ssp-ws:/local /mnt

the error is:

mount: mount to NFS server 'ssp-ws' failed: System Error: No route to host.

View this individual post's RSS Feed

Reply to: external LDAP server configuration

Wed, 25 Jun 2008 13:54:59 EDT

The ou=People is a container that is usually used to store things associated with "people", such as user accounts; however it is by no means a hard set rule. An LDAP administrator can setup his/her own LDAP hierarchy and naming scheme, so if you may want to also ask this question to the LDAP Administrator of the external LDAP server you are connecting to.

It should be ok to change the base to be just "dc=example,dc=com" it just means that you are searching more of the LDAP tree, instead of just in the "People" container. This is really only an issue if you have a large LDAP tree as searches may take longer if you have to parse the whole LDAP tree instead of just a sub-container.

For a very brief overview of LDAP and to get a visual of what I'm talking about, see section 1 on http://www.gentoo.org/doc/en/ldap-howto.xml

Post: external LDAP server configuration

Wed, 25 Jun 2008 08:52:32 EDT

I am attempting to point the nodes in an SC072 system at an external LDAP server, one running on a Fedora 9 system. I am able to run ldapsearch on the SC072 nodes and get appropriate responses from the query. However, when I try to login to the SC072, I see in the log files on the Fedora 9 server that the SC072 has selected "ou=People,dc=example,dc=com" rather than "dc=example,dc=com" as the search base. This causes the Fedora 9 ldap server to return null results. (When I run ldapsearch on the nodes, I use -b "dc=example,dc=com", which gives the expected results.)
I do not know enough about configuring ldap to determine from whence arises the "ou=People" part of the base for the ldap query on the SC072 or to configure the Fedora 9 ldap server to make use of it. Can anyone point me to some useful documentation in this area?

View this individual post's RSS Feed

Reply to: SW Raid on SC072?

Wed, 04 Jun 2008 11:41:38 EDT

So, the script (applicable w/o modification for Catapults only) is available at:

http://sicortex.com/support/program_downloads/release_v2_2b_ws

Reply to: SW Raid on SC072?

Tue, 03 Jun 2008 16:34:33 EDT

Ok,

I was able to get lsiutil to build and install on the
node with the SAS card. I had to download the following:

http://sources.gentoo.org/viewcvs.py/*checkout*/gentoo-x86/sys-block/lsiutil/lsiutil-1.52.ebuild?rev=1.1 - This is the ebuild file for lsiutil

http://www.lsi.com/support/downloads/hbas/fibre_channel/LsiUtil_10502.zip - This is the zip package that contains
lsiutil

After getting lsiutil installed, building the raid
array, and formatting it according to the instructions
below, I can not get it to mount anywhere.

Where can I get a copy of the lustre_fs.sh script? I
had made a lustre_mount.sh script that came from an
example in the Administrator's Guide (for software
Ver 2.2) but it is not working. I used to have it
working using the SIIG cards Alan Blanchard mentioned
using 2 drives per IO node (nodes 9,10,11).

Any help would be appreciated.

- Mike

Reply to: SW Raid on SC072?

Tue, 03 Jun 2008 16:31:17 EDT

While we don't provide official support for this card, here's how to make it work:

1) d/l http://www.lsi.com/support/downloads/hbas/fibre_channel/LsiUtil_10502.zip into a directory accessible by a node
2) login to a node
3) unzip the file and cd Source
4) untar the lsiutil.tar.gz
5) cd lsiutil
6) run "make lsiutil"

And you should now be good to follow the directions Rich posted on 26 May.

Reply to: SW Raid on SC072?

Tue, 03 Jun 2008 11:07:02 EDT

Like Ben, I am having trouble finding the lsiutil_portage.tgz
package. Can you provide a link to it?

Thank you

- Mike Moore

Reply to: SW Raid on SC072?

Fri, 30 May 2008 16:19:26 EDT

We're getting ready to set up a RAID on our SC072-PDS, and I was about to follow the directions that Richard posted. However, I can't seem to find the LSIUTIL package for the SAS3041E-R on the LSI website. A Google search for the file name given in the instructions "lsiutil_portage.tgz" gives only this forum thread as a result. LSI does have an MSM package available for download, which looks like it would accomplish the same. Is this the case, or can someone point me to where I can find LSIUTIL?

Thanks,
BC

Reply to: SW Raid on SC072?

Mon, 26 May 2008 19:37:38 EDT

Hi Alan -
These notes have not been thoroughly checked.
See the follow-up notes for location of the LSI utils file and a pointer to the Lustre script.

Setting up a Direct-Attached Lustre File System

You need to perform this procedure only if you have installed and connected hard disks to one or more of the I/O nodes and are setting up a Lustre file system on these disks. You must "be root" to perform this process, which involves these steps:
• Install the lsiutil package(acquired from LSI website)
• Create the Raid device
• Create the partitions
• Set up Lustre

Installing the LSIUTIL Package

LSIUTIL license restrictions prevent preinstallation of the software, but SiCortex provides a
package that retrieves lsiutils from the web, and then builds and installs it on the System.
1. Log in to the SSP as root.
2. cd into /usr/portage/sys-block .
3. tar xvfz lsiutil_portage.tgz (from the network, or some other place you have located the file such as a CD or USB thumbdrive).
If you do not want to install from the web, you can copy the package onto a CD or a USB Flash drive, and install from there.
4. Log in to a node as root.
5. Run chrootfs.
[chrootfs]
The chrootfs tool mounts the extra file systems needed to change the root file system.
6. Set ACCEPT_KEYWORDS="~amd64", and run emerge lsiutil.
7. Wait several minutes while emerge downloads the package, builds, and installs it.
8. Exit chrootfs, and re-scboot the nodes.
Creating the Raid Device
9. Log in as root to whichever node has the LSI card connected.
This node could be m0n9, m0n10, or m0n11, depending on the physical location of
the card.
10. Run lsiutil.
11. Select the LSI card.
12. Select RAID actions (option 21)
13. Select Create volume (option 30)
14. Select the disks, one per line, then an empty line to finish.
15. Accept all of the defaults except the last two:
o mirroring, not striping
o use the full volume
o no write caching
o (non-default) do zero the first/last blocks
o (non-default) skip the initial resync
16. Keep entering 0 to get out of lsiutil.
17. rmmod mptsas && modprobe mptsas
This is necessary to rediscover the RAID volume (not its component disks) properly.
18. Look in /proc/partitions to see the RAID device.

Creating the Partitions
19. fdisk /dev/sda (or whatever RAID device you want to partition)

20. Use n to create a new MDT partition. Options are:
o primary partition
o use partition 1 for the MDT
o accept default for first cylinder
o allocate 10% of total space for MDT (e.g. on an 80G RAID device, type "+8000M")
21. Repeat for the OST partition, except:
o Use partition 2 instead of 1
o Accept default (remainder of device) for size
22. Use w to update the partition table and quit.
23. Check /proc/partitions.
If you created partitions on /dev/sda, for example, at this point you should also see /dev/sda1 and /dev/sda2.

Setting up Lustre
Lustre is a fully POSIX-compliant file system. It supports all operations you’d use on any file
system, and it is semantically transparent to the application.
For this step, we assume the following:
• The node with the LSI card (this case) is node 11.
• The RAID device is /dev/sda, therefore the MDT is /dev/sda1 and the OST is
/dev/sda2
• The user chooses "fubar" as the file system name
• You are still root on the SSP
. (These examples need to be adjusted for other configurations.)
24. Pick a name for the file system. The name must be seven characters or less.
25. Format the MDT.
mkfs.lustre --mgs --mdt --fsname=fubar --fsname=fubar \
--mgsnode=sca-m0n11 --comment="fubar MDT" –reformat /dev/sda1

26. Format the OST.
mkfs.lustre --ost --fsname=fubar --fsname=fubar \
--mgsnode=sca-m0n11 --comment="fubar OST" –reformat /dev/sda2

27. Copy the lustre_fs.sh script from (updated 6/4/2008) http://sicortex.com/support/program_downloads/release_v2_2b_ws into
/opt/sicortex/config/local.d
Anything found in /opt/sicortex/config/local.d will be launched at boot up, so the lustre_fs.sh script must be copied here.
28. Edit the lustre_fs.sh script, looking for the CHANGEME lines. Make
fsname=fubar (the name you chose for your file system).
29. Optionally, you can run the lustre_fs.sh script by hand (in a terminal window)to check functionality. Copy it into /opt/sicortex/rootfs/default/root:
srun -p sca -N 12 /root/lustre_fs.sh
30. Re-scboot the nodes.
You have now finished the setting up the Lustre file system. At this point, it should be
mounted as /tmp/fubar (for example) on all of the nodes, except the server node. Take note of this when running jobs.

See further notes for more details...

Reply to: SW Raid on SC072?

Sat, 24 May 2008 13:57:18 EDT

Back on 4-April, you said:
there will be forthcoming a setup procedure for RAID and Lustre when using a different card - the LSI SAS304
1E-R

We now have that specific card. Is the setup procedure of which you speak available?

Reply to: nodes won't boot -- power off on some selected modules

Tue, 13 May 2008 16:07:01 EDT

Aha! slurmctld was not running. After starting it, the nodes boot successfully.

Reply to: nodes won't boot -- power off on some selected modules

Tue, 13 May 2008 13:27:07 EDT

Is slurmctld running?

ssp ~ $ rc-status | grep slurm
slurmctld [ started ]

If not, you can start or restart it:

root@ssp ~ $ /etc/init.d/slurmctld start

root@ssp ~ $ /etc/init.d/slurmctld restart

Reply to: nodes won't boot -- power off on some selected modules

Tue, 13 May 2008 12:33:24 EDT

After power-cycling the hardware as Richard suggested, I'm now able to boot the nodes:

sicortex-ssp ~ # scboot -p sca
/var/state/route_info.sca checks out OK!

Creating boot configuration

Setting up MSPs

Setting up node rootfs image

Halting all nodes
Restarting ev1d
* Stopping ev1d ... [ ok ]
* Starting ev1d ... [ ok ]

Starting Master Fabric Daemon
slurm_load_jobs error: Unable to contact slurm controller (connect failure)
slurm_load_part: Unable to contact slurm controller (connect failure)
slurm_update error: Unable to contact slurm controller (connect failure)
Trouble setting nodes to SLURM 'Idle' state.
Cleaning up FabriCache client partitions
Clearing event daemon state
Restarting syslog-ng
* Stopping syslog-ng ... [ ok ]
* Starting syslog-ng ... [ ok ]

Loading and booting linux
bamf: Loading vmlinux
bamf: Loading bootk
Finished loading linux (kernel boot initiated)
----------scboot-monitor----------
secs kernel fabric initfs slurm
1763 12 12 12 0

However, the slurm controller doesn't seem to come up, and the timer just keeps ticking. Where should I look from here?

Reply to: SSP site DHCP issue

Tue, 13 May 2008 11:54:18 EDT

The quickstart drawing attempts to show that eth1 on ssp-ws has mapped to it a virtual port (site).

That is why you only see DHCP from the SSP-WS.
The operation you see is correct.

A future software release is hoped to have the VMWare SSP changed to a real window on RedHat.

Post: SSP site DHCP issue

Tue, 13 May 2008 10:39:17 EDT

Hi,

I am trying to get the site interface on the SSP (VM) on the
SC072 I support to get an IP address from our network DHCP
server. I never see any requests from the SSP on my DHCP
server. I see requests for the SSP-WS (Redhat host for VMware
Player) on my DHCP server. If I check the logs for the
DNSMasq server running on the SSP, I see the DHCP requests from
the site interface, but it does not receive an address as the
DNSMasq server does not have the MAC Address for the SSP site
interface.

Is this the proper operation of the SSP site interface?
According to the quick start guide, the SSP is supposed to
connect to the outside world through the site interface.
What would I need to change to get this to work? Is it
a misconfiguration of the virtual machine that is causing
the site interface to not go out on the SSP-WS ethernet
interface?

Thank you.

- Mike Moore

View this individual post's RSS Feed

Reply to: nodes won't boot -- power off on some selected modules

Thu, 08 May 2008 14:05:11 EDT

"/var/log/sca/sca-m0n8.console (and its companions) had nothing written to them."

ven from the earlier boots?

Hmm, are conserver and scconserver running on the SSP? (run rc-status)

"Setting up MSPs
Unable to setup the following MSP(s) in 120 seconds: sca-msp0
Caught signal, cleaning up."

What's in /var/log/mspstate/sca-msp0 ?

If you see lines with "AWOL"", that's consistent with the MSP being unreachable.

Post: Disk and RAM capacity Questions

Thu, 08 May 2008 13:58:47 EDT

We've gotten the following questions from a user:

How much RAM can you put in an SC072?
We ship with 48GB.
The 24 DIMMs are 2 GB each.
Two DIMMs per chip.

How much RAM per core?
4GB/6 cores = 2/3GB per CPU.

Can you utilize 4GB DIMMs instead?
We don't yet offer 4 GB DIMMs.

Which manufacturer makes the CPUs?
TSMC fabs the chip that we designed.

View this individual post's RSS Feed

Reply to: nodes won't boot -- power off on some selected modules

Tue, 06 May 2008 16:06:29 EDT

hmmm... a few notes ago, the Module Service Processor (MSP) was seen successfully.

Try shutting all windows down, and turn off power.
Then come back on and if the MSP portion of the scboot starts to work (hopefully) but fails with err: "all 12 nodes checked in, but no router available (none are gateways)", then hit cntrl-c and re-execute the scboot -p sca command.

Also, after "Caught signal, cleaning up" - wait a few minutes (4) and see if it comes up.

Reply to: nodes won't boot -- power off on some selected modules

Tue, 06 May 2008 15:54:01 EDT

I've checked the connections in the cabinet, and all appears well. When I try to boot the nodes (using DHCP for networking), I now get:

sicortex-ssp ~ # scboot -p sca
/var/state/route_info.sca checks out OK!

Creating boot configuration

Setting up MSPs
Unable to setup the following MSP(s) in 120 seconds: sca-msp0
Caught signal, cleaning up.

sicortex-ssp ~ #

Using the -start_msp=force option gives the same results.

/var/log/sca/sca-m0n8.console (and its companions) had nothing written to them. /var/log/sca/mfd.log had:
[2008-05-06 15:38:55,590] Lost connection on fd6
[2008-05-06 15:38:55,674] orphan on kid6 died

Reply to: [SC072] /etc/modprobe.conf contains wrong ethernet drivers?

Tue, 06 May 2008 15:30:21 EDT

If you received your machine very recently (within 2 weeks) and all is working, then you are ok.

The newest r8168 driver now takes care of the r8139 chip also. In the past, we had to copy-in the r8139 driver.

Reply to: [SC072] /etc/modprobe.conf contains wrong ethernet drivers?

Tue, 06 May 2008 14:26:51 EDT

sorry, I meant to say "r8168 driver"...

Post: [SC072] /etc/modprobe.conf contains wrong ethernet drivers?

Tue, 06 May 2008 14:15:23 EDT

According to lspci, eth0 on the SC072 is an RTL8139 ethernet controller, but /etc/modprobe.conf aliases eth0 to the r8169 driver... Is this the way it's supposed to be, or is this a bug?

View this individual post's RSS Feed

Reply to: User Ethernet Port not Activating

Thu, 01 May 2008 19:26:57 EDT

Well I tried using the p8 version on sda6 with the sca profile, but it seems as if the setup is incomplete on it. It is missing the correct hosts file and no update-passwd script and such. I'll wait for the DVD update to proceed. Thanks!

Reply to: User Ethernet Port not Activating

Thu, 01 May 2008 14:22:18 EDT

The edit above reportedly allowed the machine to scboot properly from sda5.

Reply to: User Ethernet Port not Activating

Thu, 01 May 2008 11:12:21 EDT

The SC072 software is soon to be up-gradeable.
But if you tried an install/upgrade anyway and used the generic command from the Administration Guide, the profile shown is "sfx". For the SC072, please use 'sca' as the profile.

You can edit /boot/state/scprofile and set it to 'sca' if need be.

Post: User Ethernet Port not Activating

Thu, 01 May 2008 09:22:51 EDT

The system and port was running fine, but one of our users wanted the system to be upgraded to fix some gcc problems. We had version p6 on the primary partition and I tried upgrading to version p10, but it didn't seem to have the correct ssp.tgz file that the installer was looking for so I tried p9, but that one failed with an unexpected end of file error. I finally got p8 to install on sda6, so I reboot the vmware environment and it isn't reachable by ssh any more. I go down and take a look at the machine and the vmware environment is booted but there is no network now. I have tried booting back in to the original p6 environment as well and it doesn't work now either. The ethernet port light does not come on either. So I assumed the ethernet module just wasn't being loaded for the "site" interface any more so I look at lspci and it only shows the Intel Gigabit interface as being the only nic and lsmod actually shows e1000 being loaded. I've also tried modprobe -r e1000 and then modprobe e1000 because I've seen that bug out before and not activate and simply reloading the module reactivates it. No luck here though. Any suggestions or ideas? Is there a network module on the rhel side of the machine that needs to be loaded for vmware to see or is that port tied directly in to that virtual enviroment?

Thanks,

Paul

View this individual post's RSS Feed

Reply to: nodes won't boot -- power off on some selected modules

Tue, 29 Apr 2008 14:56:23 EDT

> The above point is moot temporarily. I used the
> sicortex-system.conf that Richard provided and I get the same
> problem. Any suggestions as to where I should look next?

If it fails with the same message "no router" as before, let's take a look at /var/log/sca/sca-m0n8.console from

Scanning network interfaces for MGTnet:

until

Loading scfab module

Reply to: nodes won't boot -- power off on some selected modules

Tue, 29 Apr 2008 14:52:40 EDT

Another thing to try is simply re-executing scboot -p sca or scboot -p sca --start_msp=force .

All boots should complete before the counter gets over 200s.

Reply to: nodes won't boot -- power off on some selected modules

Tue, 29 Apr 2008 14:41:56 EDT

Since the fan cable was in question, are the others ok? Looking down the rear edge of the x86 board, the blue Enet cable should be seen first and it gets connected to J3; low down on the 72processor module.

The enet cable in the middle rear of the x86 board should be green and connect to the coupler in the rear bulkhead. The far red enet cable (hardest to see) should connect to the RJ45 port that is near the rear bulkhead, but remains internal.

The recommendations from earlier notes were a way to reduce the changes and see if we can get the nodes to boot. Once that is solved, then other addressing changes can be looked into.

Here, I tell our network administrator what the HW MAC is of the port I wish to put on the net and a hostname. Then, when he sees that MAC, we automatically get the hostname that was requested.

Reply to: nodes won't boot -- power off on some selected modules

Tue, 29 Apr 2008 14:22:47 EDT

Richard Dischler wrote:

The default config for sca-m0n8 has to be used.
Simply allow the two RJ Enet ports to DHCP and then
use the addresses they receive.

***

Okay, so then how do I get my head node to have a static IP? I don't control the DHCP server so I can't give it a dynamically-assigned static IP. So then how would I give the head node a DNS host name to allow users to log in to it.

The above point is moot temporarily. I used the sicortex-system.conf that Richard provided and I get the same problem. Any suggestions as to where I should look next?

Reply to: nodes won't boot -- power off on some selected modules

Tue, 29 Apr 2008 13:11:57 EDT

The file should look like this:

# cat /etc/sicortex-system.conf
# SiCortex system configuration
# See /etc/sicortex-system.conf.example

# This header is necessary for the config
# parser module. Do not alter.
[DEFAULT]
sca.cluster.head-node = sca-m0n8
sca.node.sca-m0n8.eth1.address = dhcp
sca.node.sca-m0n8.interfaces = eth1
sca.node.sca-m0n8.router = nat

Reply to: nodes won't boot -- power off on some selected modules

Tue, 29 Apr 2008 13:10:02 EDT

The default config for sca-m0n8 has to be used.
Simply allow the two RJ Enet ports to DHCP and then
use the addresses they receive. If you edit sicortex-system.conf, then the virtual console between the 72 processor module and the x86 board will break.

Reply to: nodes won't boot -- power off on some selected modules

Tue, 29 Apr 2008 12:36:15 EDT

> As it turns out, the fans were not securely plugged to the
> power supply, so that caused the SSP to freak out. The
> nodes now boot, but now there's a complaint that none
> of them are the router:

The message means that m0n8 (which should be physically connected to the outside world) didn't come up. /var/log/sca/sca-m0n8.console will have some information about what happened there.

> Here's my sicortex-system.conf
>
> [DEFAULT]
> sca.cluster.head-node = sca-m0n8
> sca.cluster.io-nodes= sca-m0n8
> sca.cluster.default-router = ssp
>
> sca.node.sca-m0n8.router = nat
> sca.node.sca-m0n8.interfaces = eth1
> #sca.node.sca-m0n8.eth1.address = dhcp
> sca.node.sca-m0n8.eth1.address = 128.210.127.241
> sca.node.sca-m0n8.eth1.netmask = 255.255.255.0
> sca.node.sca-m0n8.eth1.gateway = 128.210.127.1

We generally run with DHCP; I'm not 100% sure what to look for if you statically define the address.

Reply to: nodes won't boot -- power off on some selected modules

Tue, 29 Apr 2008 12:22:34 EDT

As it turns out, the fans were not securely plugged to the power supply, so that caused the SSP to freak out. The nodes now boot, but now there's a complaint that none of them are the router:

sicortex-ssp ~ # scboot -p sca
/var/state/route_info.sca checks out OK!

Creating boot configuration

Setting up MSPs

Setting up node rootfs image

Halting all nodes
Restarting ev1d
* Stopping ev1d ... [ ok ]
* Starting ev1d ... [ ok ]

Starting Master Fabric Daemon
slurm_load_jobs error: Unable to contact slurm controller (connect failure)
slurm_load_part: Unable to contact slurm controller (connect failure)
slurm_update error: Unable to contact slurm controller (connect failure)
Trouble setting nodes to SLURM 'Idle' state.
Cleaning up FabriCache client partitions
Clearing event daemon state
Restarting syslog-ng
* Stopping syslog-ng ... [ ok ]
* Starting syslog-ng ... [ ok ]

Loading and booting linux
bamf: Loading vmlinux
bamf: Loading bootk
Finished loading linux (kernel boot initiated)
----------scboot-monitor----------
secs kernel fabric initfs slurm
33 12 0 0 0
err: all 12 nodes checked in, but no router available (none are gateways) - BOOT FAILED
151 12 12 0 0
Timeout (120 seconds since last activity)
1077 12 12 0 0

Here's my sicortex-system.conf

[DEFAULT]
sca.cluster.head-node = sca-m0n8
sca.cluster.io-nodes= sca-m0n8
sca.cluster.default-router = ssp

sca.node.sca-m0n8.router = nat
sca.node.sca-m0n8.interfaces = eth1
#sca.node.sca-m0n8.eth1.address = dhcp
sca.node.sca-m0n8.eth1.address = 128.210.127.241
sca.node.sca-m0n8.eth1.netmask = 255.255.255.0
sca.node.sca-m0n8.eth1.gateway = 128.210.127.1

Reply to: nodes won't boot -- power off on some selected modules

Tue, 29 Apr 2008 10:28:32 EDT

More quick tricks...

First try the reset button on the front (next to the larger power switch).

Also, make sure the four fans on the side panel are spinning.

Reply to: nodes won't boot -- power off on some selected modules

Tue, 29 Apr 2008 10:25:38 EDT

"bamf: power is off to the nodes on sca-msp0 - turn it on before booting!"

Node power is under software control, and is separate from the SSP power.

You can confirm the state of the node power with scpower:

sicortex-ssp % scpower -p sca
reading power from ['sca-msp0:0xfff']
[1L]

[0L] means it's off.

You can turn the power back on with

sicortex-ssp % scpower -p sca -s 1
setting power to 1 on ['sca-msp0:0xfff']

and try again.

If the power is off, there are a number of possible reasons:

1. MSP temperature monitoring:

check SSP's /var/log/msp-messages-YYYYMM for emerg level messages. If there's a message containing "Shutdown on overtemp", the MSP shut down the nodes for safety because it thought they got too hot.

2. SSP monitoring:

check SSP's /var/log/policyd.log for messages indicating that policyd shut down the power.

If either of these logs has interesting stuff in it, that may lead to a next step...

Post: nodes won't boot -- power off on some selected modules

Tue, 29 Apr 2008 10:02:10 EDT

When I try to boot the nodes, I get a message (below) saying that power is off to the nodes. The main power is obviously on, since I'm logged in to the SSP. I checked the internal connections and nothing appears to be unplugged. The NICs on the main board all have lights. I've added (but not yet configured) 4 SATA drives, but even when I pulled the power from them, the issue persists. What am I missing?

Thanks,
BC

sicortex-ssp ~ # scboot -p sca
/var/state/route_info.sca checks out OK!

Creating boot configuration

Setting up MSPs

Setting up node rootfs image

Halting all nodes
Restarting ev1d
* Stopping ev1d ... [ ok ]
* Starting ev1d ... [ ok ]

Starting Master Fabric Daemon
slurm_load_jobs error: Unable to contact slurm controller (connect failure)
slurm_load_part: Unable to contact slurm controller (connect failure)
slurm_update error: Unable to contact slurm controller (connect failure)
Trouble setting nodes to SLURM 'Idle' state.
Cleaning up FabriCache client partitions
Clearing event daemon state
Restarting syslog-ng
* Stopping syslog-ng ... [ ok ]
* Starting syslog-ng ... [ ok ]

Loading and booting linux
bamf: power is off to the nodes on sca-msp0 - turn it on before booting!
power off on some selected modules
Failed loading linux
Caught signal, cleaning up.
/sbin/scboot: line 326: 31983 Terminated ( trap - ${scboot_traps}; sleep "${msp_timeout}"; echo "TIMEOUT" >"${timeout_file}" )

View this individual post's RSS Feed

Reply to: Troubleshooting Guide?

Tue, 29 Apr 2008 09:38:16 EDT

I set up our SC072-PDS to use static external IPs and the networking seems fine, but I couldn't boot the nodes. The problem was definitely DNS related, but I couldn't figure it out. As I was looking through config files this morning, I realized that I took the local nameserver out of resolv.conf on the SSP. Putting that back in place fixed the problem. I'm sure I'll come up with more though. :-)

Reply to: Troubleshooting Guide?

Mon, 28 Apr 2008 13:48:24 EDT

The Troubleshooting Guide is not quite ready for prime time; it has not been made available, although as you note other docs reference it.

I can't say when it will be available, but if you have a specific question that got redirected thataway, we can probably help here...

Post: Troubleshooting Guide?

Mon, 28 Apr 2008 13:38:10 EDT

The System Administration Guide makes several references to a document called the "Troubleshooting Guide." There was no such document included in the shipment, and I can't find it on the SiCortex website. Is this available somewhere that I missed?

Thanks,
BC

View this individual post's RSS Feed

Reply to: SSP setup in SC072

Mon, 28 Apr 2008 09:02:09 EDT

Figured out that they are all Gentoo based distribution , so all related package management software are just working.

thanks

Reply to: SSP setup in SC072

Tue, 22 Apr 2008 16:23:43 EDT

Hi Richard,

Now we need to have user accounts and their home folders setup in the nodes. We have a NIS/NFS server for this and I am working on a node (with chrootfs) for the configuration. It looks like that I need to install ypbind and autofs packages on the root fs, so each node will ypbind to the NIS server and mount home folders.

What Linux distribution is the SiCortex-linux on each node based on? Is there prebuild binary or a repository for them and their dependencies?

Btw, Matt Reilly and Avi Purkayastha was giving presentation last week here and I had chances to talk with them. They directed me to you for all the administrative questions. Let me know your phone number if we need to talk over the phone and my email address is yanyh@cs.uh.edu

Thanks
Yonghong

Reply to: Running the SC072 headless

Thu, 17 Apr 2008 11:24:31 EDT

I don't think vmplayer will run without X, but we can give it a try. Otherwise, I guess we'll give vmware server a shot.

Reply to: Running the SC072 headless

Wed, 16 Apr 2008 13:46:31 EDT

I've not tried it with the SiCortex VM for the SSP, but you should be able to get VMWare Server and use that to run the SSP VM. To my knowledge, you should be able to have it start automatically on boot, without X. You could then connect to it over the network, or on the local console to view the SSP VM's console.

Reply to: Running the SC072 headless

Wed, 16 Apr 2008 12:46:01 EDT

We haven't tried anything like this so you may be heading into unchartered water.

Perhaps you can rig a script to launch vmplayer?

Post: Running the SC072 headless

Wed, 16 Apr 2008 12:13:51 EDT

We would really like to have our SC072 be able to run headless, as we typically use it remotely anyway.

Is it possible to automatically invoke the virtual machine and boot it without the need to run X? We were thinking of trying to switch from vmware-player to vmware-server, but we didn't know if this would work.

Thanks,
Jim

View this individual post's RSS Feed

Reply to: SSP setup in SC072

Wed, 16 Apr 2008 09:41:16 EDT

Hi Richard,

With your information, I managed to have the network for both the ssp and head/io node configured and the "srun -p sca -N 12 hostname" just runs fine.

With all my appreciation for your response!

Reply to: SSP setup in SC072

Wed, 16 Apr 2008 09:09:49 EDT

The normal SiCortex system network setup has been augmented in the following ways:
Workstation eth0 is statically configured with address 172.31.0.2

Workstation eth1 is the site network (external Ethernet connection). By default, it is configured by DHCP. It is not necessary for this interface to be connected for the system to work, except that time synchronization may be affected.

Workstation eth2 is statically configured with the address 172.31.150.2. This is on the “management network”, which allows it to communicate directly with the head node (sca-m0n8).

Entries for the ssp, for ssp-ws (the workstation), and head (the head node) have been added to appropriate hosts files to avoid the need for network DNS.

Note that these configuration settings will not work if the netblock is changed.

Reply to: SSP setup in SC072

Tue, 15 Apr 2008 12:44:55 EDT

On the ssp, ifconfig reveals:

ctl at 172.31.0.1
lo at 127.0.0.1
mgmt0 at 172.31.150.1
site at 10.0.0.237 ; a received DHCP address.

On sca-m0n8, eth1 will try to DHCP.

On the ssp-ws, eth1 will try to DHCP.

Reply to: SSP setup in SC072

Tue, 15 Apr 2008 12:17:45 EDT

> The virtual SSP floats pre-configured between the RedHat workstation and
> the cluster and should be ok as is.

The text above and several notes back means do not go into the sysadmin guide and start too much of anything associated with the SSP. In this SC072 product, the SSP is virtual and different from the large machines described in the sysadmin guide. It was all just fine as shipped. The ports will DHCP.

Anyway, On the ssp:
sicortex-ssp ~ # cat /etc/conf.d/net
# /etc/conf.d/net:
# $Id: net 48738 2007-12-18 19:50:32Z brooks $

# Pull in system configuration and network profile
source /opt/sicortex/config/cthlib \
SCv_ssp_site_hostname \
SCv_system_profile \
SCv_internal_domain \
|| exit 1

# The "site" interface uses DHCP by default
config_site=("dhcp"); dhcpcd_site="-H -N -h $SCv_ssp_site_hostname"

# BEGIN static site configuration
#
# Get these names and numbers from your system administrator:
#
# IP address: ip1.ip2.ip3.ip4
# Broadcast address: bc1.bc2.bc3.bc4
# Netmask: nm1.nm2.nm3.nm4
# Gateway address: gw1.gw2.gw3.gw4
# DNS server(s): DNSa1.DNSa2.DNSa3.DNSa4 DNSb1.DNSb2.DNSb3.DNSb4
# NTP server(s): some.ntp.server another.ntp.server
# Domain name ssp.domain.name
#
# Uncomment these lines and fill in the names and numbers above.
#
# config_site="ip1.ip2.ip3.ip4 broadcast bc1.bc2.bc3.bc4 netmask nm1.nm2.nm3.nm4"
# gateway="site/gw1.gw2.gw3.gw4"
# routes_site=("default via gw1.gw2.gw3.gw4")
# dns_servers_site="DNSa1.DNSa2.DNSa3.DNSa4 DNSb1.DNSb2.DNSb3.DNSb4"
# dns_domain_site="my.domain.name"
#
# Do NOT configure ntp_servers_site in this file.
# The resulting /etc/ntp.conf file won't serve the nodes.
# Edit /etc/ntp.conf and add your NTP server names there
#
# ... see /etc/conf.d/net.example for further options
#
# END static site configuration

# Source configuration for ctl and mgt* interfaces
source /etc/conf.d/net_${SCv_system_profile}

dns_search_lo=( "$SCv_internal_domain" )
dns_servers_lo=( "127.0.0.1" )

Reply to: SSP setup in SC072

Tue, 15 Apr 2008 12:02:36 EDT

Hi Richard,

I following the sysadmin guide to configure the network of the ssp. see below:
config_site="129.7.243.5 broadcast 129.7.243.255 netmask 255.255.255.0"
gateway="site/129.7.243.254"
routes_site=("default via 129.7.243.254")
dns_servers_site="129.7.240.1 129.7.1.1 127.0.0.1"

that is the site interface that I use to ssh to ssp remotely. and after reboot, the /etc/resolv.conf was regenerated from /etc/conf.d/net as
# Generated by resolvconf
nameserver 129.7.240.1
nameserver 129.7.1.1
nameserver 127.0.0.1

it cannot resolve those internal hosts, such as ssp; and thus the scboot does not work:
scboot -p sca
/var/state/route_info.sca checks out OK!

Creating boot configuration
Could not resolve sca-msp0
Could not resolve mgt0-ssp0
Could not resolve ssp
Could not resolve ssp
Could not resolve ssp
Could not resolve ssp
Could not resolve msp-ssp.scsystem

Setting up MSPs

Setting up node rootfs image

Halting all nodes
scand unresponsive, try 0/5
scand unresponsive, try 1/5
scand unresponsive, try 2/5
scand unresponsive, try 3/5
scand unresponsive, try 4/5
scand connection failed
Halt of nodes on sca-msp0 failed
Caught signal, cleaning up.

So what should be the right configuration in /etc/conf.d/net

Reply to: SSP setup in SC072

Tue, 15 Apr 2008 11:52:45 EDT

More...

sicortex-ssp ~ # cat /etc/sicortex-system.conf
# SiCortex system configuration
# See /etc/sicortex-system.conf.example

# This header is necessary for the config
# parser module. Do not alter.
[DEFAULT]
sca.cluster.head-node = sca-m0n8
sca.node.sca-m0n8.eth1.address = dhcp
sca.node.sca-m0n8.interfaces = eth1
sca.node.sca-m0n8.router = nat

Reply to: SSP setup in SC072

Tue, 15 Apr 2008 11:46:22 EDT

sca-m0n8 ~ # cat /etc/resolv.conf
domain scsystem
nameserver 172.31.150.1

sicortex-ssp ~ # cat /etc/resolv.conf
# Generated by resolvconf
search scsystem sicortex.com
nameserver 127.0.0.1

Reply to: SSP setup in SC072

Tue, 15 Apr 2008 11:20:09 EDT

Thanks Richard for your answer. I will try them out.

I messed up the /etc/resolv.conf file and do not have a backup. Also I found several dns configuration as /etc/dnsmasq*. what should be the right way to configure the resolv.conf.

thanks

Reply to: SSP setup in SC072

Tue, 15 Apr 2008 09:52:53 EDT

The virtual SSP's "site" port (physically the same as the port labled SSP) also DHCP's to get an address.

Reply to: SSP setup in SC072

Tue, 15 Apr 2008 09:47:00 EDT

To get better disk access, you can:
1. Add internal disks via PCIe SATA cards.
2. Use sca-m0n8 Gbit Enet port to go elsewhere.
3. Use some other PCIe option for IO.

Reply to: SSP setup in SC072

Tue, 15 Apr 2008 09:45:08 EDT

The setup you received is the same as others' = normal.

Some text from another discussion:

> The virtual SSP floats pre-configured between the RedHat workstation and
> the cluster and should be ok as is.
>
> RH (ssp-ws) is hard-wired to the "SSP" enet jack and the port is eth1.
> The cluster, (via head=sca-m0n8 and a Gbit Enet chip) is hard-wired to the
> other jack and seen as eth1. The pre-configuredvirtual ssp is suspended
> across.
>
>
> Once password+user changes are made in the VMWare ssp window, and scboot
> is done, you could just iconify it.
> Then, make all other contact via:
> 1. In-office RH Xterms or "Konsoles".
>
> 2. Users ssh'ing into the IP address that eth1 of sca-m0n8 wants or gets.
>
> Type "ssh head" from the VM window and see detail with "ifconfig eth1".
> Assuming your users are remote (?), have your network know about the head
> node "head" (sca-m0n8, eth1).
>
> Users can then be directed to other nodes or perhaps ssh to ssp-ws or ssp
> if need be.
> Any "srun" commands can be executed from a node or the ssp.
> Example:
> srun -p sca -N 12 hostname

Post: SSP setup in SC072

Tue, 15 Apr 2008 08:55:29 EDT

We have a SC072 system just arrived. Now it power up and is on our network. question here:

1. the SSP Workstation is just a standard RHEL 5 installation plus a VMWare player, which start the real SSP. Several administration issues here: first, I have to configure two IP for both the SSP-WS and the SSP; second, since the VMWare player relies on a Desktop or X-windows on the host system, I have to make sure those video/mouse/keyboard working anyway. Is this a normal setup? The admin guide does not mention this. why not just put in the SSP-WS.

2. The SSP, which is a guest VMWare appliance also serve as the rootfs server for the computation node. I believe there is some performance issue of file system access since you have to go through two system level for access file. I consider this a big issue. The admin guide says that the default rootfs node for SC072 is m0n8, but it is not the case with the system we have.

3.

View this individual post's RSS Feed

Reply to: SC072-PDS Support

Thu, 10 Apr 2008 16:54:12 EDT

We would like to run our SC072 headless, as we don't really need a graphical frontend most of the time. Is there a convenient way to get the virtual machine to run automatically without a need for X? We thought about trying to run it with vmware server instead of player.

And thank you for the previous reply, we are trying to use the included MPI implementation, though we would prefer openMPI if the team makes it available.

Thanks,
Jim

Reply to: SC072-PDS Support

Thu, 10 Apr 2008 13:55:32 EDT

Jim, just in case this isn't clear:

SiCortex does support MPI on all of our systems. MPI comes pre-installed with the rest of the system. It is our own MPI implementation, optimized to take advantage of our fabric hardware. The code is derived from MPICH2, not OpenMPI, but from the point of view of MPI applications there should be little if any difference.

The Programming Guide explains how to build and run MPI programs on the system.

-dg

Reply to: SC072-PDS Support

Thu, 10 Apr 2008 13:33:39 EDT

The OpenMPI people have not yet ported OpenMPI to the MIPS architecture.

The Open MPI folks AT Tennessee have an SC072 and are working on a port...

These comments came from Phil Mucci.

Reply to: SC072-PDS Support

Wed, 09 Apr 2008 12:47:01 EDT

We are trying to run it on the head cluster node.

Reply to: SC072-PDS Support

Tue, 08 Apr 2008 19:09:18 EDT

Are you running configure on the Redhat workstation, Gentoo VM or the cluster nodes?

Reply to: SC072-PDS Support

Tue, 08 Apr 2008 16:23:35 EDT

We're trying to build OpenMPI 1.2.6 on an SC072-PDS, but cannot get configure to run. There is an error that says "no atomic primitives" for the MIPS architecture.

Jim

Reply to: networking on SC072 VM

Mon, 07 Apr 2008 17:29:43 EDT

That's almost a work around. If wget were to work, then scemrge would work directly, too. But I was able to wget the file from the RH side and sftp it to the VM, then do the scemerge.

Reply to: networking on SC072 VM

Mon, 07 Apr 2008 15:21:01 EDT

Alan, using the following work-around you can install the mdadm utility and get some useful work done while we work on your networking issue.

Basically, you log into the head node, then log into the ssp and put the mdadm tarball in the portage directory. Then scemerge will install the utility without needing to download the tarball.

Here are the necessary commands:

> ssh sca-m0n8
> ssh ssp
> cd /usr/portage/distfiles
> wget http://distfiles.gentoo.org/distfiles/mdadm-2.6.4.tar.bz2
> scemerge -v sys-fs/mdadm

Reply to: networking on SC072 VM

Mon, 07 Apr 2008 14:17:01 EDT

In the DHCP server logs, we see the host DHCP requests, but not the VM requests. We also have other machines on the same network that run VMware and they don't have trouble getting IP addresses for both the host and VM.

Reply to: networking on SC072 VM

Mon, 07 Apr 2008 13:29:29 EDT

More clarification about the SC072-PDS's DHCP usage:

Both the Redhat and VM will request IP addresses via DHCP when they boot up. They will specify different MAC addresses in their DISCOVER requests. Redhat will send the MAC address associated with the physical hardware interface. The VM will generate its own MAC address to distinguish itself from Redhat.

As a result, it's important to check the your local DHCP server administrator to ensure that your local DHCP server will respond properly to both requests.

If your local DHCP server is not receiving both requests then we know we have a configuration issue in the SC072-PDS box.

You can determine the MAC address of the Redhat side with "ifconfig eth1". You can determine the MAC address of the VM side with "ifconfig site".

Reply to: networking on SC072 VM

Mon, 07 Apr 2008 12:49:00 EDT

To debug the DHCP failure on the VM's "site" interface it would be nice to know if your local DHCP server received the DHCP DISCOVER request. If it did receive the request did it log an error?

Reply to: networking on SC072 VM

Mon, 07 Apr 2008 12:37:31 EDT

The network connectivity became an issue because I want to raid some disks connected to the three I/O nodes. To do so I need some software installed, which requires network connectivity between the WM and the outside world. See the forum thread called "SW raid on Catapult?"

Reply to: networking on SC072 VM

Mon, 07 Apr 2008 11:56:40 EDT

You don't need to configure the VM's "site" interface to use the machine. From the RH workstation you can access the VM with "ssh ssp". Then, you can boot the cluster from the VM. The recommended way for users to access the machine is by logging into sca-m0n8 from the local network. Users should launch jobs from sca-m0n8.

Reply to: networking on SC072 VM

Mon, 07 Apr 2008 11:10:16 EDT

I can ssh to sca-m0n8 from the outside, then I can "ssh ssp" to get to the VM.
On the VM, device mgt0 is 172.31.150.1, and I can "ssh 172.31.150.1" from RH to get to the VM.
On RH, device eth2 is 172.31.150.2, and from the VM I can "ssh 172.31.150.2" to get to RH.

So I guess I'm unclear as to the purpose of the device "site" on the VM and how the VM is supposed to access the outside.

Reply to: networking on SC072 VM

Mon, 07 Apr 2008 10:56:39 EDT

We're still discussing this but, pls try:

Boot the machine with the Enet cable(s) connected to the LAN. You should get an address for the ports via DHCP.

Assuming you have RHEL up and have already activated VM, then users (coming in via HEAD - sca-m0n8) should be able to "SSH SSP" .

Reply to: networking on SC072 VM

Mon, 07 Apr 2008 10:38:58 EDT

Any suggestion on how to debug this problem? Are there any iptables or route table entries, on either the host or VM, that need to be set up to connect the VM to the outside world?

Reply to: User vs Admin Ethernet Ports

Mon, 07 Apr 2008 10:30:37 EDT

The second ethernet port will allow users to login to the cluster's head node (sca-m0n8). You will need to boot the cluster and create an account for each user. Access to the cluster via the VM is intended for administrators. Also, you will want to assign a hostname to the head node using DHCP on your local network.

Post: User vs Admin Ethernet Ports

Mon, 07 Apr 2008 09:43:59 EDT

The SSP ethernet port is for the admin to log in to the machine to access the actual RHEL base machine it appears. So does that mean that if I hook up another ethernet cable to the other (user) ethernet port that it will allow users to directly log in to the VM environment directly or do all users need to log in through the main machine then ssh to the VM environment?

View this individual post's RSS Feed

Reply to: networking on SC072 VM

Fri, 04 Apr 2008 15:46:31 EDT

Dale may have more info, but during factory setup, there were no Enet cables connected, so after no response during DHCP, a 169.xxx address was auto-picked and not used by us.