PPPL has two SGI Linux based systems: mhd and viz.
- Mhd is a 32 processor Altix 350 with 192GB of memory
- Viz is a 6 processor Prism with 30GB of memory
They have the same systems setups. Viz is used for development and is open to direct ssh logins. Mhd is not open to direct ssh logins and is batch accessible only.
These systems run SUSE Linux . From a user point of view, the system setups are the same as portal.pppl.gov. Most of portal setups apply to the SGI systems as well.
User's home directories are available under /u and project spaces are available under /p. These file systems are served from the same servers as they are on portals.
Like portal nodes, all systems have /local directories mounted locally and are available for everyone. This file system is served by the local host to provide fast access to a large 'tmp' like directory. It is not backed up, and files on /local that can be deleted by the system at any time.
The software environments
are setup through the
module utility. Here are a few examples to get you started quickly:
module
avail displays a list of available software packages.
$ module avail
------------------------ /usr/pppl/Modules/modulefiles -------------------------
avsexpress/7.0 intel_idb/8.1-047
freetds/0.63 intel_idb/9.0-026
globus/gt2(default) intel_idb/9.0-030(default)
histx/1.2a(default) intel_idb/9.0-031
intel_cc/8.1-029 intel_mkl/7.2.008
intel_cc/9.0-021 intel_mkl/7.2.1
intel_cc/9.0-025(default) intel_mkl/8.0.019(default)
intel_cc/9.0-026 intel_vt/3.0.814(default)
intel_fc/8.1-025 java/1.4.2_09(default)
intel_fc/9.0-021 mdsplus/v1.7-5
intel_fc/9.0-026(default) mdsplus/v1.7-5-gt2(default)
intel_fc/9.0-027
------------------- /usr/pppl/Modules/compiler/intel_fc/9.0 --------------------
fftw/2.1.5(default) ncarg/4.4.1(default) superlu_dist/2.0(default)
hdf/4.2r1(default) netcdf/3.6.0-p1(default) zlib/1.2.1(default)
hdf5/1.6.4(default) ntcc/20050712(default)
nag/mark20(default) superlu/3.0(default)
You may use module to load desired software packages into your shell environment. For example,
$ module load intel_fc
$ module load superlu
You may save module load commands in your ~/.bashrc or ~/.cshrc files based on your shell settings. Software packages available on SGI are different from those on portals. You may find /usr/pppl/bin/pppl_os handy to tell which type of system that you are on.
To run your program in a production environment, you need to use PBS batch system. There are two ways to access mhd through the batch system – interactive or batch only. For an interactive session on mhd, you may use the “use” command and select the options accordingly. The system will land you in a shell on mhd with required system resources.
kying@viz:/u/kying$ use
=================================
Welcome to PPPL Beowulf Cluster
=================================
What type of nodes do you need?
1. Falcon 100 Megabit (for single processor jobs) (default)
2. Falcon Gigabit (for multi-processor jobs)
3. SGI Prism - viz
4. SGI Altix 350 - mhd
5. NSTX pool
6. Sunfire
7. Kite (Infiniband)
Your selection is [1]: 4
How many CPUs do you need? [1]: 4
How much memory do you need? [2gb]: 16gb
This PBS session has walltime limit: walltime=16:00:00
You may use "-w hh:mm:ss" command line option to modify this limit.
qsub: waiting for job 92759.phoenix to start
qsub: job 92759.phoenix ready
kying@mhd:/u/kying $
For a batch only job, you may submit a job script against PBS “sgi” queue. The submitted job will run on mhd system. Here is an example of a simple script for a 4 processor job using 16GB of memory:
#!/bin/bash
#PBS -q sgi
#PBS -N my_PBS_job_name
#PBS -l ncpus=4
#PBS -l mem=16gb
mpirun -np 4 my_executable
To submit the job, use qsub command
$ qsub my_job_script
Some system libraries use openMP under the hood. In a few cases, they spawn excessive threads and overwhelm the system. Please set OMP_NUM_THREADS=1 in your shell setup files to prevent excessive threads.
For additional document, please check http://beowulf.pppl.gov and http://beowulf.pppl.gov/ws3/ for shell environment setups. If you need additional help, you may email unixadmin@pppl.gov.