KNL Compute Nodes

Stampede2 hosts 4,200 KNL compute nodes, including 504 KNL nodes that were formerly configured as a Stampede1 sub-system.

Each of Stampede2's KNL nodes includes 96GB of traditional DDR4 Random Access Memory (RAM). They also feature an additional 16GB of high bandwidth, on-package memory known as Multi-Channel Dynamic Random Access Memory (MCDRAM) that is up to four times faster than DDR4. The KNL's memory is configurable in two important ways: there are BIOS settings that determine at boot time the processor's memory mode and cluster mode. The processor's memory mode determines whether the fast MCDRAM operates as RAM, as direct-mapped L3 cache, or as a mixture of the two. The cluster mode determines the mechanisms for achieving cache coherency, which in turn determines latency: roughly speaking, this mode specifies the degree to which some memory addresses are "closer" to some cores than to others. See "Programming and Performance: KNL" below for a top-level description of these and other available memory and cluster modes.

Table 1. Stampede2 KNL Compute Node Specifications

SKX Compute Nodes

Stampede2 hosts 1,736 SKX compute nodes.

Table 2. Stampede2 SKX Compute Node Specifications

Login Nodes

The Stampede2 login nodes, upgraded at the start of Phase 2, are Intel Xeon Gold 6132 (SKX) nodes, each with 28 cores on two sockets (14 cores/socket). They replace the decommissioned Broadwell login nodes used during Phase 1.

Network

The interconnect is a 100Gb/sec Intel Omni-Path (OPA) network with a fat tree topology employing six core switches. There is one leaf switch for each 28-node half rack, each with 20 leaf-to-core uplinks (28/20 oversubscription).

File Systems Introduction

Stampede2 mounts three shared Lustre file systems on which each user has corresponding account-specific directories $HOME, $WORK, and $SCRATCH. Each file system is available from all Stampede2 nodes; the Stockyard-hosted work file system is available on most other TACC HPC systems as well. See Navigating the Shared File Systems for detailed information as well as the Good Citizenship file system guidelines.

The $SCRATCH file system, as its name indicates, is a temporary storage space. Files that have not been accessed* in ten days are subject to purge. Deliberately modifying file access time (using any method, tool, or program) for the purpose of circumventing purge policies is prohibited.

Table 3. Stampede2 File Systems

Secure Shell (SSH)

The "ssh" command (SSH protocol) is the standard way to connect to Stampede2. SSH also includes support for the file transfer utilities scp and sftp. Wikipedia is a good source of information on SSH. SSH is available within Linux and from the terminal app in the Mac OS. If you are using Windows, you will need an SSH client that supports the SSH-2 protocol: e.g. Bitvise, OpenSSH, PuTTY, or SecureCRT. Initiate a session using the ssh command or the equivalent; from the Linux command line the launch command looks like this:

localhost$ ssh myusername@stampede2.tacc.utexas.edu

The above command will rotate connections across all available login nodes and route your connection to one of them. To connect to a specific login node, use its full domain name:

localhost$ ssh myusername@login2.stampede2.tacc.utexas.edu

To connect with X11 support on Stampede2 (usually required for applications with graphical user interfaces), use the "-X" or "-Y" switch:

localhost$ ssh -X myusername@stampede2.tacc.utexas.edu

Use your TACC password, not your XSEDE password, for direct logins to TACC resources. You can change your TACC password through the TACC User Portal. Log into the portal, then select "Change Password" under the "HOME" tab. If you've forgotten your password, go to the TACC User Portal home page and select "Password Reset" under the Home tab.

To report a connection problem, execute the ssh command with the "-vvv" option and include the verbose output when submitting a help ticket.

Do not run the "ssh-keygen" command on Stampede2. This command will create and configure a key pair that will interfere with the execution of job scripts in the batch system. If you do this by mistake, you can recover by renaming or deleting the .ssh directory located in your home directory; the system will automatically generate a new one for you when you next log into Stampede2.

1. execute "mv .ssh dot.ssh.old"
2. log out
3. log into Stampede2 again

After logging in again the system will generate a properly configured key pair.

XSEDE Single Sign-On Hub

XSEDE users can also access Stampede2 via the XSEDE Single Sign-On Hub.

When reporting a problem to the help desk, please execute the gsissh command with the "-vvv" option and include the verbose output in your problem description.

Configuring Your Account

$ echo $SHELL

$ env | grep GIT

login1$ module load sanitytool
login1$ sanitycheck

Accessing the Compute Nodes

You connect to Stampede2 through one of four "front-end" login nodes. The login nodes are shared resources: at any given time, there are many users logged into each of these login nodes, each preparing to access the "back-end" compute nodes (Figure 2. Login and Compute Nodes). What you do on the login nodes affects other users directly because you are competing for the same memory and processing power. This is the reason you should not run your applications on the login nodes or otherwise abuse them. Think of the login nodes as a prep area where you can manage files and compile code before accessing the compute nodes to perform research computations. See Good Citizenship for more information.

You can use your command-line prompt, or the "hostname" command, to tell you whether you are on a login node or a compute node. The default prompt, or any custom prompt containing "\h", displays the short form of the hostname (e.g. c401-064). The hostname for a Stampede2 login node begins with the string "login" (e.g. login2.stampede2.tacc.utexas.edu), while compute node hostnames begin with the character "c" (e.g. c401-064.stampede2.tacc.utexas.edu). Note that the default prompts on the compute nodes include the node type (knl or skx) as well. The environment variable TACC_NODE_TYPE, defined only on the compute nodes, also displays the node type. The simplified prompts in the User Guide examples are shorter than Stampede2's actual default prompts.

While some workflows, tools, and applications hide the details, there are three basic ways to access the compute nodes:

1. Submit a batch job using the sbatch command. This directs the scheduler to run the job unattended when there are resources available. Until your batch job begins it will wait in a queue. You do not need to remain connected while the job is waiting or executing. See Running Jobs for more information. Note that the scheduler does not start jobs on a first come, first served basis; it juggles many variables to keep the machine busy while balancing the competing needs of all users. The best way to minimize wait time is to request only the resources you really need: the scheduler will have an easier time finding a slot for the two hours you need than for the 48 hours you unnecessarily request.
2. Begin an interactive session using idev or srun. This will log you into a compute node and give you a command prompt there, where you can issue commands and run code as if you were doing so on your personal machine. An interactive session is a great way to develop, test, and debug code. When you request an interactive session, the scheduler submits a job on your behalf. You will need to remain logged in until the interactive session begins.
3. Begin an interactive session using ssh to connect to a compute node on which you are already running a job. This is a good way to open a second window into a node so that you can monitor a job while it runs.

Be sure to request computing resources that are consistent with the type of application(s) you are running:

• A serial (non-parallel) application can only make use of a single core on a single node, and will only see that node's memory.

• A threaded program (e.g. one that uses OpenMP) employs a shared memory programming model and is also restricted to a single node, but the program's individual threads can run on multiple cores on that node.

• An MPI (Message Passing Interface) program can exploit the distributed computing power of multiple nodes: it launches multiple copies of its executable (MPI tasks, each assigned unique IDs called ranks) that can communicate with each other across the network. The tasks on a given node, however, can only directly access the memory on that node. Depending on the program's memory requirements, it may not be possible to run a task on every core of every node assigned to your job. If it appears that your MPI job is running out of memory, try launching it with fewer tasks per node to increase the amount of memory available to individual tasks.

• A popular type of parameter sweep (sometimes called high throughput computing) involves submitting a job that simultaneously runs many copies of one serial or threaded application, each with its own input parameters ("Single Program Multiple Data", or SPMD). The "launcher" tool is designed to make it easy to submit this type of job. For more information:

    $ module load launcher
    $ module help launcher

Using Modules to Manage your Environment

Lmod, a module system developed and maintained at TACC, makes it easy to manage your environment so you have access to the software packages and versions that you need to conduct your research. This is especially important on a system like Stampede2 that serves thousands of users with an enormous range of needs. Loading a module amounts to choosing a specific package from among available alternatives:

$ module load intel          # load the default Intel compiler
$ module load intel/17.0.4   # load a specific version of Intel compiler

A module does its job by defining or modifying environment variables (and sometimes aliases and functions). For example, a module may prepend appropriate paths to $PATH and $LD_LIBRARY_PATH so that the system can find the executables and libraries associated with a given software package. The module creates the illusion that the system is installing software for your personal use. Unloading a module reverses these changes and creates the illusion that the system just uninstalled the software:

$ module load   ddt  # defines DDT-related env vars; modifies others
$ module unload ddt  # undoes changes made by load

The module system does more, however. When you load a given module, the module system can automatically replace or deactivate modules to ensure the packages you have loaded are compatible with each other. In the example below, the module system automatically unloads one compiler when you load another, and replaces Intel-compatible versions of IMPI and PETSc with versions compatible with gcc:

$ module load intel  # load default version of Intel compiler
$ module load petsc  # load default version of PETSc
$ module load gcc    # change compiler

Lmod is automatically replacing "intel/17.0.4" with "gcc/7.1.0".

Due to MODULEPATH changes, the following have been reloaded:
1) impi/17.0.3     2) petsc/3.7

On Stampede2, modules generally adhere to a TACC naming convention when defining environment variables that are helpful for building and running software. For example, the "papi" module defines TACC_PAPI_BIN (the path to PAPI executables), TACC_PAPI_LIB (the path to PAPI libraries), TACC_PAPI_INC (the path to PAPI include files), and TACC_PAPI_DIR (top-level PAPI directory). After loading a module, here are some easy ways to observe its effects:

$ module show papi   # see what this module does to your environment
$ env | grep PAPI    # see env vars that contain the string PAPI
$ env | grep -i papi # case-insensitive search for 'papi' in environment

To see the modules you currently have loaded:

$ module list

To see all modules that you can load right now because they are compatible with the currently loaded modules:

$ module avail

To see all installed modules, even if they are not currently available because they are incompatible with your currently loaded modules:

$ module spider   # list all modules, even those not available to load

To filter your search:

$ module spider slep             # all modules with names containing 'slep'
$ module spider sundials/2.5.0   # additional details on a specific module

Among other things, the latter command will tell you which modules you need to load before the module is available to load. You might also search for modules that are tagged with a keyword related to your needs (though your success here depends on the diligence of the module writers). For example:

$ module keyword performance

You can save a collection of modules as a personal default collection that will load every time you log into Stampede2. To do so, load the modules you want in your collection, then execute:

$ module save    # save the currently loaded collection of modules 

Two commands make it easy to return to a known, reproducible state:

$ module reset   # load the system default collection of modules
$ module restore # load your personal default collection of modules

On TACC systems, the command "module reset" is equivalent to "module purge; module load TACC". It's a safer, easier way to get to a known baseline state than issuing the two commands separately.

Help text is available for both individual modules and the module system itself:

$ module help swr     # show help text for software package swr
$ module help         # show help text for the module system itself

See Lmod's online documentation for more extensive documentation. The online documentation addresses the basics in more detail, but also covers several topics beyond the scope of the help text (e.g. writing and using your own module files).

It's safe to execute module commands in job scripts. In fact, this is a good way to write self-documenting, portable job scripts that produce reproducible results. If you use "module save" to define a personal default module collection, it's rarely necessary to execute module commands in shell startup scripts, and it can be tricky to do so safely. If you do wish to put module commands in your startup scripts, see Stampede2's default startup scripts for a safe way to do so.

Do Not Run Jobs on the Login Nodes

Stampede2's few login nodes are shared among all users. Dozens, (sometimes hundreds) of users may be logged on at one time accessing the file systems. Think of the login nodes as a prep area, where users may edit and manage files, compile code, perform file management, issue transfers, submit new and track existing batch jobs etc. The login nodes provide an interface to the "back-end" compute nodes.

The compute nodes are where actual computations occur and where research is done. Hundreds of jobs may be running on all compute nodes, with hundreds more queued up to run. All batch jobs and executables, as well as development and debugging sessions, must be run on the compute nodes. To access compute nodes on TACC resources, one must either submit a job to a batch queue or initiate an interactive session using the idev utility.

A single user running computationally expensive or disk intensive task/s will negatively impact performance for other users. Running jobs on the login nodes is one of the fastest routes to account suspension. Instead, run on the compute nodes via an interactive session (idev) or by submitting a batch job.

Do not run jobs or perform intensive computational activity on the login nodes or the shared file systems.
Your account may be suspended and you will lose access to the queues if your jobs are impacting other users.

Do Not Stress the Shared File Systems

The TACC Global Shared File System, Stockyard, is mounted on most TACC HPC resources as the /work ($WORK) directory. This file system is accessible to all TACC users, and therefore experiences a lot of I/O activity (reading and writing to disk, opening and closing files) as users run their jobs, read and generate data including intermediate and checkpointing files. As TACC adds more users, the stress on the $WORK file system is increasing to the extent that TACC staff is now recommending new job submission guidelines in order to reduce stress and I/O on Stockyard.

TACC staff now recommends that you run your jobs out of the $SCRATCH file system instead of the global $WORK file system.

To run your jobs out $SCRATCH:

• Copy or move all job input files to $SCRATCH
• Make sure your job script directs all output to $SCRATCH
  
• Once your job is finished, move your output files to $WORK to avoid any data purges.

Compute nodes should not reference $WORK unless it's to stage data in/out only before/after jobs.

Consider that $HOME and $WORK are for storage and keeping track of important items. Actual job activity, reading and writing to disk, should be offloaded to your resource's $SCRATCH file system (see Table. File System Usage Recommendations. You can start a job from anywhere but the actual work of the job should occur only on the $SCRATCH partition. You can save original items to $HOME or $WORK so that you can copy them over to $SCRATCH if you need to re-generate results.

Limit Input/Output (I/O) Activity

In addition to the file system tips above, it's important that your jobs limit all I/O activity. This section focuses on ways to avoid causing problems on each resources' shared file systems.

• Limit I/O intensive sessions (lots of reads and writes to disk, rapidly opening or closing many files)

• Avoid opening and closing files repeatedly in tight loops. Every open/close operation on the file system requires interaction with the MetaData Service (MDS). The MDS acts as a gatekeeper for access to files on Lustre's parallel file system. Overloading the MDS will affect other users on the system. If possible, open files once at the beginning of your program/workflow, then close them at the end.

• Don't get greedy. If you know or suspect your workflow is I/O intensive, don't submit a pile of simultaneous jobs. Writing restart/snapshot files can stress the file system; avoid doing so too frequently. Also, use the hdf5 or netcdf libraries to generate a single restart file in parallel, rather than generating files from each process separately.

If you know your jobs will require significant I/O, please submit a support ticket and an HPC consultant will work with you. See also Managing I/O on TACC Resources for additional information.

File Transfer Guidelines

In order to not stress both internal and external networks, be mindful of the following guidelines:

• When creating or transferring large files to Stockyard ($WORK) or the $SCRATCH file systems, be sure to stripe the receiving directories appropriately. See Striping Large Files for more information.

• Avoid too many simultaneous file transfers. You share the network bandwidth with other users; don't use more than your fair share. Two or three concurrent scp sessions is probably fine. Twenty is probably not.

• Avoid recursive file transfers, especially those involving many small files. Create a tar archive before transfers. This is especially true when transferring files to or from Ranch.

Job Submission Tips

• Request Only the Resources You Need Make sure your job scripts request only the resources that are needed for that job. Don't ask for more time or more nodes than you really need. The scheduler will have an easier time finding a slot for a job requesting 2 nodes for 2 hours, than for a job requesting 4 nodes for 24 hours. This means shorter queue waits times for you and everybody else.

• Test your submission scripts. Start small: make sure everything works on 2 nodes before you try 20. Work out submission bugs and kinks with 5 minute jobs that won't wait long in the queue and involve short, simple substitutes for your real workload: simple test problems; hello world codes; one-liners like ibrun hostname; or an ldd on your executable.

• Respect memory limits and other system constraints. If your application needs more memory than is available, your job will fail, and may leave nodes in unusable states. Use TACC's Remora tool to monitor your application's needs.

Navigating the Shared File Systems

Stampede2's /home and /scratch file systems are mounted only on Stampede2, but the work file system mounted on Stampede2 is the Global Shared File System hosted on Stockyard. Stockyard is the same work file system that is currently available on Frontera, Lonestar5, and several other TACC resources.

The $STOCKYARD environment variable points to the highest-level directory that you own on the Global Shared File System. The definition of the $STOCKYARD environment variable is of course account-specific, but you will see the same value on all TACC systems that provide access to the Global Shared File System. This directory is an excellent place to store files you want to access regularly from multiple TACC resources.

Your account-specific $WORK environment variable varies from system to system and is a sub-directory of $STOCKYARD (Figure 3). The sub-directory name corresponds to the associated TACC resource. The $WORK environment variable on Stampede2 points to the $STOCKYARD/stampede2 subdirectory, a convenient location for files you use and jobs you run on Stampede2. Remember, however, that all subdirectories contained in your $STOCKYARD directory are available to you from any system that mounts the file system. If you have accounts on both Stampede2 and Maverick, for example, the $STOCKYARD/stampede2 directory is available from your Maverick account, and $STOCKYARD/maverick is available from your Stampede2 account.

Your quota and reported usage on the Global Shared File System reflects all files that you own on Stockyard, regardless of their actual location on the file system.

See the example for fictitious user bjones in the figure below. All directories are accessible from all systems, however a given sub-directory (e.g. lonestar5, maverick2) will exist only if you have an allocation on that system.

Note that resource-specific sub-directories of $STOCKYARD are nothing more than convenient ways to manage your resource-specific files. You have access to any such sub-directory from any TACC resources. If you are logged into Stampede2, for example, executing the alias cdw (equivalent to "cd $WORK") will take you to the resource-specific sub-directory $STOCKYARD/stampede2. But you can access this directory from other TACC systems as well by executing "cd $STOCKYARD/stampede2". These commands allow you to share files across TACC systems. In fact, several convenient account-level aliases make it even easier to navigate across the directories you own in the shared file systems:

Table 4. Built-in Account Level Aliases

Striping Large Files

Stampede2's Lustre file systems look and act like a single logical hard disk, but are actually sophisticated integrated systems involving many physical drives (dozens of physical drives for $HOME, hundreds for $WORK and $SCRATCH).

Lustre can stripe (distribute) large files over several physical disks, making it possible to deliver the high performance needed to service input/output (I/O) requests from hundreds of users across thousands of nodes. Object Storage Targets (OSTs) manage the file system's spinning disks: a file with 16 stripes, for example, is distributed across 16 OSTs. One designated Meta-Data Server (MDS) tracks the OSTs assigned to a file, as well as the file's descriptive data.

Before transferring to, or creating large files on Stampede2, be sure to set an appropriate default stripe count on the receiving directory.

To avoid exceeding your fair share of any given OST, a good rule of thumb is to allow at least one stripe for each 100GB in the file. For example, to set the default stripe count on the current directory to 30 (a plausible stripe count for a directory receiving a file approaching 3TB in size), execute:

$ lfs setstripe -c 30 $PWD

Note that an "lfs setstripe" command always sets both stripe count and stripe size, even if you explicitly specify only one or the other. Since the example above does not explicitly specify stripe size, the command will set the stripe size on the directory to Stampede2's system default (1MB). In general there's no need to customize stripe size when creating or transferring files.

Remember that it's not possible to change the striping on a file that already exists. Moreover, the "mv" command has no effect on a file's striping if the source and destination directories are on the same file system. You can, of course, use the "cp" command to create a second copy with different striping; to do so, copy the file to a directory with the intended stripe parameters.

You can check the stripe count of a file using the "lfs getstripe" command:

$ lfs getstripe myfile

Transfer Using scp

You can transfer files between Stampede2 and Linux-based systems using either scp or rsync. Both scp and rsync are available in the Mac Terminal app. Windows ssh clients typically include scp-based file transfer capabilities.

The Linux scp (secure copy) utility is a component of the OpenSSH suite. Assuming your Stampede2 username is bjones, a simple scp transfer that pushes a file named "myfile" from your local Linux system to Stampede2 $HOME would look like this:

localhost$ scp ./myfile bjones@stampede2.tacc.utexas.edu:  # note colon after net address

You can use wildcards, but you need to be careful about when and where you want wildcard expansion to occur. For example, to push all files ending in ".txt" from the current directory on your local machine to /work/01234/bjones/scripts on Stampede2:

localhost$ scp *.txt bjones@stampede2.tacc.utexas.edu:/work/01234/bjones/stampede2

To delay wildcard expansion until reaching Stampede2, use a backslash ("\") as an escape character before the wildcard. For example, to pull all files ending in ".txt" from /work/01234/bjones/scripts on Stampede2 to the current directory on your local system:

localhost$ scp bjones@stampede2.tacc.utexas.edu:/work/01234/bjones/stampede2/\*.txt .

You can of course use shell or environment variables in your calls to scp. For example:

localhost$ destdir="/work/01234/bjones/stampede2/data"
localhost$ scp ./myfile bjones@stampede2.tacc.utexas.edu:$destdir

You can also issue scp commands on your local client that use Stampede2 environment variables like $HOME, $WORK, and $SCRATCH. To do so, use a backslash ("\") as an escape character before the "$"; this ensures that expansion occurs after establishing the connection to Stampede2:

localhost$ scp ./myfile bjones@stampede2.tacc.utexas.edu:\$WORK/data   # Note backslash

Avoid using scp for recursive ("-r") transfers of directories that contain nested directories of many small files:

localhost$ scp -r  ./mydata     bjones@stampede2.tacc.utexas.edu:\$WORK  # DON'T DO THIS

Instead, use tar to create an archive of the directory, then transfer the directory as a single file:

localhost$ tar cvf ./mydata.tar mydata                                   # create archive
localhost$ scp     ./mydata.tar bjones@stampede2.tacc.utexas.edu:\$WORK  # transfer archive

Transfer Using rsync

The rsync (remote synchronization) utility is a great way to synchronize files that you maintain on more than one system: when you transfer files using rsync, the utility copies only the changed portions of individual files. As a result, rsync is especially efficient when you only need to update a small fraction of a large dataset. The basic syntax is similar to scp:

localhost$ rsync       mybigfile bjones@stampede2.tacc.utexas.edu:\$WORK/data
localhost$ rsync -avtr mybigdir  bjones@stampede2.tacc.utexas.edu:\$WORK/data

The options on the second transfer are typical and appropriate when synching a directory: this is a recursive update ("-r") with verbose ("-v") feedback; the synchronization preserves time stamps ("-t") as well as symbolic links and other meta-data ("-a"). Because rsync only transfers changes, recursive updates with rsync may be less demanding than an equivalent recursive transfer with scp.

See Striping Large Files for additional important advice about striping the receiving directory when transferring or creating large files on TACC systems.

As detailed in the Citizenship section above, it is important to monitor your quotas on the $HOME and $WORK file systems, and limit the number of simultaneous transfers. Remember also that $STOCKYARD (and your $WORK directory on each TACC resource) is available from several other TACC systems: there's no need for scp when both the source and destination involve sub-directories of $STOCKYARD. See Managing Your Files for more information about transfers on $STOCKYARD.

Transfer Using Globus

Globus is another way for XSEDE users to transfer data between XSEDE sites; see Globus at XSEDE and Data Transfer and Management for more information. You can also use Globus if you're affiliated with an institution like the University of Texas that provides access to CILogin.

Sharing Files with Collaborators

If you wish to share files and data with collaborators in your project, see Sharing Project Files on TACC Systems for step-by-step instructions. Project managers or delegates can use Unix group permissions and commands to create read-only or read-write shared workspaces that function as data repositories and provide a common work area to all project members.

Basics of Building Software

This section of the user guide does nothing more than introduce the big ideas with simple one-line examples. You will undoubtedly want to explore these concepts more deeply using online resources. You will quickly outgrow the examples here. We recommend that you master the basics of makefiles as quickly as possible: even the simplest computational research project will benefit enormously from the power and flexibility of a makefile-based build process.

$ icc mycode.c                    # C source file; executable a.out
$ icc main.c calc.c analyze.c     # multiple source files
$ icc mycode.c     -o myexe       # C source file; executable myexe
$ icpc mycode.cpp  -o myexe       # C++ source file
$ ifort mycode.f90 -o myexe       # Fortran90 source file

$ icc -qopenmp mycode.c -o myexe  # OpenMP

$ gcc mycode.c                    # C source file; executable a.out
$ gcc mycode.c          -o myexe  # C source file; executable myexe
$ g++ mycode.cpp        -o myexe  # C++ source file
$ gfortran mycode.f90   -o myexe  # Fortran90 source file
$ gcc -fopenmp mycode.c -o myexe  # OpenMP; GNU flag is different than Intel

$ icc -c main.c calc.c results.c

$ icc main.o calc.o results.o -o myexe

$ icc        -c main.c -I${WORK}/mylib/inc -I${TACC_HDF5_INC}                  # compile
$ icc main.o -o myexe  -L${WORK}/mylib/lib -L${TACC_HDF5_LIB} -lmylib -lhdf5   # link

$ mpicc    mycode.c   -o myexe   # C source, full build
$ mpicc -c mycode.c              # C source, compile without linking
$ mpicxx   mycode.cpp -o myexe   # C++ source, full build
$ mpif90   mycode.f90 -o myexe   # Fortran source, full build

$ mpicc -show  # Show compile line generated by call to mpicc; similarly for other wrappers

$ export INSTALLDIR=$WORK/apps/t3pio
$ ./configure --prefix=$INSTALLDIR
$ make
$ make install

$ pip install netCDF4     --user                    # install netCDF4 package to $HOME/.local
$ python setup.py install --user                    # install to $HOME/.local
$ pip install netCDF4     --prefix=$INSTALLDIR      # custom location; add to PYTHONPATH

$ module load Rstats            # load TACC's default R
$ R                             # launch R
> install.packages('devtools')  # R will prompt for install location

$ icc   -mkl mycode.c
$ ifort -mkl mycode.c

$ module load gcc
$ module load mkl                         # available/needed only for GNU compilers
$ gcc -fopenmp -I$MKLROOT/include         \
         -Wl,-L${MKLROOT}/lib/intel64     \
         -lmkl_intel_lp64 -lmkl_core      \
         -lmkl_gnu_thread -lpthread       \
         -lm -ldl mycode.c

Building for Performance on Stampede2

$ icc   -xMIC-AVX512  -O3 mycode.c   -o myexe         # will run only on KNL

$ ifort -xCORE-AVX512 -O3 mycode.f90 -o myexe         # will run only on SKX

-xCORE-AVX2 -axCORE-AVX512,MIC-AVX512

$ icc -xCORE-AVX2 -axCORE-AVX512,MIC-AVX512 -O3 mycode.c -o myexe

$ icc -xCORE-AVX512 -axMIC-AVX512 -O3 mycode.c -o myexe       # NO! Base incompatible with alternate

$ echo $TACC_VEC_FLAGS                         # env variable available only for intel/18.0.0 and later
-xCORE-AVX2 -axCORE-AVX512,MIC-AVX512
$ icc $TACC_VEC_FLAGS -O3 mycode.c -o myexe

Job Accounting

Like all TACC systems, Stampede2's accounting system is based on node-hours: one unadjusted Service Unit (SU) represents a single compute node used for one hour (a node-hour). For any given job, the total cost in SUs is the use of one compute node for one hour of wall clock time plus any additional charges for the use of specialized queues, e.g. Stampede2's largemem queue, Lonestar5's gpu queue and Longhorn's v100 queue. The queue charge rates are determined by the supply and demand for that particular queue or type of node used.

Stampede2 SUs billed = (# nodes) x (job duration in wall clock hours) x (charge rate per node-hour)

The Slurm scheduler tracks and charges for usage to a granularity of a few seconds of wall clock time. The system charges only for the resources you actually use, not those you request. If your job finishes early and exits properly, Slurm will release the nodes back into the pool of available nodes. Your job will only be charged for as long as you are using the nodes.

TACC does not implement node-sharing on any compute resource. Each Stampede2 node can be assigned to only one user at a time; hence a complete node is dedicated to a user's job and accrues wall-clock time for all the node's cores whether or not all cores are used.

Tip: Your queue wait times will be less if you request only the time you need: the scheduler will have a much easier time finding a slot for the 2 hours you really need than say, for the 12 hours requested in your job script.

Principal Investigators can monitor allocation usage via the TACC User Portal under "Allocations->Projects and Allocations". Be aware that the figures shown on the portal may lag behind the most recent usage. Projects and allocation balances are also displayed upon command-line login.

To display a summary of your TACC project balances and disk quotas at any time, execute:

login1$ /usr/local/etc/taccinfo        # Generally more current than balances displayed on the portals.

Slurm Job Scheduler

Stampede2's job scheduler is the Slurm Workload Manager. Slurm commands enable you to submit, manage, monitor, and control your jobs.

Slurm Partitions (Queues)

Currently available queues include those in Stampede2 Production Queues. See KNL Compute Nodes, SKX Compute Nodes, Memory Modes, and Cluster Modes for more information on node types.

Table 5. Stampede2 Production Queues

Submitting Batch Jobs with sbatch

Use Slurm's "sbatch" command to submit a batch job to one of the Stampede2 queues:

login1$ sbatch myjobscript

Here "myjobscript" is the name of a text file containing #SBATCH directives and shell commands that describe the particulars of the job you are submitting. The details of your job script's contents depend on the type of job you intend to run.

In your job script you (1) use #SBATCH directives to request computing resources (e.g. 10 nodes for 2 hrs); and then (2) use shell commands to specify what work you're going to do once your job begins. There are many possibilities: you might elect to launch a single application, or you might want to accomplish several steps in a workflow. You may even choose to launch more than one application at the same time. The details will vary, and there are many possibilities. But your own job script will probably include at least one launch line that is a variation of one of the examples described here.

Launching Applications

The primary purpose of your job script is to launch your research application. How you do so depends on several factors, especially (1) the type of application (e.g. MPI, OpenMP, serial), and (2) what you're trying to accomplish (e.g. launch a single instance, complete several steps in a workflow, run several applications simultaneously within the same job). While there are many possibilities, your own job script will probably include a launch line that is a variation of one of the examples described in this section:

• Launching One Serial Application
• Launching One Multi-Threaded Application
• Launching One MPI Application
• Launching One Hybrid (MPI+Threads) Application
• More Than One Serial Application in the Same Job
• More than One MPI Application Running Concurrently
• More than One OpenMP Application Running Concurrently

mycode.exe                   # executable in a directory listed in $PATH
$WORK/apps/myprov/mycode.exe # explicit full path to executable
./mycode.exe                 # executable in current directory
./mycode.exe -m -k 6 input1  # executable with notional input options

export OMP_NUM_THREADS=68    # 68 total OpenMP threads (1 per KNL core)
./mycode.exe

ibrun ./mycode.exe           # use ibrun instead of mpirun or mpiexec

export OMP_NUM_THREADS=8    # 8 OpenMP threads per MPI rank
ibrun ./mycode.exe          # use ibrun instead of mpirun or mpiexec

module load git
module list
./preprocess.sh
ibrun ./mycode.exe input1   # runs after preprocess.sh completes
ibrun ./mycode.exe input2   # runs after previous MPI app completes

ibrun -n 64 -o  0 task_affinity ./mycode.exe input1 &   # 64 tasks; offset by  0 entries in hostfile.
ibrun -n 64 -o 64 task_affinity ./mycode.exe input2 &   # 64 tasks; offset by 64 entries in hostfile.
wait                                                    # Required; else script will exit immediately.

export OMP_NUM_THREADS=2
numactl -C 0-1 ./mycode.exe inputfile1 &  # HW threads (hence cores) 0-1. Note ampersand.
numactl -C 2-3 ./mycode.exe inputfile2 &  # HW threads (hence cores) 2-3. Note ampersand.

wait

Interactive Sessions with idev and srun

TACC's own idev utility is the best way to begin an interactive session on one or more compute nodes. To launch a thirty-minute session on a single node in the development queue, simply execute:

login1$ idev

You'll then see output that includes the following excerpts:

...
-----------------------------------------------------------------
      Welcome to the Stampede2 Supercomputer          
-----------------------------------------------------------------
...

-> After your idev job begins to run, a command prompt will appear,
-> and you can begin your interactive development session. 
-> We will report the job status every 4 seconds: (PD=pending, R=running).

->job status:  PD
->job status:  PD
...
c449-001$

The "job status" messages indicate that your interactive session is waiting in the queue. When your session begins, you'll see a command prompt on a compute node (in this case, the node with hostname c449-001). If this is the first time you launch idev, the prompts may invite you to choose a default project and a default number of tasks per node for future idev sessions.

For command line options and other information, execute "idev --help". It's easy to tailor your submission request (e.g. shorter or longer duration) using Slurm-like syntax:

login1$ idev -p normal -N 2 -n 8 -m 150 # normal queue, 2 nodes, 8 total tasks, 150 minutes

For more information see the idev documentation.

You can also launch an interactive session with Slurm's srun command, though there's no clear reason to prefer srun to idev. A typical launch line would look like this:

login1$ srun --pty -N 2 -n 8 -t 2:30:00 -p normal /bin/bash -l # same conditions as above

Interactive Sessions using ssh

If you have a batch job or interactive session running on a compute node, you "own the node": you can connect via ssh to open a new interactive session on that node. This is an especially convenient way to monitor your applications' progress. One particularly helpful example: login to a compute node that you own, execute "top", then press the "1" key to see a display that allows you to monitor thread ("CPU") and memory use.

There are many ways to determine the nodes on which you are running a job, including feedback messages following your sbatch submission, the compute node command prompt in an idev session, and the squeue or showq utilities. The sequence of identifying your compute node then connecting to it would look like this:

login1$ squeue -u bjones
 JOBID       PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
858811     development idv46796   bjones  R       0:39      1 c448-004
1ogin1$ ssh c448-004
...
C448-004$

SLURM Environment Variables

Be sure to distinguish between internal Slurm replacement symbols (e.g. "%j" described above) and Linux environment variables defined by Slurm (e.g. SLURM_JOBID). Execute "env | grep SLURM" from within your job script to see the full list of Slurm environment variables and their values. You can use Slurm replacement symbols like "%j" only to construct a Slurm filename pattern; they are not meaningful to your Linux shell. Conversely, you can use Slurm environment variables in the shell portion of your job script but not in an #SBATCH directive. For example, the following directive will not work the way you might think:

#SBATCH -o myMPI.o${SLURM_JOB_ID}   # incorrect

Instead, use the following directive:

#SBATCH -o myMPI.o%j     # "%j" expands to your job's numerical job ID

Similarly, you cannot use paths like $WORK or $SCRATCH in an #SBATCH directive.

For more information on this and other matters related to Slurm job submission, see the Slurm online documentation; the man pages for both Slurm itself ("man slurm") and its individual command (e.g. "man sbatch"); as well as numerous other online resources.

Monitoring Queue Status with sinfo and qlimits

To display resource limits for the Stampede2 queues, execute "qlimits". The result is real-time data; the corresponding information in this document's table of Stampede2 queues may lag behind the actual configuration that the qlimits utility displays.

Slurm's "sinfo" command allows you to monitor the status of the queues. If you execute sinfo without arguments, you'll see a list of every node in the system together with its status. To skip the node list and produce a tight, alphabetized summary of the available queues and their status, execute:

login1$ sinfo -S+P -o "%18P %8a %20F"    # compact summary of queue status

An excerpt from this command's output looks like this:

PARTITION          AVAIL    NODES(A/I/O/T)
development*       up       41/70/1/112
normal             up       3685/8/3/3696

The AVAIL column displays the overall status of each queue (up or down), while the column labeled "NODES(A/I/O/T)" shows the number of nodes in each of several states ("Allocated", "Idle", "Other", and "Total"). Execute "man sinfo" for more information. Use caution when reading the generic documentation, however: some available fields are not meaningful or are misleading on Stampede2 (e.g. TIMELIMIT, displayed using the "%l" option).

Monitoring Job Status with squeue

Slurm's squeue command allows you to monitor jobs in the queues, whether pending (waiting) or currently running:

login1$ squeue             # show all jobs in all queues
login1$ squeue -u bjones   # show all jobs owned by bjones
login1$ man squeue         # more info

An excerpt from the default output looks like this:

 JOBID   PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
170361      normal   spec12   bjones PD       0:00     32 (Resources)
170356      normal    mal2d slindsey PD       0:00     30 (Priority)
170204      normal   rr2-a2 tg123456 PD       0:00      1 (Dependency)
170250 development idv59074  aturing  R      29:30      1 c455-044
169669      normal  04-99a1  aturing CG    2:47:47      1 c425-003

The column labeled "ST" displays each job's status:

• "PD" means "Pending" (waiting);
• "R" means "Running";
• "CG" means "Completing" (cleaning up after exiting the job script).

Pending jobs appear in order of decreasing priority. The last column includes a nodelist for running/completing jobs, or a reason for pending jobs. If you submit a job before a scheduled system maintenance period, and the job cannot complete before the maintenance begins, your job will run when the maintenance/reservation concludes. The squeue command will report "ReqNodeNotAvailable" ("Required Node Not Available"). The job will remain in the PD state until Stampede2 returns to production.

The default format for squeue now reports total nodes associated with a job rather than cores, tasks, or hardware threads. One reason for this change is clarity: the operating system sees each KNL node's 272 hardware threads (and each SKX node's 96 hardware threads) as "processors", and output based on that information can be ambiguous or otherwise difficult to interpret.

The default format lists all nodes assigned to displayed jobs; this can make the output difficult to read. A handy variation that suppresses the nodelist is:

login1$ squeue -o "%.10i %.12P %.12j %.9u %.2t %.9M %.6D"  # suppress nodelist

The "--start" option displays job start times, including very rough estimates for the expected start times of some pending jobs that are relatively high in the queue:

login1$ squeue --start -j 167635     # display estimated start time for job 167635

Monitoring Job Status with showq

TACC's "showq" utility mimics a tool that originated in the PBS project, and serves as a popular alternative to the Slurm "squeue" command:

login1$ showq            # show all jobs; default format
login1$ showq -u         # show your own jobs
login1$ showq -U bjones  # show jobs associated with user bjones
login1$ showq -h         # more info

The output groups jobs in four categories: ACTIVE, WAITING, BLOCKED, and COMPLETING/ERRORED. A BLOCKED job is one that cannot yet run due to temporary circumstances (e.g. a pending maintenance or other large reservation.).

If your waiting job cannot complete before a maintenance/reservation begins, showq will display its state as "WaitNod" ("Waiting for Nodes"). The job will remain in this state until Stampede2 returns to production.

The default format for showq now reports total nodes associated with a job rather than cores, tasks, or hardware threads. One reason for this change is clarity: the operating system sees each KNL node's 272 hardware threads (and each SKX node's 96 hardware threads) as "processors", and output based on that information can be ambiguous or otherwise difficult to interpret.

Other Job Management Commands (scancel, scontrol, and sacct)

It's not possible to add resources to a job (e.g. allow more time) once you've submitted the job to the queue.

To cancel a pending or running job, first determine its jobid, then use scancel:

login1$ squeue -u bjones    # one way to determine jobid
   JOBID   PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
  170361      normal   spec12   bjones PD       0:00     32 (Resources)
login1$ scancel 170361      # cancel job

For detailed information about the configuration of a specific job, use scontrol:

login1$ scontrol show job=170361

To view some accounting data associated with your own jobs, use sacct:

login1$ sacct --starttime 2017-08-01  # show jobs that started on or after this date

Dependent Jobs using sbatch

You can use sbatch to help manage workflows that involve multiple steps: the "--dependency" option allows you to launch jobs that depend on the completion (or successful completion) of another job. For example you could use this technique to split into three jobs a workflow that requires you to (1) compile on a single node; then (2) compute on 40 nodes; then finally (3) post-process your results using 4 nodes.

login1$ sbatch --dependency=afterok:173210 myjobscript

For more information see the Slurm online documentation. Note that you can use $SLURM_JOBID from one job to find the jobid you'll need to construct the sbatch launch line for a subsequent one. But also remember that you can't use sbatch to submit a job from a compute node.

Remote Desktop Access

Remote desktop access to Stampede2 is formed through a VNC connection to one or more visualization nodes. Users must first connect to a Stampede2 login node (see System Access) and submit a special interactive batch job that:

• allocates a set of Stampede2 visualization nodes
• starts a vncserver process on the first allocated node
• sets up a tunnel through the login node to the vncserver access port

Once the vncserver process is running on the visualization node and a tunnel through the login node is created, an output message identifies the access port for connecting a VNC viewer. A VNC viewer application is run on the user's remote system and presents the desktop to the user.

Note: If this is your first time connecting to Stampede2, you must run vncpasswd to create a password for your VNC servers. This should NOT be your login password! This mechanism only deters unauthorized connections; it is not fully secure, as only the first eight characters of the password are saved. All VNC connections are tunneled through SSH for extra security, as described below.

Follow the steps below to start an interactive session.

1. Start a Remote Desktop

   TACC has provided a VNC job script (/share/doc/slurm/job.vnc) that requests one node in the development queue for two hours, creating a VNC session.

   login1$ sbatch /share/doc/slurm/job.vnc

   You may modify or overwrite script defaults with sbatch command-line options:

   • "-t hours:minutes:seconds" modify the job runtime
   • "-A projectnumber" specify the project/allocation to be charged
   • "-N nodes" specify number of nodes needed
   • "-p partition" specify an alternate queue.

   See more sbatch options in the Common sbatch Options

   All arguments after the job script name are sent to the vncserver command. For example, to set the desktop resolution to 1440x900, use:

   login1$ sbatch /share/doc/slurm/job.vnc -geometry 1440x900

   The "vnc.job" script starts a vncserver process and writes to the output file, "vncserver.out" in the job submission directory, with the connect port for the vncviewer. Watch for the "To connect via VNC client" message at the end of the output file, or watch the output stream in a separate window with the commands:

   login1$ touch vncserver.out ; tail -f vncserver.out

   The lightweight window manager, xfce, is the default VNC desktop and is recommended for remote performance. Gnome is available; to use gnome, open the "~/.vnc/xstartup" file (created after your first VNC session) and replace "startxfce4" with "gnome-session". Note that gnome may lag over slow internet connections.

2. Create an SSH Tunnel to Stampede2

   TACC requires users to create an SSH tunnel from the local system to the Stampede2 login node to assure that the connection is secure. On a Unix or Linux system, execute the following command once the port has been opened on the Stampede2 login node:

    localhost$ ssh -f -N -L xxxx:stampede2.tacc.utexas.edu:yyyy
          username@stampede2.tacc.utexas.edu

   where

   • "yyyy" is the port number given by the vncserver batch job
   • "xxxx" is a port on the remote system. Generally, the port number specified on the Stampede2 login node, yyyy, is a good choice to use on your local system as well
   • "-f" instructs SSH to only forward ports, not to execute a remote command
   • "-N" puts the ssh command into the background after connecting
   • "-L" forwards the port

   On Windows systems find the menu in the Windows SSH client where tunnels can be specified, and enter the local and remote ports as required, then ssh to Stampede2.

3. Connecting vncviewer

   Once the SSH tunnel has been established, use a VNC client to connect to the local port you created, which will then be tunneled to your VNC server on Stampede2. Connect to localhost:xxxx, where xxxx is the local port you used for your tunnel. In the examples above, we would connect the VNC client to localhost::xxxx. (Some VNC clients accept localhost:xxxx).

   We recommend the TigerVNC VNC Client, a platform independent client/server application.

   Once the desktop has been established, two initial xterm windows are presented (which may be overlapping). One, which is white-on-black, manages the lifetime of the VNC server process. Killing this window (typically by typing "exit" or "ctrl-D" at the prompt) will cause the vncserver to terminate and the original batch job to end. Because of this, we recommend that this window not be used for other purposes; it is just too easy to accidentally kill it and terminate the session.

   The other xterm window is black-on-white, and can be used to start both serial programs running on the node hosting the vncserver process, or parallel jobs running across the set of cores associated with the original batch job. Additional xterm windows can be created using the window-manager left-button menu.

Running Applications on the VNC Desktop

From an interactive desktop, applications can be run from icons or from xterm command prompts. Two special cases arise: running parallel applications, and running applications that use OpenGL.

Running Parallel Applications from the Desktop

Parallel applications are run on the desktop using the same ibrun wrapper described above (see Running). The command:

c442-001$ ibrun ibrunoptions application applicationoptions

will run application on the associated nodes, as modified by the ibrun options.

Running OpenGL/X Applications On The Desktop

Stampede2 uses the OpenSWR OpenGL library to perform efficient rendering. At present, the compute nodes on Stampede2 do not support native X instances. All windowing environments should use a VNC desktop launched via the job script in /share/doc/slurm/job.vnc or using the TACC Vis portal.

swr: To access the accelerated OpenSWR OpenGL library, it is necessary to use the swr module to point to the swr OpenGL implementation and configure the number of threads to allocate to rendering.

c442-001$ module load swr
c442-001$ swr options application application-args

Parallel VisIt on Stampede2

VisIt was compiled under the Intel compiler and the mvapich2 and MPI stacks.

After connecting to a VNC server on Stampede2, as described above, load the VisIt module at the beginning of your interactive session before launching the Visit application:

c442-001$ module load swr visit
c442-001$ swr visit

VisIt first loads a dataset and presents a dialog allowing for selecting either a serial or parallel engine. Select the parallel engine. Note that this dialog will also present options for the number of processes to start and the number of nodes to use; these options are actually ignored in favor of the options specified when the VNC server job was started.

Parallel ParaView on Stampede2

After connecting to a VNC server on Stampede2, as described above, do the following:

1. Set up your environment with the necessary modules. Load the swr, qt5, ospray, and paraview modules in this order:

   c442-001$ module load swr qt5 ospray paraview

2. Launch ParaView:

    c442-001$ swr -p 1 paraview [paraview client options]

3. Click the "Connect" button, or select File -> Connect

4. Select the "auto" configuration, then press "Connect". In the Paraview Output Messages window, you'll see what appears to be an ‘lmod' error, but can be ignored. Then you'll see the parallel servers being spawned and the connection established.

Programming and Performance: General

Programming for performance is a broad and rich topic. While there are no shortcuts, there are certainly some basic principles that are worth considering any time you write or modify code.

perf stat ./a.out    # report basic performance stats for a.out
time ./a.out         # report the time required to execute a.out
time ibrun ./a.out   # time an MPI code
ibrun time ./a.out   # crude timings for each MPI task (no rank info)

real*8 :: a(m,n), b(m,n), c(m,n)
 ...
! inner loop strides through col i
do i=1,n
  do j=1,m
    a(j,i)=b(j,i)+c(j,i)
  end do
end do

double a[m][n], b[m][n], c[m][n];
 ...
// inner loop strides through row i
for (i=0;i<m;i++){
  for (j=0;j<n;j++){
    a[i][j]=b[i][j]+c[i][j];
  }
}

Programming and Performance: KNL

numactl       --membind=0    ./a.out    # launch a.out (non-MPI); use DDR (default)
ibrun numactl --membind=0    ./a.out    # launch a.out (MPI-based); use DDR (default)

numactl       --membind=1    ./a.out    # use only MCDRAM
numactl       --preferred=1  ./a.out    # (RECOMMENDED) MCDRAM if possible; else DDR
numactl       --hardware                # show numactl settings
numactl       --help                    # list available numactl options

    ibrun mem_affinity a.out

Programming and Performance: SKX

Hyperthreading. It is rarely a good idea to use 96 hardware threads simultaneously, and it's certainly not the first thing you should try. In most cases it's best to specify no more than 48 MPI tasks or independent processes per node, and 1-2 threads/core. One exception is worth noting: when calling threaded MKL from a serial code, it's safe to set OMP_NUM_THREADS or MKL_NUM_THREADS to 96. This is because MKL will choose an appropriate thread count less than or equal to the value you specify. See Controlling Threading in MKL for more information. In any case remember that the default value of OMP_NUM_THREADS is 1.

Clock Speed. The published nominal clock speed of the Stampede2 SKX processors is 2.1GHz. But actual clock speed varies widely: it depends on the vector instruction set, number of active cores, and other factors affecting power requirements and temperature limits. At one extreme, a single serial application using the AVX2 instruction set may run at frequencies approaching 3.7GHz, because it's running on a single core (in fact a single hardware thread). At the other extreme, a large, fully-threaded MKL dgemm (a highly vectorized routine in which all cores operate at nearly full throttle) may run at 1.4GHz.

Vector Optimization and AVX2. In some cases, using the AVX2 instruction set may produce better performance than AVX512. This is largely because cores can run at higher clock speeds when executing AVX2 code. To compile for AVX2, replace the multi-architecture flags described above with the single flag "-xCORE-AVX2". When you use this flag you will be able to build and run on any Stampede2 node.

Vector Optimization and 512-Bit ZMM Registers. If your code can take advantage of wide 512-bit vector registers, you may want to try compiling for SKX with (for example):

-xCORE-AVX512 -qopt-zmm-usage=high

The "qopt-zmm-usage" flag affects the algorithms the compiler uses to decide whether to vectorize a given loop with AVX51 intrinsics (wide 512-bit registers) or AVX2 code (256-bit registers). When the flag is set to "-qopt-zmm-usage=low" (the default when compiling for the SKX using CORE-AVX512), the compiler will choose AVX2 code more often; this may or may not be the optimal approach for your application. The qopt-zmm-usage flag is available only on Intel compilers newer than 17.0.4. Do not use $TACC_VEC_FLAGS when specifying qopt-zmm-usage. This is because $TACC_VEC_FLAGS specifies AVX2-CORE as the base architecture, and the compiler will ignore qopt-zmm-usage unless the base target is a variant of AVX512. See the recent Intel white paper, the compiler documentation, the compiler man pages, and the notes above for more information.

Vector Optimization and COMMON-AVX512. We have encountered a few complex packages that currently fail to build or run when compiled with CORE-AVX512 (native SKX). In all cases so far, these packages build and run well on both KNL and SKX when compiled as a single-architecture binary with -xCOMMON-AVX512.

Task Affinity. If you run one MPI application at a time, the ibrun MPI launcher will spread each node's tasks evenly across an SKX node's two sockets, with consecutive tasks occupying the same socket when possible.

Hardware Thread Numbering. Execute "lscpu" or "lstopo" on an SKX node to see the numbering scheme for hardware threads. Note that hardware thread numbers alternate between the sockets: even numbered threads are on NUMA node 0, while odd numbered threads are on NUMA node 1. Furthermore, the two hardware threads on a given core have thread numbers that differ by exactly 48 (e.g. threads 3 and 51 are on the same core).

Tuning the Performance Scaled Messaging (PSM2) Library. When running on SKX with MVAPICH2, setting the environment variable PSM2_KASSIST_MODE to the value "none" may or may not improve performance. For more information see the MVAPICH2 User Guide. Do not use this environment variable with IMPI; doing so may degrade performance. The ibrun launcher will eventually control this environment variable automatically.

File Operations: I/O Performance

This section includes general advice intended to help you achieve good performance during file operations. See Navigating the Shared File Systems for a brief overview of Stampede2's Lustre file systems and the concept of striping. See TACC Training material for additional information on I/O performance.

Follow the advice in Good Citizenship to avoid stressing the file system.

Stripe for performance. If your application writes large files using MPI-based parallel I/O (including MPI-IO, parallel HDF5, and parallel netCDF, you should experiment with stripe counts larger than the default values (2 stripes on $SCRATCH, 1 stripe on $WORK). See Striping Large Files for the simplest way to set the stripe count on the directory in which you will create new output files. You may also want to try larger stripe sizes up to 16MB or even 32MB; execute "man lfs" for more information. If you write many small files you should probably leave the stripe count at its default value, especially if you write each file from a single process. Note that it's not possible to change the stripe parameters on files that already exist. This means that you should make decisions about striping when you create input files, not when you read them.

Aggregate file operations. Open and close files once. Read and write large, contiguous blocks of data at a time; this requires understanding how a given programming language uses memory to store arrays.

Be smart about your general strategy. When possible avoid an I/O strategy that requires each process to access its own files; such strategies don't scale well and are likely to stress a Lustre file system. A better approach is to use a single process to read and write files. Even better is genuinely parallel MPI-based I/O.

Use parallel I/O libraries. Leave the details to a high performance package like MPI-IO (built into MPI itself), parallel HDF5 ("module load phdf5"), and parallel netCDF ("module load pnetcdf").

When using the Intel Fortran compiler, compile with "-assume buffered_io". Equivalently, set the environment variable FORT_BUFFERED=TRUE. Doing otherwise can dramatically slow down access to variable length unformatted files. More generally, direct access in Fortran is typically faster than sequential access, and accessing a binary file is faster than ASCII.

• 04/24/18 Changes to Table 1 and Table 5 associated with new long queue.
• 04/03/18 Stampede1 decommissioned; removed/revised references to Stampede1 as appropriate.
• 03/26/18 Corrected and relocated material on qopt-zmm-usage.
• 02/23/18 New functionality associated with task_affinity, tacc_affinity, and mem_affinity (scripts related to MPI task pinning and KNL memory management).
• 11/30/17 Initial release supporting Phase 2 (SKX).
• 08/02/17 Removed references and links to Stampede2 Transition Guide (now deprecated).
• 06/12/17 Initial public release.