« Previous - Version 31/36 (diff) - Next » - Current version
Gerrett Diamond, 06/09/2014 09:36 am


Building and Running for the Intel Phi on Stampede

This page covers the process of building and running the SCOREC software on Stampede and specifically for the Intel Phi. This is a page in progress.

Building for the Phi

Check out the scorec software. This can be done most easily using the command:

svn co http://redmine.scorec.rpi.edu/svn/buildutil/trunk/cmake/SCOREC

More on building the SCOREC software stack can be found at:
http://redmine.scorec.rpi.edu/projects/pumi/wiki/Build_Instructions#Build-from-SVN

Stampede uses the modules system for software and libraries. For more on the modules system and the compute environment see:
http://www.tacc.utexas.edu/user-services/user-guides/stampede-user-guide#compenv-modules
Before building can commence a few must be loaded

  • Currently only the Intel implementation of MPI supports building for the Phi coprocessors. However mpich is the default. The intel libraries can be loaded with the command:
    module swap mvapich2 impi 
    
  • By default only cmake 2.6 is loaded. Cmake 2.8 requires loading the cmake module:
    module load cmake
    
  • Observant users may notice the metis, pmetis, and trilinos. Unfortunately these were not compiled for the Phi. Any attempts to link to these will fail.

To cross-compile for the Phi, use the compiler flag -mmic. Also, the preprocessor variables MPICH_SKIP_MPICXX and MPICH_IGNORE_CXX_SEEK must be set for proper compilation. Note that linking with '-mt_mpi' is required to enable MPI_THREAD_MULTIPLE support.

export CMAKE_PREFIX_PATH=$CMAKE_PREFIX_PATH:/work/02422/cwsmith/shared/installPhi
mkdir build
cd build
cmake \
-DCMAKE_INSTALL_PREFIX=/path/to/install/dir \
-DCMAKE_CXX_COMPILER=`which mpicxx` \
-DCMAKE_C_COMPILER=`which mpicc` \
-DCMAKE_BUILD_TYPE=None \
-DCMAKE_EXE_LINKER_FLAGS='-mt_mpi ' \
-DCMAKE_C_FLAGS='-mmic -g ' \
-DCMAKE_CXX_FLAGS="-mmic -g -DMPICH_SKIP_MPICXX -DMPICH_IGNORE_CXX_SEEK" \
-DENABLE_MA2=ON \
-DENABLE_THREADS=ON \
../
make install -j 8
make testParmaHybrid ma_test ma_colorado -j 3

Running on the Phi

Stampede uses the SLURM job scheduler. For detailed coverage on running jobs and the available queues on Stampede see:
http://www.tacc.utexas.edu/user-services/user-guides/stampede-user-guide#running
To run tests exclusively on the phi, run in symmetric mode supplying only the Phi binary. For example:

ibrun.symm -m "<binary> <args>" 

Download the meshes repo

svn co http://redmine.scorec.rpi.edu/anonsvn/meshes svnMeshes

Below is an example run script, 'testPumi.mic', for running parma hybrid (testParmaHybrid) and mesh adapt (ma_test and ma_colorado) on various number of MPI processes (controlled by the MIC_PPN variable multiplied by the -N variable):

#!/bin/bash -ex
#SBATCH -J testPumi
#SBATCH -o testPumi.%j.out
#SBATCH -p development
#SBATCH -N 1
#SBATCH -n 1
#SBATCH -t 00:10:00
#SBATCH -A TG-ASC130020

build=/path/to/build/dir
meshes=/path/to/svnMeshes

parma=$build/parma/testParmaHybrid
torus4=$meshes/torus/4imb/torus.smb
torusMdl=$meshes/torus/torus.dmg
export MIC_PPN=4
/usr/bin/time -p ibrun.symm  -m "$parma -m $torus4 -g $torusMdl" 

ma_test=$build/ma/ma_test
cube4=$meshes/cube/pumi7k/4/cube.smb
cubeMdl=$meshes/cube/cube.dmg
export MIC_PPN=4
/usr/bin/time -p ibrun.symm  -m "$ma_test $cubeMdl $cube4" 

ma_colorado=$build/ma/ma_colorado
match=$meshes/match/match.smb
matchMdl=$meshes/match/match.dmg
export MIC_PPN=1
/usr/bin/time -p ibrun.symm  -m "$ma_colorado $matchMdl $match 2.smb 1 2" 

Run the script with:

  sbatch ./testPumi.mic

Debugging

The ddt debugger is provided and can be used with the Phis
Login with X enabled
In order to get an interactive session, use:

 ~cazes/bin/idev [-A <project id>] 

NOTE: The interactive session will fail to source your .bashrc unless the call to idev was made from the home directory.

To debug a program on the Phi:

source ~cazes/setup_ddt_mic 
ddt <binary> [<args>]