« Previous - Version 2/36 (diff) - Next » - Current version
Micah Corah, 11/18/2013 02:23 pm


Building and Running for the Intel Phi on Stampede

This page covers the process of building and running the SCOREC software on Stampede and specifically for the Intel Phi. This is a page in progress.

Building for the Phi

Check out the scorec software. This can be done most easily using the command:

svn co http://redmine.scorec.rpi.edu/svn/buildutil/trunk/cmake/SCOREC

More on building the SCOREC software stack can be found at:
http://redmine.scorec.rpi.edu/projects/pumi/wiki/Build_Instructions#Build-from-SVN

Stampede uses the modules system for software and libraries. For more on the modules system and the compute environment see:
http://www.tacc.utexas.edu/user-services/user-guides/stampede-user-guide#compenv-modules
Before building can commence a few must be loaded

  • Currently only the Intel implementation of MPI supports building for the Phi coprocessors. However mpich is the default. The intel libraries can be loaded with the command:
    module swap mvapich2 impi 
    
  • By default only cmake 2.6 is loaded. Cmake 2.8 requires loading the cmake module:
    module load cmake
    
  • Observant users may notice the metis, pmetis, and trilinos. Unfortunately these were not compiled for the Phi. Any attempts to link to these will fail.

To cross-compile for the Phi, use the compiler flag -mmic. Also, the preprocessor variables MPICH_SKIP_MPICXX and MPICH_IGNORE_CXX_SEEK must be set for proper compilation. Some variation of the following commands will results in properly building the libraries:

mkdir build
cd build
cmake \
-DCMAKE_INSTALL_PREFIX=$PWD/../install \
-DCMAKE_CXX_COMPILER=`which mpicxx` \
-DCMAKE_C_COMPILER=`which mpicc` \
-DCMAKE_C_FLAGS=-mmic \
-DCMAKE_CXX_FLAGS="-mmic -DMPICH_SKIP_MPICXX -DMPICH_IGNORE_CXX_SEEK" \
-DENABLE_THREADS=true \
-DPARMA=false \
-DPUMI_GEOM_MODEL=NULL \
../
make zoltan
make install -j 8

Running on the Phi

Stampede uses the SLURM job scheduler. For detailed coverage on running jobs and the available queues on Stampede see:
http://www.tacc.utexas.edu/user-services/user-guides/stampede-user-guide#running
To run tests exclusively on the phi, run in symmetric mode supplying only the Phi binary. For example:
ibrun.symm -m "<binary> <args>" 

The Phi is configured via a number of environment variables
  • Number of MPI tasks: MIC_PPN
  • number of openmp threads per task: MIC_OMP_NUM_THREADS

Debugging
The ddt debugger is provided and can be used with the Phis
Login with X enabled
In order to get an interactive session, use:
 idev 

NOTE: The interactive session will fail to source your .bashrc unless the call to idev was made from the home directory.

To debug a program on the Phi:

source ~cazes/setup_ddt_mic 
ddt <binary> <args>