Build Instructions

OpenSpeedShop now builds with the spack package manager. A brief description of spack and how to use it to build OpenSpeedShop follows. This spack build instructions/description page is divided into these sections:

GENERAL
VARIANT DESCRIPTION/OVERVIEW
MODULE CREATION
EXTERNAL PACKAGING INSTRUCTIONS
RESTRICTED ACCESS / SPACK MIRROR

GENERAL:

spack is a multi-platform package manager that builds and installs multiple versions and configurations of software. It works on Linux, macOS, and many supercomputers. Spack is non-destructive: installing a new version of a package does not break existing installations, so many configurations of the same package can coexist. Most importantly, Spack is simple. It offers a simple spec syntax so that users can specify versions and configuration options concisely. Spack is also simple for package authors: package files are written in pure Python, and specs allow package authors to maintain a single file for many different builds of the same package. If you’re new to spack and want to start using it, see Getting Started, or refer to the full manual below.

Open|SpeedShop (O|SS) is an open source multi-platform performance tool enabling performance analysis of HPC applications running on both single node and large scale Intel, AMD, ARM, Intel Phi, Power PC, Power 8, GPU processor based systems, including Cray and IBM Blue Gene platforms.

O|SS can be built with spack by downloading the spack package source, as described below. It is possible to build O|SS with one spack install command. Spack will download, on the fly from the package websites, all the dependent packages that O|SS needs, build and install them. After the dependent packages are built, spack will download, build, and install O|SS. Spack creates module file for all the packages that it builds. The instructions below identify where to find the module file and how to load it.

To download and set up spack to be ready to build a package, follow these steps:

$ git clone https://github.com/spack/spack.git
$ cd spack
$ source share/spack/setup-env.sh

Now you are ready to use spack although, you on most systems it is recommended that you do a spack command to find the available compilers on the system:
$ spack compiler find

spack allows for enabling and disabling O|SS optional build arguments/parameters via a spack feature named variants. For O|SS, the main variants are MPI implementation identifiers which are used to build the MPI collectors for those MPI implementations. The spack build command:

$ spack install openspeedshop +openmpi +mvapich2

will build O|SS with all the collectors, including MPI collectors that will work on OpenMPI and Mvapich2 based applications. Without those variants, all the non-mpi specific experiments will be built, but not the MPI collectors (no ossmpi, ossmpip, and ossmpit support).

As an optimization to reduce the number of packages that need to be built by the spack install command, spack has a mechanism to specify package installation locations for packages already built and installed on the system. Telling spack where to find these already installed packages, like Qt and/or MPI installations, can save a lot of build time and also allow usage of standard package installation that non-spack built executables and tools are using. The spack mechanism for this feature is the packages.yaml file that you create in your $HOME/.spack/linux directory. This file contains a list of packages that you don’t want spack to build. The packages.yaml file is a list of package names and the directories where the package is already installed. See the example in the quick start guide section. This can save a lot of build time for packages like Qt, boost, python, mpi, etc… On a Cray system, packages.yaml should be created in the $HOME/.spack/cray directory. See the section, below, named EXTERNAL PACKAGING INSTRUCTIONS for an example of what the packages.yaml file contains.

Support for building O|SS on heterogeneous platforms is now supported as of February 2018 in the development branch of spack. Support for building on laptops, desktops, and homogeneous clusters (including Cray platforms) is working and available through the default and development version of spack.

There is now a way to have access to the latest development version of O|SS via the develop version specification when building O|SS with spack. To access the development version (master git tree), use:

$ spack install openspeedshop@develop [variants]

Using the @develop clause on the openspeedshop package build will also build newer versions of all the components that O|SS uses. Especially important are the cbtf packages, as they are now the core of performance data collection in O|SS.

Without the “@develop” on the openspeedshop By default, spack installs the newest/latest numbered version of a package. O|SS currently has numbered versions: 2.4.1, 2.4.0 and others. By default, the 2.4.1 version will be built when this command is used:

$ spack install openspeedshop [variants]

To build another, specific version, use the version number:

$ spack install openspeedshop@2.4.1 [variants]

VARIANT DESCRIPTION/OVERVIEW:

This is the list of spack packages related to the OpenSpeedShop project:

• openspeedshop
• cbtf
• cbtf-krell
• cbtf-argonavis
• cbtf-lanl
• cbtf-argonavis-gui
• qtgraph (support for the cbtf-argonavis-gui)

Component Variant Explanation:
Each of the OpenSpeedShop related supported packages allows a number of spack variants. A variant is a build option that the user can specify to control build options for the component. For example, if the user wants cbtf-krell to build openmpi MPI data collectors then they would specify the spack variant +openmpi on the spack install command line for building cbtf-krell, like this:

$ spack install cbtf-krell +openmpi

Internal to the cbtf-krell package file, this looks like this, where the variant is causing a cmake variable to be set:
if spec.satisfies(‘+openmpi’):
MPIOptions.append(‘-DOPENMPI_DIR=%s’ % spec[‘openmpi’].prefix)

Without the openmpi variant, the spack build for cbtf-krell would not build MPI experiment collectors for the openmpi MPI implementation.

The following tables list the component, variant name and explanation of the purpose for the variant.

openspeedshop Component Variant Explanation / Purpose

ComponentVariantExplanation / Purpose
openspeedshop cbtfBuild for the cbtf instrumentor (default)
crayfeBuild only the front-end tool using the runtime libraries from the compute node (heterogeneous builds.)
cudaBuild with cuda packages included.
guiBuild or not build a GUI of choice, options: none, qt3, qt4
runtimeBuild only the runtime libraries and collectors. Used for heterogeneous builds.
mpichBuild the mpich MPI implementation collectors, passed to cbtf-krell for actual build
mpt Build the mpt MPI implementation collectors, passed to cbtf-krell for actual build
mvapichBuild the mvapich MPI implementation collectors, passed to cbtf-krell for actual build
mvapich2Build the mvapich2 MPI implementation collectors, passed to cbtf-krell for actual build
openmpiBuild the openmpi MPI implementation collectors, passed to cbtf-krell for actual build

cbtf Component Variant Explanation / Purpose

ComponentVariant Explanation / Purpose
cbtfruntime Build only the runtime libraries and collectors. Used for heterogeneous builds.

cbtf-krell Component Variant Explanation / Purpose

ComponentVariantExplanation / Purpose
cbtf-krellcrayfeBuild only the front-end tool using the runtime libraries from the compute node (heterogeneous) builds.
runtimeBuild only the runtime libraries and collectors. Used for heterogeneous builds.
mpichBuild the mpich MPI implementation collectors
mptBuild the mpt MPI implementation collectors
mvapichBuild the mvapich MPI implementation collectors
mvapich2Build the mvapich2 MPI implementation collectors
openmpiBuild the openmpi MPI implementation collectors

MODULE FILE CREATION:

spack will create module files for each component, but they will be very basic, mainly setting the PATH variable to the install bin directory for the package being built. However, spack allow the authors of packages to add to the module file by supplying spack functions that will set environment variables and append to library paths or path variables. The OpenSpeedShop team has enhanced the cbtf-krell and openspeedshop module files to set the necessary environment variables for successful execution.

The module file directory can be found at this location: /share/spack/modules/. All the spack created modules will be located at this location. For O|SS, all one should have to do is load the openspeedshop- module file in order to run experiments and view performance databases.

EXTERNAL PACKAGING INSTRUCTIONS:

As mentioned above, spack provides the external packages feature that allows you to specify packages that you want spack to use but not build. The file that specifies the packages is named packages.yaml and it resides in $HOME/.spack/linux on most systems and in $HOME/.spack/cray on Cray platforms. Here is an example file from a generic Linux system:

packages:
   all:
      providers:
         pkgconfig: [pkg-config]
   mpich:
      buildable: False
      paths:
         mpich@3.2.0%gcc@7.3.1 arch=linux-fedora27-x86_64: /opt/mpich-3.2
   openmpi:
      buildable: False
      paths:
         openmpi@3.1.3%gcc@7.3.1 arch=linux-fedora27-x86_64: /opt/openmpi-3.1.3
         openmpi@2.0.3%gcc@7.3.1 arch=linux-fedora27-x86_64: /opt/openmpi-2.0.3
         openmpi@1.8.2%gcc@7.3.1 arch=linux-fedora27-x86_64: /opt/openmpi-1.8.2
   qt:
      paths:
         qt3@3.3.8b%gcc@7.3.1 arch=linux-fedora27-x86_64: /usr/lib64/qt-3.3
   cmake:
      paths:
         cmake@3.10.0%gcc@7.3.1 arch=linux-fedora27-x86_64: /usr

The “arch=” tells spack what architecture the entries apply to.  If you have multiple architectures that are serviced by this file, then you will have to specify entries for each architecture in order for spack to find the external packages. To obtain the install paths used in the packages.yaml file, one can do a module show show command:

$ module show openmpi-3.1.3

This will give you the path to the openmpi-3.1.3 installation that you can use in packages.yaml file.   In this case, the path from the “module show” command corresponds to the line:   openmpi@3.1.3%gcc@7.3.1 arch=linux-fedora27-x86_64: /opt/openmpi-3.1.3 in the packages.yaml example above.

RESTRICTED ACCESS / SPACK MIRROR:

spack has a feature to handle cases when your network prevents accessing source on the platform you are building on. Many laboratory systems have security measures in place that prevent the default workflow of spack from working. spack is unable to download packages from across the web due to the security checks. For this situation, spack provides a mirror capability, where a directory of the package source files can be created based on a spack mirror create command. For more spack documentation see this url:http://spack.readthedocs.io/en/latest/mirrors.html?highlight=mirror

See below for an example of an O|SS mirror creation on a platform that allows downloading package sources and activation on the platform that doesn’t.

Note that for the build on the restricted access platform can succeed, make sure that the restricted platform version of spack matches with the spack being used to create the mirror. If not, some package source versions might not be available in the mirror, causing the build to fail.

# Create mirror which contains all possible variants you might run
# Do this on a machine that has full access to the internet and the files needed
# Note: the gcc version information is an example only, your gcc version may be different or not required
$ spack mirror create -d /home//spack-mirror -D openspeedshop+openmpi+mvapich2+mpich%gcc@4.8

# Because the new O|SS gui is not built in the base openspeedshop package,
# a separate mirror command is necessary if building the Qt4/Qt5 GUI is desired
$ spack mirror create -d /home//spack-mirror –D cbtf-argonavis-gui

# Tar and gzip this directory for move to build platform
$ tar -cvf spack_mirror.tar /home//spack-mirror-2018-04-05
$ gzip spack_mirror-2018-04-05.tar

# Un-zip and tar on secure/build platform
$ tar -zxvf spack_mirror-2018-04-05.tar.gz

# Set up spack to use this mirror directory
# After executing this command, the spack build will read files from the mirror directory.
$ spack mirror add linux-spack-mirror /home/user/spack-mirror-2018-04-05

# Note: For heterogeneous builds the compute node build needs its own mirror and be sure to change the module file for the compute node build and mirror. The linux front-end mirror is not visible when building for the compute node. This is a spack-ism, not related to O|SS.
$ spack mirror add compute-spack-mirror /home/user/spack-mirror-2018-04-05

# To remove a spack mirror from being used to find the package file source use this command:
$ spack mirror remove linux-spack-mirror