Selecting the Software Environment
GSWHC-B Getting Started with HPC Clusters \(\rightarrow\) USE1-B Use of the Cluster Operating System \(\rightarrow\) USE1.3-B Selecting the Software Environment
Relevant for: Tester, Builder, and Developer
Short Background:
In general, multiple versions of software tools and software environments are installed on an HPC system.
A widely used system for handling different software environments are “Environment Modules”. (basic level)
Description:
- You will learn to how to use Environment Modules (basic level)
This skill requires no sub-skills
Level: basic
Environment Modules
Introduction
This text gives a short overview of Environment Modules. More information can be found on the homepage of the project.
Environment Modules are a tool for managing environment variables of the shell. The module load command extends variables containing search paths (e.g. PATH or MANPATH). The module unload command is the corresponding inverse operation, it removes entries from search paths. By extending search paths software is made callable. Effectively software can be provided through Modules. An advantage over defining environment variables directly in the shell is that Modules allow to undo changes of environment variables. The idea of introducing Modules is to be able to define software environments in a modular way. In the context of HPC, Modules make it easy to switch compilers or libraries, or to choose between different versions of an application software package.
Initialization
The module command is a shell function. As a consequence this function needs to be defined in every instance of the shell including batch job scripts. Typically, this happens by sourcing an initialization script. In interactive environments the module function is probably available without further initialization. But in batch jobs it might be necessary to source an initialization script. In that latter case the documentation of your computing center will provide information on how this is performed.
Naming
Names of Modules have the format program/version or just program. Modules can be loaded (and always be unloaded) without specifying a version. If the version is not specified the default version will be loaded. The default version is either explicitly defined (and will be marked in the output of module avail) or module will load the version that appears to be the latest one. Because defaults can change versions should always be given if reproducibility is required.
Dependences and conflicts
Modules can have dependences, i.e. a Module can enforce that other Modules that it depends on must be loaded before the Module itself can be loaded.
Module can be conflicting, i.e. these modules must not be loaded at the same time (e.g. two version of a compiler). A conflicting Module must be unloaded before the Module it conflicts with can be loaded.
Caveats
The name Modules suggest that Modules can be picked and combined in a modular fashion. For Modules providing application packages this is true (up to possible dependences and conflicts described above), i.e. it is possible to chose any combination of application software.
However, today, environments for building software are not modular anymore. In particular, it is no longer guaranteed that a library that was built with one compiler can be used with code generated by a different compiler. Hence, the corresponding Modules cannot be modular either. A popular way to handle this situation is to append compiler information to the version information of library Modules. Firstly, this leads to long names and secondly, to very many Modules that are hard to overlook. A more modern way is to build up toolchains with Modules. For example, in such a toolchain only compiler Modules are available at the beginning. Once a compiler Module is loaded, MPI libraries (the next level of tools) become available and after that all other Modules (that were built with that chain).
Important commands
Important Module commands are:
| list Modules currently loaded | module list |
| list available Modules | module avail |
| load a Module | module load program[/version] |
| unload a Module | module unload program |
| switch a Module (e.g. compiler version) | module switch program program/version |
| add or remove a directory/path to the Module search path (e.g. by an own Module directory) | module [un]use [--append] path |
Self-documentation
Modules are self-documented:
| show the actions of a Module | module display program/version |
| short description of [one or] all Modules | module whatis [program/version] |
| longer help text on a Module | module help program/version |
help on module itself |
module help |