It is possible to install own software on CSC supercomputers if you cannot find a software
suitable for your needs from the list of pre-installed applications
module spider. The installation procedure will vary based on your specific
application. There are, however, some general rules you should be mindful of:
- You cannot use
sudo, i.e. typical
sudo yumcommands you might find online will not work on CSC supercomputers.
- You cannot install into "standard" system locations, e.g.
/usr/libetc. Instead, the best location for your own installations is the
/projappldirectory of your project.
- Use the fast local disk
$TMPDIRwhen compiling to avoid stressing the parallel file system. Compiling applications typically cause quite a bit of I/O load.
- Many software might require some dependencies, e.g. HPC libraries such as FFTW or ScaLAPACK. Note that many of these are available as pre-installed modules, so you may not need to install everything from scratch.
- New software are not automatically added to your
$PATH. To access the software, either provide the full path or add with
Help is available!
Don't hesitate to contact CSC Service Desk if you encounter issues with installing your own software.
Native installations refer to applications that are installed directly to the system.
Typically, one downloads the source code of the software, compiles the code, and installs
to a location where the user has write-access, e.g. the project's
Native installation from source code might sometimes be the only way to install an
application, and is recommended especially for software with few or no dependencies.
HPC software written using programming languages such as C, C++ or Fortran need to be compiled before installing. Guidelines on compiling software on CSC supercomputers can be found from the links below. A list of available HPC libraries that may need to be linked upon compiling is also provided.
Spack is a flexible package manager that can be used to install software on supercomputers and Linux and macOS systems. The basic module tree including compilers, MPI libraries and many of the available software on CSC supercomputers have been installed using Spack. Spack is similar to the EasyBuild package manager extensively used on LUMI.
CSC provides user Spack modules on Puhti and Mahti that can be used to build software on top of the available compilers and libraries. It is also possible to install different customized versions of packages available in the module tree for special use cases. See here for a short tutorial on how to install software on CSC supercomputers using Spack. Spack is also available on LUMI.
Ready-made binaries typically exhibit optimal performance only on the system on which they have been compiled on. This applies especially for MPI codes, which should always be re-compiled for best performance. However, if the binary you want to use is a simple serial or threaded application, you can try running it directly.
Containerizing applications can be a very efficient way to install software and libraries, especially if the application has complex dependencies such as most Python environments (see below). CSC supercomputers support Apptainer/Singularity containers, which are similar to Docker, but better suited for multi-user HPC systems. In most cases, ready-made Docker containers can be easily converted into an Apptainer image. Another option is to build your own container from scratch. More details on working with containers in CSC's computing environment can be found from the links below:
Best practice guidelines on installing own Python and R packages can be found in the Python, R and Tykky container wrapper pages below.
- Installing Python packages and environments
- Containerizing Conda and pip environments with Tykky
- R package installations
Briefly, individual Python packages with no/few dependencies can be installed
alongside CSC's pre-installed Python modules with
pip install --user <package>.
More complicated environments should always be containerized. This is accomplished
easily with Tykky.
Similarly, the pre-installed R module provided by CSC is a containerized environment
containing more than 1300 packages. If these do not suit your needs, you can install your
own packages into a project-specific location under
/projappl and add this to your library
trees in R. See here for specific details.
- Installing your own software (CSC Computing Environment slides and tutorials)