SNAP
SNAP (Sentinel Application Platform) is a remote sensing toolbox architecture developed by the European Space Agency. It includes tools for all common remote sensing satellites.
Available
SNAP is available in Puhti with following versions:
- 7.0.0 (with snappy 6.0 and python 2.7.5)
Installed plug-ins
- Sentinel toolboxes (1,2,3)
- All Idepix processors
- SMOS toolbox
- SNAPHU
- Radarsat toolbox
- PROBA-V toolbox
- Sen2Cor (external tool)
Usage
SNAP is included in the snap module and can be loaded with
module load snap
SNAP userdir configuration (Do this the first time!)
SNAP uses significant amount of storage space for cache and temporary files. These files easily fill your HOME directory so a script was written for configuring the snap user directories easily. You should always run this the first time you start using SNAP in Puhti or if you switch projects.
After loading the module run
source snap_add_userdir <YOUR-PROJECTS-SCRATCH-FOLDER>
If you run the command with another folder again, it overwrites the previous settings.
You could also request a fast nvme disk in a batch job and run the command first in the batch job so that all the temp/cache files are written to a fast disk rather than the scratch. It might provide speed improvement in demanding calculations.
Using SNAP with graphical user interface
If you have connected with a ssh connection that has X11 forwarding enabled, you can launch a graphical user interface of SNAP with
snap
For X11 forwarding to be enabled you need to install a suitable program for your own computer first (unless you are using Linux or Mac). You can read instructions how to do that here
Note
Do not run long CPU intensive jobs on the login nodes! This means you can't run computationally intensive analysis on graphical user interfaces on Puhti until interactive queue is available in Puhti. You need to use the batch job system on Puhti.
Using SNAP with Graph Processing Tool (gpt) command
The Graph Processing Tool gpt is a command line tool used for bulk processing. It can be run with
gpt <full_path_to_graph_xml_file> -Pfile=<inputfile> -t <outputfile>
Some relevant gpt options include
- -q Number of threads the gpt instance will use. Set it to the number of CPU cores requested or more
- -c Cache size in bytes. Change this if storage space becomes an issue
- -x Clear internal tile cache after writing a complete row of tiles to output file. Add this if memory becomes an issue
More information on the SNAP command line tutorial
There is a also a custom made gpt_array command that allows the usage of gpt with Puhti array jobs. It solves the problem of multiple jobs using the same cache folder. The command is otherwise the same as gpt but you include the cache-folder's path as first argument. In an array job you can define that cache folder dynamically with the iterating environment variable $SLURM_ARRAY_TASK_ID and make sure each job has an individual cache folder.
gpt_array /scratch/<project>/snap/tmp_snap_userdir_"$SLURM_ARRAY_TASK_ID" <normal gpt arguments>
Using SNAP with the Python library snappy
It is also possible to access SNAP functionalities from Python with the snappy Python library. When loading the snap module with module load snap, a conda environment is also loaded that has python 2.7 and snappy installed so you can just start python and import snappy.
This conda environment also includes: * pandas * numpy * geopandas * rasterio * rasterstats * sentinelsat * spyder
And many more, for retrieving the full list in Puhti use: list-packages If you need additional libraries, contact servicedesk@csc.fi.
License and citing
All SNAP software is published under the GPL-3 license and its sources are available on GitHub.
In your publications please acknowledge also oGIIR and CSC, for example “The authors wish to acknowledge for computational resources CSC – IT Center for Science, Finland (urn:nbn:fi:research-infras-2016072531) and the Open Geospatial Information Infrastructure for Research (oGIIR, urn:nbn:fi:research-infras-2016072513).”