Geoconda

Geoconda is a collection of python packages that facilitate the development of python scripts for geoinformatics applications. It includes following python packages:

  • ArcGIS Python API - provides simple and efficient tools for sophisticated vector and raster analysis, geocoding, map making, routing and directions.
  • boto3 - for working files in S3 storage, for example Allas. Example.
  • cartopy - for map plotting.
  • descartes - use Shapely or GeoJSON-like geometric objects as matplotlib paths and patches.
  • fiona - reads and writes spatial data files.
  • gdal - reads and writes spatial data files, and GDAL/OGR data manipulation tools.
  • geoalchemy2 - provides extensions to SQLAlchemy for working with spatial databases, primarily PostGIS.
  • igraph - for fast routing.
  • geopandas - GeoPandas extends the datatypes used by pandas.
  • laspy
  • laxpy
  • networkx - for the creation, manipulation, and study of the structure, dynamics, and functions of complex networks.
  • pyproj - performs cartographic transformations and geodetic computations.
  • osmnx - download spatial geometries and construct, project, visualize, and analyze street networks from OpenStreetMap's APIs.
  • pysal - spatial analysis functions.
  • pdal - for lidar data
  • rasterio - access to geospatial raster data.
  • rasterstats - for summarizing geospatial raster datasets based on vector geometries. It includes functions for zonal statistics and interpolated point queries.
  • rtree - spatial indexing and search.
  • sentinelsat - downloading Sentinel images
  • shapely - manipulation and analysis of geometric objects in the Cartesian plane.
  • scikit-learn - machine learning for Python.
  • skimage - algorithms for image processing.
  • swiftclient, keystoneclient - for working with SWIFT storage, for example Allas.
  • xarray - for multidimensional raster data.
  • And many more, for retrieving the full list in Puhti use: list-packages

Additionally geoconda includes:

  • spyder - Scientific Python Development Environment with graphical interface (similar to RStudio for R).
  • GDAL/OGR commandline tools 3.0.2
  • PDAL 2.0.1
  • QGIS 3.10.0
  • LasTools 20171231
  • ncview for visualizing netcdf files
  • proj4, geos and many more, see /appl/soft/geo/geoconda/miniconda3/envs/geoconda-3.7/binfor full list.

Note

If you want to use Spyder, QGIS, ncview or other tools with graphical user interfaces, you should connect to Puhti using -X connection.

Python has also packages for parallel computing, for example multiprocessing. In our examples there is two cases using Python multiprocessing: zonal statistics and focal mean.

(If you think that some important GIS package for Python is missing from here, you can ask for installation from servicedesk@csc.fi.)

Available

The geoconda module is available in Puhti:

  • 3.7 (version number is the same as Python version)

Usage

Using geoconda

For using Python packages and other tools listed above, you can initialize them with:

module load geoconda

By default the latest geoconda module is loaded. If you want a specific version you can specify the version number of geoconda:

module load geoconda/[VERSION]

For using the Spyder IDE give:

spyder

To check the exact packages and versions included in the loaded module:

list-packages

Adding more Python packages to GeoConda

You can add more Python packages to Geoconda for your own use with pip, for example:

pip install [newPythonPackageName] --user

The packages are by default installed to your home directory under .local/lib/python3.7/site-packages. If you would like to change the installation folder define PYTHONUSERBASE environmentvariable with new installation location first:

export PYTHONUSERBASE=/projappl/<your_project>/python3.7_pip

You should use the same export command then always also before using, inc in the batch job files.

If you would like to make a own conda environment, it is recommended to make also own Miniconda installation. Or then you can use bioconda.

Using Allas from Python

There are two Python libraries installed in Geoconda that can interact with Allas. Swiftclient uses the swift protocol and boto3 uses S3 protocol. You can find CSC examples how to use both here.

License and citing

All packages are licensed under various free and open source licenses (FOSS), see the linked pages above for exact details. In your publications please acknowledge also oGIIR and CSC, for example “The authors wish to acknowledge for computational resources CSC – IT Center for Science, Finland (urn:nbn:fi:research-infras-2016072531) and the Open Geospatial Information Infrastructure for Research (oGIIR, urn:nbn:fi:research-infras-2016072513).”

References