Skip to content

Docs CSC now features an automatic Finnish translation. Click here for more information.

Warning!

Puhti and Mahti are being decommissioned in stages, and their storage areas will become fully unavailable from 15 October 2026. Clean up unnecessary files and move any data you need to keep by 31 August 2026. See the Roihu data migration guide for instructions on transferring your data to Roihu.

Puhti scratch is very full: keep only active data there and move or delete everything else. No new Puhti scratch quota will be granted.

trankit

trankit is a light-weight transformer-based Python toolkit for multilingual Natural Language Processing (NLP). It provides trainable pipelines for fundamental NLP tasks such as sentence segmentation, tokenization, part-of-speech tagging, morphological feature tagging, lemmatization and dependency parsing across many languages.

Available

  • Puhti: 1.1.0
  • Roihu: 1.1.0

License

trankit is open source, licensed under the Apache License 2.0.

Usage

Initialize trankit with:

module load trankit

On CSC systems trankit is run from the command line with python -m trankit. It parses plain text files (UTF-8) from a flat input directory and writes one JSON output file per input file into an empty output directory. Three input formats are supported: plaindoc (plain text), plainsen (one sentence per line) and pretok (pretokenized, one token per line, sentences separated by blank lines).

The shared language models are provided in the $LB_MODELS directory. For example, to parse Finnish text:

python -m trankit --input /directory/with/input/files \
  --output_dir /directory/with/output/files \
  --input_format plaindoc \
  --cache_dir $LB_MODELS \
  --lang finnish

trankit produces JSON output. The companion script trankit2conllu converts it to CoNLL-U format:

trankit2conllu --input /path/to/trankit/output \
  --output_dir /path/to/conllu-output

For more help once the module is loaded, run module help trankit. trankit can also be used as a Python library; see the upstream documentation.

More information