Moruna#
Moruna is a fast, lightweight runtime that runs your Python functions over datasets larger than memory, inside a memory limit you set, and sizes the work itself so the job neither runs out of memory nor leaves the machine idle.
Use cases
Run a model over a large dataset
Apply a tokenizer, a scoring model or any Python function to every row of a Parquet dataset that does not fit in memory, and write the results as Parquet.
Stay inside a memory limit
Give a job a budget, or let Moruna read its container's limit. Moruna measures what your function uses and adjusts batch sizes and workers to stay under it.
Resume a job that was stopped
Restart a long job from its last checkpoint instead of from the beginning. The output is the same as a run that was never interrupted.
Run a job described in a file
Describe the input, functions, output and budget in a job document, then run it with the moruna command from a scheduler, a container or a virtual machine.