Conda

NOTE:

The source code for the examples below can be found on GitHub.

Conda packages

You can use conda within workflows install Conda packages (under the hood, it uses Miniforge).

Create the following Python script:

import pandas as pd

if __name__ == '__main__':
    df = pd.DataFrame(
        {
            "Name": [
                "Braund, Mr. Owen Harris",
                "Allen, Mr. William Henry",
                "Bonnell, Miss. Elizabeth",
            ],
            "Age": [22, 35, 58],
            "Sex": ["male", "male", "female"],
        }
    )

    print(df)

Now, define a workflow YAML file:

workflows:
  - name: hello-conda
    provider: bash
    commands:
      - conda install pandas
      - python python/hello_pandas.py

Run it locally using the dstack run command:

$ dstack run hello-conda

Conda environments

You can create your custom Conda environments using conda env create, save them as artifact, and reuse from other workflows via deps and conda activate.

Say you have the following Conda environment YAML file:

name: myenv

dependencies:
  - python=3.10
  - pandas

Now, create the following workflow YAML file:

workflows:
  - name: setup-conda
    provider: bash
    commands:
      - conda env create --file conda/environment.yaml
    artifacts:
      - path: /opt/conda/envs/myenv

  - name: use-conda
    provider: bash
    deps:
      - workflow: setup-conda
    commands:
      - conda activate myenv
      - conda env list

Now, run the setup-conda workflow:

$ dstack run setup-conda

And then, run the use-conda workflow:

$ dstack run use-conda

conda environments:

base                     /opt/anaconda3
workflow                 /opt/conda/envs/workflow
myenv                *   /opt/conda/envs/myenv

The use-conda workflow reuses the myenv environment from the setup-conda workflow.

NOTE:

Conda environments are always bound to a specific architecture and cannot be reused across machines with different architectures (e.g. AMD64 vs ARM64).