Quickstart
Step 1: Set up runners
Runners are machines that run submitted workflows. dstack supports two types of runners: on-demand runners and self-hosted runners.
The on-demand runners are created automatically by dstack (in the computing vendor, configured by the user, e.g. AWS) for the time of running workflows. The self-hosted runners can be set up manually to run workflows using the user's own hardware.
Option 1: Set up on-demand runners
To use on-demand runners, go to the Settings
, then AWS
.
Here, you have to provide AWS Access Key ID
and AWS Secret Access Key
that have the
corresponding permissions to create EC2 instances in your AWS account.
Once you've provided credentials, use the Add limit
button to configure limits:
The configured Limits
represent the maximum number of EC2 instances of the specific Instance Type
and in the specific Region
, that
dstack can create at one time to run workflows.
Option 2: Set up self-hosted runners
As an alternative to on-demand runners, you can run workflows on your own hardware.
To do that, you have to run the following command on your server:
curl -fsSL https://get.dstack.ai/runner -o get-dstack-runner.sh
sudo sh get-dstack-runner.sh
dstack-runner config --token <token>
dstack-runner start
Your token
value can be found in Settings
:
If you've done this step properly, you'll see your server on the Runners
page:
Step 2: Install the CLI
Now, to be able to run workflows, install and configure the dstack CLI:
pip install dstack -U
dstack config --token <token>
Step 3: Clone the repo
In this tutorial, we'll run workflows defined in
github.com/dstackai/dstack-examples
.
git clone https://github.com/dstackai/dstack-examples.git
cd dstack-examples
This project includes two workflows: download-mnist
and train-mnist
. The frst workflow downloads the MNIST dataset,
whilst the second workflow trains a model using the output of the first workflow as an input:
workflows:
- name: download-mnist
provider: python
requirements: requirements.txt
script: download.py
artifacts:
- data
- name: train-mnist
provider: python
requirements: requirements.txt
script: train.py
artifacts:
- model
depends-on:
- download-mnist
resources:
gpu: ${{ gpu }}
variables:
train-mnist:
gpu: 1
batch-size: 64
test-batch-size: 1000
epochs: 1
lr: 1.0
gamma: 0.7
seed: 1
log-interval: 10
Step 4: Run workflows
Go ahead, and run the train-mnist
workflow using the following command:
dstack run train-mnist
If you want to change any of the variables, you can do that in .dstack/variables.yaml
, or from the CLI:
dstack run train-mnist --gpu 2 --epoch 100 --seed 2
When you run train-mnist
, because train-mnist
depends on download-mnist
, dstack will create a run with two jobs:
one for train-mnist
and one for download-mnist
:
Step 5: Tag runs
When the run is finished, you can assign a tag to it, e.g. latest
:
dstack tag cowardly-goose-1 latest
Now, you can refer to this tagged workflow from .dstack/workflows.yaml
:
workflows:
- name: download-mnist
provider: python
requirements: requirements.txt
script: download.py
artifacts:
- data
- name: train-mnist
provider: python
requirements: requirements.txt
script: train.py
artifacts:
- model
depends-on:
- download-mnist:latest
resources:
gpu: 1
Now, if you run the train-mnist
workflow, dstack won't create a job for the download-mnist
workflow.
Instead, it will reuse the artifacts of the tagged workflow.
Keep in mind that you can tag as many runs as you want. When you refer to a workflow via a tag, dstack will use the job that has the corresponding tag, workflow name, and variables.
It's now time to give a try to dstack, and run your first workflow.