Easy-to-run ML workflows on any cloud¶
Welcome to dstack
's documentation! Here you can learn what it is, how it works, and how to get started.
What is dstack?¶
dstack
is an open-source tool that enables defining ML workflows as code, running them easily on any cloud while saving
artifacts for reuse. It offers freedom to use any ML frameworks, cloud vendors, or third-party tools without requiring
code changes.
How does it work?¶
Define workflows¶
Define ML workflows, their output artifacts, hardware requirements, and dependencies via YAML.
workflows:
- name: mnist-data
provider: bash
commands:
- pip install torchvision
- python tutorials/mnist/mnist_data.py
artifacts:
- path: ./data
- name: train-mnist
provider: bash
deps:
- workflow: mnist-data
commands:
- pip install torchvision pytorch-lightning tensorboard
- python tutorials/mnist/train_mnist.py
artifacts:
- path: ./lightning_logs
YAML eliminates the need to modify code in your scripts, giving you the freedom to choose frameworks, experiment trackers, and cloud providers.
Run workflows¶
Once a workflow is defined, you can use the dstack run
command to run it either locally or remotely.
Run locally¶
By default, workflows run locally on your machine:
$ dstack run mnist-data
RUN WORKFLOW SUBMITTED STATUS TAG BACKENDS
penguin-1 mnist-data now Submitted local
Provisioning... It may take up to a minute. ✓
To interrupt, press Ctrl+C.
Downloading http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz
---> 100%
$
The artifacts from local workflows are also stored and can be reused in other local workflows.
Run remotely¶
To run a workflow remotely (e.g. in a configured cloud account), add the --remote
flag to the dstack run
command:
$ dstack run mnist-data --remote
RUN WORKFLOW SUBMITTED STATUS TAG BACKENDS
mangust-1 mnist-data now Submitted aws
Provisioning... It may take up to a minute. ✓
To interrupt, press Ctrl+C.
Downloading http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz
---> 100%
$
The output artifacts from remote workflows are also stored remotely and can be reused by other remote workflows.
The necessary hardware resources can be configured either via YAML or through arguments in the dstack run
command, such
as --gpu
and --gpu-name
.
$ dstack run train-mnist --remote --gpu 1
RUN WORKFLOW SUBMITTED STATUS TAG BACKENDS
turtle-1 train-mnist now Submitted aws
Provisioning... It may take up to a minute. ✓
To interrupt, press Ctrl+C.
GPU available: True, used: True
Epoch 1: [00:03<00:00, 280.17it/s, loss=1.35, v_num=0]
---> 100%
$
Upon running a workflow remotely, dstack
automatically creates resources in the configured cloud account and destroys them
once the workflow is complete.
NOTE:
For questions or feedback, reach us through our community Slack channel or GitHub.