Managing code locations in Dagster+#

A code location specifies a single Python package or file that defines your Dagster code. When you add a code location in Dagster+, you're instructing your deployment's agent where to find your code. This applies to both Hybrid and Serverless deployments.

Dagster+ agents use this location configuration to load your code and upload metadata about your jobs. Each full deployment - for example, prod - can include code from one or more code locations.

By the end of this guide, you'll understand:

• The requirements for Dagster code to work with Dagster+
• How to interact with code locations using the Dagster UI
• How to interact with code locations using the dagster-cloud CLI

Dagster+ code requirements#

To work with Dagster+, your Dagster code:

• Must be loaded from a single entry point, either a Python file or package. This entry point can load repositories from other files or packages.

• Must run in an environment where the dagster and dagster-cloud 0.13.2 or later Python packages are installed.

• If using Hybrid Deployment:

  • And you're using an Amazon Elastic Container Service (ECS), Kubernetes, or Docker agent, your code must be packaged into a Docker image and pushed to a registry your agent can access. Dagster+ doesn't need access to your image - your agent only needs to be able to pull it.

    Additionally, the Dockerfile for your image doesn't need to specify an entry point or command. These will be supplied by the agent when it runs your code using your supplied image.

  • And you're using a local agent, your code must be in a Python environment that can be accessed on the same machine as your agent.

Additionally, note that:

• Your code doesn't need to use the same version of Dagster as your agent
• Different code locations can use different versions of Dagster
• Dagster+ doesn't require a workspace.yaml file. You can still create a workspace.yaml file to load your code in an open source Dagster webserver instance, but doing so won't affect how your code is loaded in Dagster+.

Interacting with code locations#

Interacting with code locations can be done using the Dagster+ UI or the dagster-cloud CLI. The CLI commands perform the same underlying operations as those in the Deployment tab of the UI.

Adding code locations#

# Set up YAML file for example location
cat > example_location.yaml <<EOL
location_name: cloud-examples
image: dagster/dagster-cloud-examples:latest
code_source:
  package_name: dagster_cloud_examples
EOL

dagster-cloud deployment add-location --from example_location.yaml

dagster-cloud deployment add-location test_location \
    --image dagster/dagster-cloud-examples:latest \
    --package-name dagster_cloud_examples

dagster-cloud deployment add-location --help

Modifying code locations#

Dagster+ UIdagster-cloud CLI

To modify a code location, click the dropdown menu to the right of the location. In the menu, click Modify:

After a code location is updated, the agent will perform a rolling update of your code and jobs will update in Dagster+. Note: Updating code won't interrupt any currently launched runs.

Redeploying code locations#

To reload your code and upload job metadata to Dagster+ without modifying the code location, click the Redeploy button:

For example, if the agent was unable to pull your image due to a permissions issue that's since been addressed, clicking Redeploy will tell the agent to try again.

Deleting code locations#

Dagster+ UIdagster-cloud CLI

To delete a code location, click the dropdown menu to the right of the location. In the menu, click Remove:

When prompted, confirm the deletion.

Syncing the workspace with the CLI#

The YAML configuration for all code locations in a workspace can be kept in a dagster_cloud.yaml file and reconciled with the workspace config in Dagster+ using the deployment sync-locations command.

For example, if you have the following dagster_cloud.yaml file:

locations:
  - location_name: machine-learning
    image: myregistry/dagster-machine-learning:mytag
    code_source:
      package_name: dagster_cloud_machine_learning
    executable_path: /my/folder/python_executable
    attribute: my_repo
  - location_name: data-eng
    image: myregistry/dagster-data-eng:myothertag
    code_source:
      python_file: repo.py
    working_directory: /my/folder/working_dir/

Run the following to reconcile the file with Dagster+'s remote workspace:

dagster-cloud deployment sync-locations -w dagster_cloud.yaml

To see all available options, run:

dagster-cloud deployment sync-locations -h

Related#