Overview¶

Your Vulcan project needs a configuration file. It tells Vulcan how to connect to your data warehouse, where to store state, and what defaults to use for your models. Without it, Vulcan doesn't know where your data lives or how to run your transformations.

Configuration File¶

Create a configuration file in your project root. Choose one:

• config.yaml: YAML format. Use this for most projects. Simple and readable.

• config.py: Python format. Use this if you need dynamic configuration or want to generate settings programmatically.

Example Configuration¶

Here's what a typical configuration file looks like:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70

# Project identity
name: orders-analytics
display_name: Orders Analytics Platform
description: Orders Analytics is a centralized data product delivering clean, trusted insights across the full order lifecycle.

# Catalog metadata
discoverable: true
version: 0.1.2
alignment: consumer_aligned

# Environment behaviour
vde: false   # set to true to enable Virtual Data Environments; not supported on spark/trino

# Classification
tags:
  - e-commerce
  - retail
  - sales_analytics
  - customer_analytics
  - postgres

terms:
  - glossary.data_product
  - glossary.analytics_platform
  - glossary.sales_operations

# Metadata
metadata:
  domain: sales_operations
  use_cases:
    - Daily and weekly sales reporting
    - Customer segmentation and RFM analysis
    - Sales funnel conversion tracking
    - Product performance analytics
  limitations:
    - Demo dataset with synthetic data (100 customers, 1000 orders)
    - Historical data from November 2025 onwards

# Gateway Connection
gateways:
  default:
    connection:
      type: postgres
      host: warehouse
      port: 5432
      database: warehouse
      user: vulcan
      password: "{{ env_var('DB_PASSWORD') }}"
    state_connection:
      type: postgres
      host: statestore
      port: 5432
      database: statestore
      user: vulcan
      password: "{{ env_var('STATE_DB_PASSWORD') }}"

default_gateway: default

# Model Defaults (required)
model_defaults:
  dialect: postgres
  start: 2024-01-01
  cron: '@daily'

# Linting Rules
linter:
  enabled: true
  rules:
    - ambiguousorinvalidcolumn
    - invalidselectstarexpansion

Configuration Structure¶

graph TB
    Config[config.yaml]
    Config --> Project[Project Settings]
    Config --> Metadata[Metadata]
    Config --> Gateways[Gateways]
    Config --> ModelDefaults[Model Defaults]
    Config --> Options[Optional Features]
    Project --> Name[name, display_name]
    Project --> Desc[description]
    Project --> Tags[tags, terms]
    Metadata --> Domain[domain]
    Metadata --> UseCases[use_cases]
    Metadata --> Limitations[limitations]
    Gateways --> Connection[connection]
    Gateways --> StateConn[state_connection]
    Gateways --> TestConn[test_connection]
    Gateways --> Scheduler[scheduler]
    Options --> Linter[linter]
    Options --> Notifications[notifications]
    Options --> Variables[variables]
    Options --> ExecHooks[execution_hooks]

Configuration Sections¶

Project Settings¶

Metadata fields that identify your project. They don't affect how Vulcan runs, but catalog tools rely on them for organization and discovery.

# Project identity
name: orders-analytics
display_name: Orders Analytics Platform
description: Orders Analytics delivers insights across the full order lifecycle.

# Catalog metadata
discoverable: true
version: 0.1.2
alignment: consumer_aligned

# Classification
tags:
  - e-commerce
  - retail
  - sales_analytics

terms:
  - glossary.data_product
  - glossary.analytics_platform
  - glossary.sales_operations

Metadata¶

Metadata fields provide additional context about your project's purpose and scope. Use these to document what your project does, where it applies, and any known constraints.

# Metadata
metadata:
  domain: sales_operations
  use_cases:
    - Daily and weekly sales reporting
    - Customer segmentation and RFM analysis
    - Sales funnel conversion tracking
  limitations:
    - Demo dataset with synthetic data (100 customers, 1000 orders)
    - Historical data from November 2025 onwards

Gateways¶

Gateways define how Vulcan connects to your data warehouse and state backend. Define multiple gateways for different environments: dev, staging, prod. Each gateway has its own connection settings.

# Gateway Connection
gateways:
  default:
    connection:
      type: postgres
      host: warehouse
      port: 5432
      database: warehouse
      user: vulcan
      password: "{{ env_var('DB_PASSWORD') }}"
    state_connection:
      type: postgres
      host: statestore
      port: 5432
      database: statestore
      user: vulcan
      password: "{{ env_var('STATE_DB_PASSWORD') }}"
    test_connection:
      type: duckdb
    scheduler:
      type: builtin
    state_schema: my_project

default_gateway: default

Model Defaults¶

The model_defaults section is required. At minimum, specify dialect to tell Vulcan what SQL dialect your models use. Other defaults are optional but apply to all models automatically, so you don't repeat the same settings in every model file.

model_defaults:
  dialect: postgres     # Required
  owner: data-team
  start: 2024-01-01
  cron: '@daily'

See Model Defaults for all available options.

Variables¶

Store sensitive information like passwords and API keys without hardcoding them. Use environment variables, .env files, or configuration overrides. Variables also let you override configuration values dynamically.

variables:
  warehouse_schema: analytics
  refresh_window_days: 7

gateways:
  default:
    variables:
      warehouse_schema: analytics_dev  # override per gateway

See Variables for details.

Execution Hooks¶

Run SQL statements automatically at the start and end of vulcan plan and vulcan run commands. Use before_all for setup tasks like creating temporary tables or granting permissions. Use after_all for cleanup or post-processing.

before_all:
  - GRANT SELECT ON ALL TABLES IN SCHEMA analytics TO reporting_role

after_all:
  - ANALYZE analytics.daily_sales

See Execution Hooks for detailed examples and use cases.

Linter¶

Automatic code quality checks that run when you create a plan or run the lint command. Catches common mistakes and enforces coding standards. Use built-in rules or create custom ones.

See Linter for rules and custom linter configuration.

Notifications¶

Set up alerts via Slack or email. Get notified when plans start or finish, when runs complete, or when audits fail.

notification_targets:
  - type: slack
    url: "{{ env_var('SLACK_WEBHOOK_URL') }}"
    notify_on:
      - run_end
      - audit_failure
  - type: console
    notify_on:
      - plan_change

See Notifications for Slack webhooks, API, and email setup.

Supported Engines¶

Vulcan works with these data warehouses and compute engines:

Complete Configuration Reference¶

This table lists all available configuration keys in config.yaml. Click the links for detailed documentation.

Project Identity & Metadata¶

Gateway & Connection Configuration¶

* At least one gateway with a connection is required.
** With root-level state connection, defaults to {name} (normalized).

Model Configuration¶

* The model_defaults.dialect field is required.

Variables & Environment¶

Execution Hooks¶

Code Quality & Linting¶

Notifications & Users¶

Environment & Schema Management¶

environment_catalog_mapping:
  dev: dev_catalog
  staging: staging_catalog
  prod: prod_catalog

Project Management¶

Command Configuration¶

Integrations & External Services¶

Minimal Valid Configuration¶

The non-skippable parts of config.yaml are: a non-empty name, a non-empty description, at least one working gateways.<name>.connection, and model_defaults.dialect. The runtime also needs DATAOS_TENANT_ID in the environment.

name: my-project
description: My project description

gateways:
  default:
    connection:
      type: postgres
      host: localhost
      port: 5432
      database: mydb
      user: myuser
      password: mypass

model_defaults:
  dialect: postgres

# Required at runtime, not in YAML
export DATAOS_TENANT_ID=my-tenant

Everything else has a default and you can omit it.

Validation Rules¶

Some fields become required only when another field is enabled:

• name must be non-empty (or supplied via DATAOS_RESOURCE_NAME).
• description must be non-empty.
• metadata.domain is required when the metadata: block is present.
• hera.url and hera.token are required when hera.enabled: true.
• heimdall.base_url is required when heimdall.enabled: true.
• analytics.api_key is required when analytics.enabled: true.
• vde: true is rejected for spark and trino gateway types.
• version must be valid SemVer 2.0 (e.g. 0.1.2, 1.0.0-rc.1).

Environment Variables¶

A few values come from the shell or .env, not from YAML:

Migration from the Legacy Schema¶

If you have an older config.yaml, these keys have moved or been replaced:

Quick migration checklist:

1. Replace virtual_environment_mode: full with vde: true.
2. Remove virtual_environment_mode: dev_only (or set vde: false explicitly).
3. Add discoverable, version, alignment near the top of the file if you want non-default values.
4. Make sure version is valid SemVer (0.1.2, not 0.1 or v0.1.2).
5. Remove any deprecated keys listed above.
6. Set DATAOS_TENANT_ID in your shell or .env.

Best Practices¶

Use environment variables for sensitive data like passwords and API keys. Keeps secrets out of your config files and makes it easier to manage different environments.

Set meaningful defaults in model_defaults to reduce boilerplate. If most of your models use the same dialect, start date, or cron schedule, set it once here instead of repeating it everywhere.

Enable linting to catch common errors early in development. Fix issues before they make it to production.

Separate state connection from your data warehouse for better isolation. Prevents state operations from interfering with your data processing.

Use multiple gateways for different environments: dev, staging, prod. Test changes safely before deploying to production. Use different database configurations for each environment.