# DAG Desk

> Paste one Apache Airflow DAG file and run one of five jobs over it: audit it, migrate it to
> Airflow 3, annotate its data-asset lineage, generate its pytest suite, or root-cause a failing
> task run from its log.

https://dag-desk.skillsafe.ai/

## What it is for

A data engineer holding one DAG file, in one sitting. The five lanes are the things you actually do
with a DAG you have just been handed or have just written: work out whether it is fit to merge, find
out what breaks when the deployment moves to Airflow 3, discover what data it really reads and
writes, write the tests that stop it regressing, and find out why last night's run went red.

It is a review desk, not a runtime. DAG Desk never executes your DAG, never runs pytest, never
connects to an Airflow deployment, a warehouse or object storage, and never reaches your repository.
Everything it says is about the text you pasted.

## The five lanes

Every lane takes the same primary input - one DAG file - and returns the same outer envelope, so the
export, history and reconciliation paths are identical across them. Only the `body` differs.

| `task` | What it returns |
| --- | --- |
| `audit` | A twelve-check status table, an inventory of what the file declares, findings with fixes, and quick wins. No file is emitted. |
| `migrate` | A table of Airflow 2 to 3 breakages with the exact replacement for each, an ordered migration plan, and the DAG rewritten in full for Airflow 3. |
| `lineage` | The data assets the DAG consumes and produces with a confidence and the evidence for each, a suggested asset schedule, and the DAG annotated with `inlets` and `outlets`. |
| `tests` | Test cases split into integrity, unit and behaviour layers, the fixtures they need, the gaps deliberately left, and a complete pytest file implementing every case. |
| `diagnose` | A failure classification, a root cause with the verbatim log lines that prove it, a timeline, ranked hypotheses each with the command that kills it, the blast radius, and a patch. |

The `diagnose` lane takes a second input, `log` - the Airflow task log of the failing run.

## Input fields

```json
{
  "task": "audit | migrate | lineage | tests | diagnose",
  "dag": "the full text of the DAG .py file",
  "log": "the Airflow task log (diagnose lane only)",
  "airflow_source": "2.x | 3.x | unknown",
  "airflow_target": "3.x",
  "prescan": {
    "resources": [{"id": "...", "label": "..."}],
    "flags": [{"id": "DS-SCHEDULE-INTERVAL", "label": "...", "severity": "critical", "line": 35}]
  },
  "clip_note": "present only when the input was too long and was clipped"
}
```

## Output contract

One JSON object. The outer envelope is identical in all five lanes:

```json
{
  "lane": "audit",
  "lane_inferred": false,
  "dag_id": "orders_etl",
  "title": "short human title",
  "posture": "ready | hardening-recommended | blocked",
  "verdict": "one sentence naming the thing that decides the posture",
  "airflow_source": "2.x",
  "airflow_target": "3.x",
  "summary": "a paragraph a reviewer could paste into a pull request",
  "assumptions": [],
  "open_questions": [],
  "findings": [
    {"id": "DD-001", "title": "", "severity": "critical | high | medium | low",
     "area": "parsing | scheduling | dependencies | data | reliability | performance | security | testing | style",
     "task_id": "", "line": 0, "evidence": "", "why": "", "fix": "", "fix_code": ""}
  ],
  "coverage_check": [{"flag_id": "", "status": "confirmed | set-aside | superseded", "finding_id": "", "note": ""}],
  "artifact": {"kind": "none | python | markdown", "filename": "", "content": ""},
  "next_lane": {"lane": "", "reason": ""},
  "body": {}
}
```

`severity` means impact, not tidiness: `critical` is "the DAG will not parse, will not run, or will
corrupt data"; `high` is "it will fail or misbehave under normal operation"; `medium` is "it works
but it will hurt"; `low` is style.

## The free prescan

Before any model run, and without an account, the page runs a deterministic scanner over the pasted
DAG entirely in the browser. It is a real parser, not a keyword search: a character-state pass builds
a masked view in which every comment and every string body is blanked out, so `schedule_interval`
appearing in a docstring or inside an SQL literal is not reported, while the same token in a call is.
It also tracks block depth, which is what lets it tell a `Variable.get()` at module scope - executed
on every scheduler parse, roughly every thirty seconds forever - apart from the same call inside a
task body, which runs only when the task runs.

The prescan extracts the DAG objects and their arguments, the task list and operators, the dependency
edges (`>>`, `<<`, `chain()`, `set_upstream`/`set_downstream`), the asset declarations, the tables
named in SQL string literals and the storage URIs, then builds the dependency graph, computes each
task's depth, detects cycles and orphans, and draws the graph as inline SVG.

It carries 34 rules with stable ids across two families. The Airflow 2 to 3 breaking set covers
`schedule_interval`, the core operator module moves, `DummyOperator`, `SubDagOperator`,
`execution_date`, `Dataset` to `Asset`, `days_ago`, `airflow.contrib`, `sla=`, `provide_context`,
the `airflow.sdk` authoring move, direct metadata-database access, and XCom pickling. The authoring
set covers parse-time cost, moving and naive start dates, missing schedule, catchup and
`max_active_runs`, missing retries, timeouts and alerting, missing tags and docs, cycles, duplicate
task ids, unwired and orphaned tasks, shell interpolation into `bash_command`, hardcoded credentials,
`time.sleep`, oversized XCom payloads, non-idempotent `INSERT`, `print` instead of the logger,
module-scope DAG generation loops and shared mutable `default_args`.

Every flag raised is sent with the run, and the model must return exactly one `coverage_check` entry
per flag. Anything it fails to account for is surfaced in the interface as unreconciled rather than
quietly dropped - the free lane exists to hold the paid lane accountable.

A credential the scanner finds is reported by position only. The value is never echoed into the
findings, the exports, the saved history or the model input.

## What costs what

The prescan, the dependency graph, the worked examples and every export are free and need no account.
Running a lane is metered and needs an account; the reservation is shown before you press the button
and the actual charge is usually far lower. Each lane ships a saved example run that replays for
free, so all five renderers can be seen end to end without spending anything.

## Also on this site

- `/api.html` - driving the app programmatically, in cURL, Python, JavaScript, Go, Java, Ruby, PHP and C#
- `/tokens.html` - token management for this app, without opening a developer console

## Provenance

DAG Desk is a derived work built on five agent skills published by Astronomer in the
`@astronomer/agents` repository: `authoring-dags`, `migrating-airflow-2-to-3`,
`annotating-task-lineage`, `testing-dags` and `debugging-dags`.

It is not affiliated with, endorsed by, or maintained by Astronomer, the Apache Software Foundation,
or the Apache Airflow project. Apache Airflow is a trademark of the Apache Software Foundation.