Airflow 3 DAG Authoring: how to write DAGs that survive production

I'm Gabriel Henrique, a data engineer, and I use Apache Airflow to orchestrate pipelines and more than ten data governance automations. In November 2025 I earned Apache Airflow 3 DAG Authoring from Astronomer, one month after Airflow 3 Fundamentals.

Fundamentals covers how Airflow works. DAG Authoring covers how to write it well.

TaskFlow API

With the @task decorator, a Python function becomes a task, and its return value flows to the next one through XCom with no extra code. The DAG reads like a regular script, easier to follow and to review.

from airflow.sdk import dag, task

@dag(schedule="@daily", catchup=False)
def daily_load():
    @task
    def extract():
        return ["file_a.csv", "file_b.csv"]

    @task
    def load(file: str):
        ...

    load.expand(file=extract())

daily_load()

Dynamic task mapping

In the example above, .expand() creates one task per file at runtime. If five files arrive tomorrow, five tasks appear, each with its own log and retries. This replaces hand-written loops inside a single task.

Data-aware scheduling with Assets

In Airflow 3, the old Datasets were renamed Assets. A DAG can run when an asset is updated instead of waiting for a fixed time. For chains of pipelines, this removes the guesswork of "has the previous load finished yet?".

Branching, trigger rules and sensors

• Branching: choosing the DAG's path based on a condition.
• Trigger rules: deciding whether a task runs when the previous one fails or is skipped.
• Deferrable sensors: waiting for an event without holding an idle worker.

Idempotency: the rule that matters most

An idempotent task produces the same result whether it runs once or ten times. In practice:

• Use the run's logical date to filter the data window, not the current time.
• Prefer overwriting the partition or using MERGE over plain inserts.
• Keep retries configured, since the task can safely run again.

With that, reprocessing a failed day takes one click instead of a manual duplicate cleanup.

Test before you ship

dag.test() runs the whole DAG locally in a single process, which makes debugging with breakpoints easy. Together with tests that check every DAG loads without errors, it catches most problems before deployment.

Next step

After this one, I took AI Orchestration Fundamentals, which applies the same patterns to tasks that call AI models.

More orchestration projects are on the portfolio of Gabriel Henrique, Data Engineer. I'm Gabriel Henrique, and I write about Airflow, Databricks and data engineering.