Dockerized Apache Airflow Setup for Local Development

A Dockerized Apache Airflow setup designed for straightforward orchestration and management of workflows in a local development environment. This repository provides a Docker Compose configuration integrating Airflow with PostgreSQL and Redis, enabling rapid deployment and testing.

Features

• Complete Docker Compose environment including Airflow components, PostgreSQL, and Redis
• Custom Dockerfile to build Airflow images tailored for this setup
• Sample DAGs demonstrating basic and advanced workflow orchestration
• Fernet key generation script to secure Airflow metadata
• Organized directories for DAGs, logs, plugins, and SQL scripts

Tech Stack

• Apache Airflow 2.x
• Docker & Docker Compose
• PostgreSQL 13
• Redis 6.2
• Python (for DAG definitions and utility scripts)
• Neo4j (integrated in sample DAG for graph database operations)

Getting Started

Prerequisites

• Docker
• Docker Compose

Installation & Running

1. Clone the repository:

git clone https://github.com/justin-napolitano/airflow-docker.git
cd airflow-docker

2. Generate a Fernet key (used by Airflow for encryption):

python3 fernet_key_generator.py

3. Export the Fernet key and set the SQL Alchemy connection string for PostgreSQL in your shell environment:

export AIRFLOW__CORE__FERNET_KEY="<your_generated_fernet_key>"
export AIRFLOW__DATABASE__SQL_ALCHEMY_CONN="postgresql+psycopg2://airflow:airflow@postgres/airflow"

4. Build and start the Docker containers:

docker-compose up --build

5. Access the Airflow webserver at http://localhost:8089.

6. Place your DAG files inside the dags/ directory to have them automatically loaded.

Project Structure

airflow-docker/
├── dags/                    # Airflow DAG definitions
│   ├── hello-world.py       # Example DAG printing "Hello, world!"
│   └── sup_court_graph_workflow.py  # DAG interacting with Neo4j graph database
├── logs/                   # Airflow logs
├── plugins/                # Custom Airflow plugins (empty by default)
├── sql/                    # Cypher and SQL scripts for workflows
├── Dockerfile              # Custom Dockerfile to build Airflow image
├── docker-compose.yml      # Docker Compose configuration
├── fernet_key_generator.py # Script to generate Fernet key
├── generate_fernet_key.sh  # Shell script alternative to generate Fernet key
├── README.md               # This file
├── requirements.txt        # Python dependencies

Future Work / Roadmap

• Expand sample DAGs with more complex workflows and integrations
• Add automated tests for DAGs and environment setup
• Provide support for additional Airflow plugins and operators
• Enhance documentation with troubleshooting and advanced configuration guides
• Explore deployment options beyond local Docker, e.g., Kubernetes