Docker部署airflow

docker部署airflow

系统: CentOS7

airflow版本: 2.9.1

docker版本: 18.09.0

docker-compose版本: 2.27

数据库: MySQL8.0.37

参考文档: docker部署airflow2.9.1

部署要求

  • 系统需要有4G以上内存
  • docker-compose版本需要在v2.14.0以上

部署

下载docker-compose.yaml

curl -LfO 'https://airflow.apache.org/docs/apache-airflow/2.9.1/docker-compose.yaml'

创建需要挂载的目录

# airflow挂载目录
mkdir -p dags logs plugins config

# MySQL的挂载目录
mkdir mysql/data mysql/conf

设置正常的airflow用户

echo -e "AIRFLOW_UID=$(id -u)" > .env

配置MySQL数据库

将PostgreSQL镜像配置改为MySQL


  mysql:
    image: mysql:8.0.37
    environment:
      MYSQL_ROOT_PASSWORD: root
      MYSQL_USER: airflow
      MYSQL_PASSWORD: airflow
      MYSQL_DATABASE: airflow
    command:
      - --default-authentication-plugin=mysql_native_password  # 指定默认的认证插件
      - --collation-server=utf8_general_ci  # 依据官方指定字符集
      - --character-set-server=utf8  # 依据官方指定字符编码
    volumes:
      - ./mysql/data:/var/lib/mysql # 持久化MySQL数据
      - ./mysql/conf/mysql.cnf:/etc/mysql.cnf # 持久化MySQL配置文件
    healthcheck:
      test: mysql --user=$$MYSQL_USER --password=$$MYSQL_PASSWORD -e 'SHOW DATABASES;'
      interval: 30s
      retries: 5
    restart: always

修改airflow连接MySQL的方式

AIRFLOW__DATABASE__SQL_ALCHEMY_CONN: mysql+mysqldb://airflow:airflow@mysql/airflow
AIRFLOW__CELERY__RESULT_BACKEND: db+mysql://airflow:airflow@mysql/airflow

airflow依赖PostgreSQL改为MySQL

  depends_on:
    &airflow-common-depends-on
    redis:
      condition: service_healthy
    mysql01:
      condition: service_healthy

初始化数据库

docker-compose up airflow-init

创建网络

docker network create wavepoint

启动

docker-compose up