Docker部署airflow
docker部署airflow
系统: CentOS7
airflow版本: 2.9.1
docker版本: 18.09.0
docker-compose版本: 2.27
数据库: MySQL8.0.37
参考文档: docker部署airflow2.9.1
部署要求
- 系统需要有4G以上内存
- docker-compose版本需要在v2.14.0以上
部署
下载docker-compose.yaml
curl -LfO 'https://airflow.apache.org/docs/apache-airflow/2.9.1/docker-compose.yaml'
创建需要挂载的目录
# airflow挂载目录
mkdir -p dags logs plugins config
# MySQL的挂载目录
mkdir mysql/data mysql/conf
设置正常的airflow用户
echo -e "AIRFLOW_UID=$(id -u)" > .env
配置MySQL数据库
将PostgreSQL镜像配置改为MySQL
mysql:
image: mysql:8.0.37
environment:
MYSQL_ROOT_PASSWORD: root
MYSQL_USER: airflow
MYSQL_PASSWORD: airflow
MYSQL_DATABASE: airflow
command:
- --default-authentication-plugin=mysql_native_password # 指定默认的认证插件
- --collation-server=utf8_general_ci # 依据官方指定字符集
- --character-set-server=utf8 # 依据官方指定字符编码
volumes:
- ./mysql/data:/var/lib/mysql # 持久化MySQL数据
- ./mysql/conf/mysql.cnf:/etc/mysql.cnf # 持久化MySQL配置文件
healthcheck:
test: mysql --user=$$MYSQL_USER --password=$$MYSQL_PASSWORD -e 'SHOW DATABASES;'
interval: 30s
retries: 5
restart: always
修改airflow连接MySQL的方式
AIRFLOW__DATABASE__SQL_ALCHEMY_CONN: mysql+mysqldb://airflow:airflow@mysql/airflow
AIRFLOW__CELERY__RESULT_BACKEND: db+mysql://airflow:airflow@mysql/airflow
airflow依赖PostgreSQL改为MySQL
depends_on:
&airflow-common-depends-on
redis:
condition: service_healthy
mysql01:
condition: service_healthy
初始化数据库
docker-compose up airflow-init
创建网络
docker network create wavepoint
启动
docker-compose up