What is Airflow ?

Airflow is an open-source workflow management platform written in Python

Requirements

  • Python3
  • Any operating system (Linux, Windows, MacOS)

Step 1. Preparation

Here I will take Ubuntu for an example. First of all, I will highly recommand you to use virtualenv (venv)

sudo apt update && \
sudo apt install -y python3-venv

Step 2. Create Python Environment

Secondly create a folder airflow and create your env

$ mkdir $HOME/airflow
$ cd $HOME/airflow
$ python3 -m venv env
$ source env/bin/activate

# update pip 
(env)$ pip3 install -U pip

Step 3. Install apache-airflow

After install airflow by pip, try to type airflow and all configurations will appear in the folder

(env)$ pip install apache-airflow

(env)$ airflow

Your folder will involve files as follows:

$ tree
├── airflow.cfg
├── env
├── logs
├── unittests.cfg
└── webserver_config.py

Step 4. Initialize db and Create web user(default: sqlite)

airflow.db (sqlite file) will be created after enter the following command

$ airflow db init

after sucessfully initialize database, we can start creating our admin user for airflow webserver

$ airflow users create \
      --username admin \
      --firstname FIRST_NAME \
      --lastname LAST_NAME \
      --role Admin \
      --email admin@example.org

Step 5. Start webserver and scheduler

Open two new command line terminals

$ airflow webserver

$ airflow scheduler

Step 6. Enter your Airflow website

Open your broswer and Enter URL: localhost:8080

Optional 1. Turn off official examples

line 98

load_examples = True \

load_examples = False

Optional 2. Change type of database

line 29

sql_alchemy_conn = sqlite:////home/arthur/airflow/airflow.db \

sql_alchemy_conn = postgresql+psycopg2://USERNAME:PASSWORD@IP_ADDRESS:PORT/DATABASE

After changing the config, you should install required pip libraries and reinitialize database as well as creating user for airflow webserver in Step 5.

(env)$ pip install apache-airflow[postgresql]

Reference

To get more useful information related to the configuation please check the official website here