Docker Installation

What is Docker? Docker is a set of platform as a service products that use OS-level virtualization to deliver software in packages called containers. Containers are isolated from one another and bundle their own software, libraries and configuration files; they can communicate with each other through well-defined channels. What is Docker Engine? Docker Engine is a client-server application with these major components: A server which is a type of long-running program called a daemon process (the dockerd command).

January 9, 2021 Read
Airflow Dags

Airflow Contents Installation Dags What is DAG Directed Acyclic Graph (DAG) Directed - If multiple tasks exist, each must have at least one defined upstream (previous) or downstream (subsequent) tasks, although they could easily have both. Acyclic - No task can create data that goes on to reference itself. This could cause an infinite loop that would be, um, it’d be bad. Don’t do that. Graph - All tasks are laid out in a clear structure with discrete processes occurring at set points and clear relationships made to other tasks.

January 5, 2021 Read
Airflow Installation

What is Airflow ? Airflow is an open-source workflow management platform written in Python Requirements Python3 Any operating system (Linux, Windows, MacOS) Step 1. Preparation Here I will take Ubuntu for an example. First of all, I will highly recommand you to use virtualenv (venv) sudo apt update && \ sudo apt install -y python3-venv Step 2. Create Python Environment Secondly create a folder airflow and create your env

January 4, 2021 Read