Complete Pipeline Simulation (Test Environment)

In this article, I will focus on only a part of this pipeline above. I put this above image just to show a big picture of my test environment. But, in this second picture, there is the necessary piece for this article.

Airflow launched a new version (2.0.0) with news features, so it’s time to initiate tests before deploying it in production. This article is offering a couple of scripts that can help someone with the same proposition.

Pixabay — EdWhiteImages

My definitions for deployment were:

  • Azure Cloud — Kubernetes Service
  • Kubernetes executor
  • Azure Storage Account (remote log)
  • Azure registry to container images
  • Postgresql container (airflow back-end)
  • Git-Sync (synchronizing dags)

My objective with this article is to help those who are spending a lot of time searching for examples to deploy with the same or equivalent infrastructure. The airflow documentation isn’t sufficient good for help in all scenarios.

First of all, I chose Kubernetes executor because I’m working in a cloud and everyone knows how the pay-as-you-go cloud system works. Then it’s important to economy resources always you can. Then Kubernetes…

Overview about outliers techniques aborted in WestWorld series.

Warning: there is a harmless spoiler for WestWold Season 3 Episode 6.

Initially, a briefing about the subject that I will explore. In WestWorld, a company build a system, called Rehoboam. This is an intelligence artificial software which store data about each person life. Using this data, the Rehoboam can predict the future for each person or the humanity. The system can simulate scenarios and the Rehoboam owners can make decisions and to choose how will be the future for individual person or everyone. …

Felipe Calixto Filho

Data Engineer

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store