Orchestrating cost-efficient data pipelines at Insignis
Running Airflow with a one-person data engineering team means every decision about cost, tooling, and best practices has to earn its keep.
Podcast profile
By Astronomer
Welcome to The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI— the podcast where we keep you up to date with insights and ideas propelling the Airflow community forward.
Latest episode
31 min
Topics
Building and operating data pipelines, transformations, orchestration, and data platforms.
Data modeling, transformation, semantic layers, metrics, and analytics workflows.
Advanced manufacturing, production systems, factories, and industrial digitalization.
Formats
Detailed technical, architectural, or research-oriented examination.
Host-led conversation where a guest supplies most of the subject matter.
Audience level
Regularly assumes specialist knowledge and discusses implementation, architecture, protocols, code, or research in depth.
Stay connected
From the feed
Episodes shown — 10 of 25
Running Airflow with a one-person data engineering team means every decision about cost, tooling, and best practices has to earn its keep.
Scaling Airflow to over 1,000 active DAGs on a team of 3-4 engineers takes structure.
Running LLMs over sensitive data means external APIs are off the table, but open-weight models still need reproducibility, validation, and a paper trail before downstream systems can depend on their output.
Security is one of the least visible parts of Airflow for most users, but it shapes what features teams can turn on in production.
Orchestrating data across more than 30 companies means most Airflow users on the platform aren't data engineers. In this episode, Kenten Danas talks with Lucas Trubiano, Data Engineer at itti, the technology company within Grupo Vázquez in Paraguay.
Manufacturing data pipelines can't afford silent failures. When quality decisions and shop-floor visibility depend on Airflow, a broken DAG at midnight can cost real money.
Strict sequential execution across DAGs sounds simple until you have scheduled pipelines and event-driven pipelines writing to the same MongoDB collections.
Financial data pipelines have to be right the first time.
Running a travel platform means dealing with fast-moving inventory, real-time fraud detection, and heavy performance marketing attribution, all while keeping data trustworthy for a data-driven org.
Airflow and Spark are one of the most common combinations in modern data platforms, but the orchestration decisions made on the Airflow side often determine whether Spark jobs run fast and cheap or slow and expensive.