BBC Radio 5 live’s award winning gaming podcast, discussing the world of video games and games culture.
…
continue reading
Player FM - Internet Radio Done Right
Checked 6d ago
Đã thêm cách đây bốn năm
Nội dung được cung cấp bởi The Data Flowcast. Tất cả nội dung podcast bao gồm các tập, đồ họa và mô tả podcast đều được The Data Flowcast hoặc đối tác nền tảng podcast của họ tải lên và cung cấp trực tiếp. Nếu bạn cho rằng ai đó đang sử dụng tác phẩm có bản quyền của bạn mà không có sự cho phép của bạn, bạn có thể làm theo quy trình được nêu ở đây https://vi.player.fm/legal.
Tương tự như The Data Flowcast: Mastering Airflow for Data Engineering & AI
What is the internet doing to us? The Times tech columnist Kevin Roose discovers what happens when our lives move online. Unlock full access to New York Times podcasts and explore everything from politics to pop culture. Subscribe today at nytimes.com/podcasts or on Apple Podcasts and Spotify.
…
continue reading
Android Backstage, a podcast by and for Android developers. Hosted by developers from the Android engineering team, this show covers topics of interest to Android programmers, with in-depth discussions and interviews with engineers on the Android team at Google. Subscribe to Android Developers YouTube → https://goo.gle/AndroidDevs
…
continue reading
Africa-focused technology, digital and innovation ecosystem insight and commentary.
…
continue reading
Show notes are at https://stevelitchfield.com/sshow/chat.html
…
continue reading
The power of Data is undeniable. And unharnessed - it’s nothing but chaos. Making data your ally. Using it to lead with confidence and clarity. Host Jess Carter is solving problems in real-time to reveal what’s possible. Helping communities and people thrive. This is Data Driven Leadership, a show brought to you by Resultant.
…
continue reading
This is the audio podcast version of Troy Hunt's weekly update video published here: https://www.troyhunt.com/tag/weekly-update/
…
continue reading
Redefining AI is the 2024 New York Digital Award winning tech podcast! Discover a whole new take on Artificial Intelligence in joining host Lauren Hawker Zafer, a top voice in Artificial Intelligence on LinkedIn, for insightful chats that unravel the fascinating world of tech innovation, use case exploration and AI knowledge. Dive into candid discussions with accomplished industry experts and established academics. With each episode, you'll expand your grasp of cutting-edge technologies and ...
…
continue reading
Download This Show is your weekly guide to the world of media, culture, and technology. From social media to gadgets, streaming services to privacy issues. Each week Rae Johnston and guests take a fun, deep dive into how technology is reshaping our lives.
…
continue reading
The director’s commentary track for Daring Fireball. Long digressions on Apple, technology, design, movies, and more.
…
continue reading
Player FM - Ứng dụng Podcast
Chuyển sang chế độ ngoại tuyến với ứng dụng Player FM !
Chuyển sang chế độ ngoại tuyến với ứng dụng Player FM !
))
Daily Deals
Overcoming Airflow Scaling Challenges at Monzo Bank with Jonathan Rainer
Manage episode 465365556 series 2948506
Nội dung được cung cấp bởi The Data Flowcast. Tất cả nội dung podcast bao gồm các tập, đồ họa và mô tả podcast đều được The Data Flowcast hoặc đối tác nền tảng podcast của họ tải lên và cung cấp trực tiếp. Nếu bạn cho rằng ai đó đang sử dụng tác phẩm có bản quyền của bạn mà không có sự cho phép của bạn, bạn có thể làm theo quy trình được nêu ở đây https://vi.player.fm/legal.
Scaling a data orchestration platform to manage thousands of tasks daily demands innovative solutions and strategic problem-solving. In this episode, we explore the complexities of scaling Airflow and the challenges of orchestrating thousands of tasks in dynamic data environments. Jonathan Rainer, Former Platform Engineer at Monzo Bank, joins us to share his journey optimizing data pipelines, overcoming UI limitations and ensuring DAG consistency in high-stakes scenarios.
Key Takeaways:
(03:11) Using Airflow to schedule computation in BigQuery.
(07:02) How DAGs with 8,000+ tasks were managed nightly.
(08:18) Ensuring accuracy in regulatory reporting for banking.
(11:35) Handling task inconsistency and DAG failures with automation.
(16:09) Building a service to resolve DAG consistency issues in Airflow.
(25:05) Challenges with scaling the Airflow UI for thousands of tasks.
(27:03) The role of upstream and downstream task management in Airflow.
(37:33) The importance of operational metrics for monitoring Airflow health.
(39:19) Balancing new tools with root cause analysis to address scaling issues.
(41:35) Why scaling solutions require both technical and leadership buy-in
Resources Mentioned:
https://www.linkedin.com/in/jonathan-rainer/
https://www.linkedin.com/company/monzo-bank/
https://airflow.apache.org/
BigQuery -
https://airflow.apache.org/docs/apache-airflow-providers-google/stable/operators/cloud/bigquery.html
https://kubernetes.io/
Thanks for listening to “The Data Flowcast: Mastering Airflow for Data Engineering & AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.
#AI #Automation #Airflow #MachineLearning
51 tập
Overcoming Airflow Scaling Challenges at Monzo Bank with Jonathan Rainer
The Data Flowcast: Mastering Airflow for Data Engineering & AI
Chia sẻManage episode 465365556 series 2948506
Nội dung được cung cấp bởi The Data Flowcast. Tất cả nội dung podcast bao gồm các tập, đồ họa và mô tả podcast đều được The Data Flowcast hoặc đối tác nền tảng podcast của họ tải lên và cung cấp trực tiếp. Nếu bạn cho rằng ai đó đang sử dụng tác phẩm có bản quyền của bạn mà không có sự cho phép của bạn, bạn có thể làm theo quy trình được nêu ở đây https://vi.player.fm/legal.
Scaling a data orchestration platform to manage thousands of tasks daily demands innovative solutions and strategic problem-solving. In this episode, we explore the complexities of scaling Airflow and the challenges of orchestrating thousands of tasks in dynamic data environments. Jonathan Rainer, Former Platform Engineer at Monzo Bank, joins us to share his journey optimizing data pipelines, overcoming UI limitations and ensuring DAG consistency in high-stakes scenarios.
Key Takeaways:
(03:11) Using Airflow to schedule computation in BigQuery.
(07:02) How DAGs with 8,000+ tasks were managed nightly.
(08:18) Ensuring accuracy in regulatory reporting for banking.
(11:35) Handling task inconsistency and DAG failures with automation.
(16:09) Building a service to resolve DAG consistency issues in Airflow.
(25:05) Challenges with scaling the Airflow UI for thousands of tasks.
(27:03) The role of upstream and downstream task management in Airflow.
(37:33) The importance of operational metrics for monitoring Airflow health.
(39:19) Balancing new tools with root cause analysis to address scaling issues.
(41:35) Why scaling solutions require both technical and leadership buy-in
Resources Mentioned:
https://www.linkedin.com/in/jonathan-rainer/
https://www.linkedin.com/company/monzo-bank/
https://airflow.apache.org/
BigQuery -
https://airflow.apache.org/docs/apache-airflow-providers-google/stable/operators/cloud/bigquery.html
https://kubernetes.io/
Thanks for listening to “The Data Flowcast: Mastering Airflow for Data Engineering & AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.
#AI #Automation #Airflow #MachineLearning
51 tập
Tất cả các tập
×
1 From ETL to Airflow: Transforming Data Engineering at Deloitte Digital with Raviteja Tholupunoori 27:42
27:42 
Nghe Sau 
Nghe Sau 
Danh sách 
Thích 
Đã thích27:42
Nghe Sau Nghe Sau Danh sách Thích Đã thíchData orchestration at scale presents unique challenges, especially when aiming for flexibility and efficiency across cloud environments. Choosing the right tools and frameworks can make all the difference. In this episode, Raviteja Tholupunoori, Senior Engineer at Deloitte Digital , joins us to explore how Airflow enhances orchestration, scalability and cost efficiency in enterprise data workflows. Key Takeaways: (01:45) Early challenges in data orchestration before implementing Airflow. (02:42) Comparing Airflow with ETL tools like Talend and why flexibility matters. (04:24) The role of Airflow in enabling cloud-agnostic data processing. (05:45) Key lessons from managing dynamic DAGs at scale. (13:15) How hybrid executors improve performance and efficiency. (14:13) Best practices for testing and monitoring workflows with Airflow. (15:13) The importance of mocking mechanisms when testing DAGs. (17:57) How Prometheus, Grafana and Loki support Airflow monitoring. (22:03) Cost considerations when running Airflow on self-managed infrastructure. (23:14) Airflow’s latest features, including hybrid executors and dark mode. Resources Mentioned: Raviteja Tholupunoori https://www.linkedin.com/in/raviteja0096/?originalSubdomain=in Deloitte Digital https://www.linkedin.com/company/deloitte-digital/ Apache Airflow https://airflow.apache.org/ Grafana https://grafana.com/solutions/apache-airflow/monitor/ Astronomer Presents: Exploring Apache Airflow® 3 Roadshows https://www.astronomer.io/events/roadshow/ https://www.astronomer.io/events/roadshow/london/ https://www.astronomer.io/events/roadshow/new-york/ https://www.astronomer.io/events/roadshow/sydney/ https://www.astronomer.io/events/roadshow/san-francisco/ https://www.astronomer.io/events/roadshow/chicago/ Thanks for listening to “The Data Flowcast: Mastering Airflow for Data Engineering & AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations. #AI #Automation #Airflow #MachineLearning…
1 A Deep Dive Into the 2025 State of Airflow Survey Results with Tamara Fingerlin of Astronomer 23:26
23:26 Nghe Sau Nghe Sau Danh sách Thích Đã thích23:26
Nghe Sau Nghe Sau Danh sách Thích Đã thíchThe 2025 State of Airflow report sheds light on how global users are adopting, evolving and innovating with Apache Airflow. With over 5,000 responses from 116 countries, the survey reveals critical insights into Airflows’ role in business operations, new use cases and what’s ahead for the community. In this episode, Tamara Fingerlin , Developer Advocate at Astronomer , walks us through her process of analyzing survey data, key trends from the report and what to expect from Airflow 3.0. Key Takeaways: (02:14) The State of Airflow report combines anonymized telemetry and survey results. (03:25) The survey received thousands of responses from many countries, showcasing global reach. (04:49) The survey process involves multiple steps, from question selection to report creation. (09:00) Many users expect to increase Airflow usage for revenue-generating or external use cases. (11:04) Experienced users tend to utilize Airflow more for advanced use cases like MLOps. (15:13) UI improvements offer enhanced navigation and error visibility. (18:15) Architectural changes enable new capabilities like remote execution and language support. (19:40) Long-requested features will be available in the new major release. (21:00) Future aspirations include integrating data visualization capabilities into the UI. Resources Mentioned: Tamara Fingerlin https://www.linkedin.com/in/tamara-janina-fingerlin/ Astronomer | LinkedIn https://www.linkedin.com/company/astronomer/ Astronomer | Website https://www.astronomer.io Apache Airflow https://airflow.apache.org/ 2025 State of Airflow Webinar https://www.astronomer.io/airflow/state-of-airflow/ Airflow Slack https://apache-airflow-slack.herokuapp.com/ Astronomer Presents: Exploring Apache Airflow® 3 Roadshows https://www.astronomer.io/events/roadshow/ https://www.astronomer.io/events/roadshow/london/ https://www.astronomer.io/events/roadshow/new-york/ https://www.astronomer.io/events/roadshow/sydney/ https://www.astronomer.io/events/roadshow/san-francisco/ https://www.astronomer.io/events/roadshow/chicago/ Thanks for listening to “The Data Flowcast: Mastering Airflow for Data Engineering & AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations. #AI #Automation #Airflow #MachineLearning…
1 Airflow’s Role in the Rise of DataOps with Andy Byron 26:15
26:15 Nghe Sau Nghe Sau Danh sách Thích Đã thích26:15
Nghe Sau Nghe Sau Danh sách Thích Đã thíchThe orchestration layer is evolving into a critical component of the modern data stack. Understanding its role in DataOps is key to optimizing workflows, improving reliability and reducing complexity. In this episode, Andy Byron , CEO at Astronomer , discusses the rapid growth of Apache Airflow, the increasing importance of orchestration and how Astronomer is shaping the future of DataOps. Key Takeaways: (01:54) Orchestration is central to modern data workflows. (03:16) Airflow 3.0 will enhance usability and flexibility. (05:14) AI-driven workloads demand zero-downtime orchestration. (08:13) DataOps relies on orchestration for seamless operations. (11:05) Integration across ingestion, transformation and governance is key. (17:24) The future of DataOps is consolidation and automation. (19:13) Enterprises use Airflow to process massive data volumes. (23:20) Product innovation is driven by customer needs and feedback. Resources Mentioned: Andy Byron https://www.linkedin.com/in/andy-byron-417a429/ Astronomer | LinkedIn https://www.linkedin.com/company/astronomer/ Astronomer | Website https://www.astronomer.io Apache Airflow https://airflow.apache.org/ State of Airflow Webinar https://www.astronomer.io/events/webinars/the-state-of-airflow-2025-video/ Astronomer Observe https://www.astronomer.io/product/observe/ Astronomer Roadshow: Exploring Apache Airflow 3 | London https://www.astronomer.io/events/roadshow/london/ Astronomer Roadshow: Exploring Apache Airflow 3 | New York https://www.astronomer.io/events/roadshow/new-york/ Astronomer Roadshow: Exploring Apache Airflow 3 | Sydney https://www.astronomer.io/events/roadshow/sydney/ Astronomer Roadshow: Exploring Apache Airflow 3 | San Francisco https://www.astronomer.io/events/roadshow/san-francisco/ Astronomer Roadshow: Exploring Apache Airflow 3 | Chicago https://www.astronomer.io/events/roadshow/chicago/ Thanks for listening to “The Data Flowcast: Mastering Airflow for Data Engineering & AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations. #AI #Automation #Airflow #MachineLearning…
1 The Software Risk That Affects Everyone and How To Address It with Michael Winser and Jarek Potiuk 28:27
28:27 Nghe Sau Nghe Sau Danh sách Thích Đã thích28:27
Nghe Sau Nghe Sau Danh sách Thích Đã thíchThe security of open-source software is a growing concern, especially as dependencies and regulations become more complex, making it essential to understand how to manage software supply chains effectively. In this episode, we sit down with Michael Winser , Co-Founder at Alpha-Omega and Security Strategy Ambassador at Eclipse Foundation , and Jarek Potiuk , Member of the Security Committee at the Apache Software Foundation , to discuss the challenges of securing Airflow’s dependencies, the evolving landscape of open-source security and how contributors can help strengthen the ecosystem. Key Takeaways: (02:43) Jarek quit his full-time engineer position and uses Airflow as a freelancer. (04:32) Michael finds happiness in having meaningful work with open-source security. (07:01) Software supply chain security focuses on correctness, integrity and availability. (08:44) Airflow’s 790 dependencies present a unique security challenge. (09:43) Airflow’s security team has significantly improved its vulnerability response. (10:22) The transition to Airflow 3 emphasizes enterprise security readiness. (16:20) The ‘Three Fs’ approach: fix it, fork it, or forget it. (18:45) Dependency health is often more critical than fixing known vulnerabilities. (23:32) The ‘Three Fs’ in action. (26:26) Open-source contributors play a key role in supply chain security. Resources Mentioned: Michael Winser - https://www.linkedin.com/in/michaelw/ Jarek Potiuk - https://www.linkedin.com/in/jarekpotiuk/ Apache Airflow - https://airflow.apache.org/ Apache Software Foundation | LinkedIn - https://www.linkedin.com/company/the-apache-software-foundation/ Apache Software Foundation | Website - https://www.apache.org/ Eclipse Foundation | LinkedIn - https://www.linkedin.com/company/eclipse-foundation/ Eclipse Foundation | Website - https://www.eclipse.org/org/foundation/ OpenSSF Working Groups - https://openssf.org/community/openssf-working-groups/ Astronomer Roadshow: Exploring Apache Airflow 3 | London https://www.astronomer.io/events/roadshow/london/ Astronomer Roadshow: Exploring Apache Airflow 3 | New York https://www.astronomer.io/events/roadshow/new-york/ Astronomer Roadshow: Exploring Apache Airflow 3 | Sydney https://www.astronomer.io/events/roadshow/sydney/ Astronomer Roadshow: Exploring Apache Airflow 3 | San Francisco https://www.astronomer.io/events/roadshow/san-francisco/ Astronomer Roadshow: Exploring Apache Airflow 3 | Chicago https://www.astronomer.io/events/roadshow/chicago/ Thanks for listening to “The Data Flowcast: Mastering Airflow for Data Engineering & AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations. #AI #Automation #Airflow #MachineLearning…
1 Building Scalable ML Infrastructure at Outerbounds with Savin Goyal 36:46
36:46 Nghe Sau Nghe Sau Danh sách Thích Đã thích36:46
Nghe Sau Nghe Sau Danh sách Thích Đã thíchMachine learning is changing fast, and companies need better tools to handle AI workloads. The right infrastructure helps data scientists focus on solving problems instead of managing complex systems. In this episode, we talk with Savin Goyal , Co-Founder and CTO at Outerbounds , about building ML infrastructure, how orchestration makes workflows easier and how Metaflow and Airflow work together to simplify data science. Key Takeaways: (02:02) Savin spent years building AI and ML infrastructure, including at Netflix. (04:05) ML engineering was not a defined role a decade ago. (08:17) Modernizing AI and ML requires balancing new tools with existing strengths. (10:28) ML workloads can be long-running or require heavy computation. (15:29) Different teams at Netflix used multiple orchestration systems for specific needs. (20:10) Stable APIs prevent rework and keep projects moving. (21:07) Metaflow simplifies ML workflows by optimizing data and compute interactions. (25:53) Limited local computing power makes running ML workloads challenging. (27:43) Airflow UI monitors pipelines, while Metaflow UI gives ML insights. (33:13) The most successful data professionals focus on business impact, not just technology. Resources Mentioned: Savin Goyal - https://www.linkedin.com/in/savingoyal/ Outerbounds - https://www.linkedin.com/company/outerbounds/ Apache Airflow - https://airflow.apache.org/ Metaflow - https://metaflow.org/ Netflix’s Maestro Orchestration System - https://netflixtechblog.com/maestro-netflixs-workflow-orchestrator-ee13a06f9c78?gi=8e6a067a92e9#:~:text=Maestro%20is%20a%20fully%20managed,data%20between%20different%20storages%2C%20etc. TensorFlow - https://www.tensorflow.org/ PyTorch - https://pytorch.org/ Thanks for listening to “The Data Flowcast: Mastering Airflow for Data Engineering & AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations. #AI #Automation #Airflow #MachineLearning…
1 Customizing Airflow for Complex Data Environments at Stripe with Nick Bilozerov and Sharadh Krishnamurthy 27:40
27:40 Nghe Sau Nghe Sau Danh sách Thích Đã thích27:40
Nghe Sau Nghe Sau Danh sách Thích Đã thíchKeeping data pipelines reliable at scale requires more than just the right tools — it demands constant innovation. In this episode, Nick Bilozerov , Senior Data Engineer at Stripe , and Sharadh Krishnamurthy , Engineering Manager at Stripe, discuss how Stripe customizes Airflow for its needs, the evolution of its data orchestration framework and the transition to Airflow 2. They also share insights on scaling data workflows while maintaining performance, reliability and developer experience. Key Takeaways: (02:04) Stripe’s mission is to grow the GDP of the internet by supporting businesses with payments and data. (05:08) 80% of Stripe engineers use data orchestration, making scalability critical. (06:06) Airflow powers business reports, regulatory needs and ML workflows. (08:02) Custom task frameworks improve dependencies and validation. (08:50) "User scope mode" enables local testing without production impact. (10:39) Migrating to Airflow 2 improves isolation, safety and scalability. (16:40) Monolithic DAGs caused database issues, prompting a service-based shift. (19:24) Frequent Airflow upgrades ensure stability and access to new features. (21:38) DAG versioning and backfill improvements enhance developer experience. (23:38) Greater UI customization would offer more flexibility. Resources Mentioned: Nick Bilozerov - https://www.linkedin.com/in/nick-bilozerov/ Sharadh Krishnamurthy - https://www.linkedin.com/in/sharadhk/ Apache Airflow - https://airflow.apache.org/ Stripe | LinkedIn - https://www.linkedin.com/company/stripe/ Stripe | Website - https://stripe.com/ Thanks for listening to “The Data Flowcast: Mastering Airflow for Data Engineering & AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations. #AI #Automation #Airflow #MachineLearning…
1 Harnessing Airflow for Data-Driven Policy Research at CSET with Jennifer Melot 17:54
17:54 Nghe Sau Nghe Sau Danh sách Thích Đã thích17:54
Nghe Sau Nghe Sau Danh sách Thích Đã thíchTurning complex datasets into meaningful analysis requires robust data infrastructure and seamless orchestration. In this episode, we’re joined by Jennifer Melot , Technical Lead at the Center for Security and Emerging Technology (CSET) at Georgetown University, to explore how Airflow powers data-driven insights in technology policy research. Jennifer shares how her team automates workflows to support analysts in navigating complex datasets. Key Takeaways: (02:04) CSET provides data-driven analysis to inform government decision-makers. (03:54) ETL pipelines merge multiple data sources for more comprehensive insights. (04:20) Airflow is central to automating and streamlining large-scale data ingestion. (05:11) Larger-scale databases create challenges that require scalable solutions. (07:20) Dynamic DAG generation simplifies Airflow adoption for non-engineers. (12:13) DAG Factory and dynamic task mapping can improve workflow efficiency. (15:46) Tracking data lineage helps teams understand dependencies across DAGs. (16:14) New Airflow features enhance visibility and debugging for complex pipelines. Resources Mentioned: Jennifer Melot - https://www.linkedin.com/in/jennifer-melot-aa710144/ Center for Security and Emerging Technology (CSET) - https://www.linkedin.com/company/georgetown-cset/ Apache Airflow - https://airflow.apache.org/ Zenodo - https://zenodo.org/ OpenLineage - https://openlineage.io/ Cloud Dataplex - https://cloud.google.com/dataplex Thanks for listening to “The Data Flowcast: Mastering Airflow for Data Engineering & AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations. #AI #Automation #Airflow #MachineLearning…
1 Leveraging Airflow To Build Scalable and Reliable Data Platforms at 99acres.com with Samyak Jain 25:08
25:08 Nghe Sau Nghe Sau Danh sách Thích Đã thích25:08
Nghe Sau Nghe Sau Danh sách Thích Đã thíchData orchestration is evolving rapidly, with dynamic workflows becoming the cornerstone of modern data engineering. In this episode, we are joined by Samyak Jain , Senior Software Engineer - Big Data at 99acres.com . Samyak shares insights from his journey with Apache Airflow, exploring how his team built a self-service platform that enables non-technical teams to launch data pipelines and marketing campaigns seamlessly. Key Takeaways: (02:02) Starting a career in data engineering by troubleshooting Airflow pipelines. (04:27) Building self-service portals with Airflow as the backend engine. (05:34) Utilizing API endpoints to trigger dynamic DAGs with parameterized templates. (09:31) Managing a dynamic environment with over 1,400 active DAGs. (11:14) Implementing fault tolerance by segmenting data workflows into distinct layers. (14:15) Tracking and optimizing query costs in AWS Athena to save $7K monthly. (16:22) Automating cost monitoring with real-time alerts for high-cost queries. (17:15) Streamlining Airflow metadata cleanup to prevent performance bottlenecks. (21:30) Efficiently handling one-time and recurring marketing campaigns using Airflow. (24:18) Advocating for Airflow features that improve resource management and ownership tracking. Resources Mentioned: Samyak Jain - https://www.linkedin.com/in/samyak-jain-ab5830169/ 99acres.com - https://www.linkedin.com/company/99acres/ Apache Airflow - https://airflow.apache.org/ AWS Athena - https://aws.amazon.com/athena/ Kafka - https://kafka.apache.org/ Thanks for listening to “The Data Flowcast: Mastering Airflow for Data Engineering & AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations. #AI #Automation #Airflow #MachineLearning…
1 Hybrid Testing Solutions for Autonomous Driving at Bosch with Jens Scheffler and Christian Schilling 33:45
33:45 Nghe Sau Nghe Sau Danh sách Thích Đã thích33:45
Nghe Sau Nghe Sau Danh sách Thích Đã thíchTesting autonomous vehicles demands precision, scalability and powerful orchestration tools — enter Apache Airflow, a key component of Bosch’s cutting-edge testing framework. In this episode, we sit down with Jens Scheffler , Test Execution Cluster Technical Architect, and Christian Schilling , Product Owner Open Loop Testing Automated Driving, both at Bosch , to explore how Bosch harnesses Airflow to streamline complex testing scenarios. They share insights on scaling workflows, integrating hybrid infrastructures and ensuring vehicle safety through rigorous automated testing. Key Takeaways: (01:35) Airflow orchestrates millions of test hours for autonomous systems. (03:15) Jens scales distributed systems with Kubernetes for job orchestration. (06:02) Airflow runs hundreds of tests simultaneously. (06:44) Virtual testing reduces costs and on-road trials. (12:19) Unified APIs and GUIs streamline operations. (15:05) Self-service setups empower Bosch teams. (18:00) Physical hardware integration ensures real-world timing. (20:30) Dynamic task mapping scales workflows efficiently. (25:22) Open-source contributions improve stability. (31:06) Edge and Celery executors power Bosch's hybrid scheduling. Resources Mentioned: Jens Scheffler - https://www.linkedin.com/in/jens-scheffler/ Christian Schilling - https://www.linkedin.com/in/christian-schilling-a5078831a/ Bosch - https://www.linkedin.com/company/bosch/ Apache Airflow - https://airflow.apache.org/ Kubernetes - https://kubernetes.io GitHub - https://github.com Edge Executor - https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/executor/index.html Thanks for listening to “The Data Flowcast: Mastering Airflow for Data Engineering & AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations. #AI #Automation #Airflow #MachineLearning…
1 Overcoming Airflow Scaling Challenges at Monzo Bank with Jonathan Rainer 43:39
43:39 Nghe Sau Nghe Sau Danh sách Thích Đã thích43:39
Nghe Sau Nghe Sau Danh sách Thích Đã thíchScaling a data orchestration platform to manage thousands of tasks daily demands innovative solutions and strategic problem-solving. In this episode, we explore the complexities of scaling Airflow and the challenges of orchestrating thousands of tasks in dynamic data environments. Jonathan Rainer , Former Platform Engineer at Monzo Bank , joins us to share his journey optimizing data pipelines, overcoming UI limitations and ensuring DAG consistency in high-stakes scenarios. Key Takeaways: (03:11) Using Airflow to schedule computation in BigQuery. (07:02) How DAGs with 8,000+ tasks were managed nightly. (08:18) Ensuring accuracy in regulatory reporting for banking. (11:35) Handling task inconsistency and DAG failures with automation. (16:09) Building a service to resolve DAG consistency issues in Airflow. (25:05) Challenges with scaling the Airflow UI for thousands of tasks. (27:03) The role of upstream and downstream task management in Airflow. (37:33) The importance of operational metrics for monitoring Airflow health. (39:19) Balancing new tools with root cause analysis to address scaling issues. (41:35) Why scaling solutions require both technical and leadership buy-in Resources Mentioned: Jonathan Rainer - https://www.linkedin.com/in/jonathan-rainer/ Monzo Bank - https://www.linkedin.com/company/monzo-bank/ Apache Airflow - https://airflow.apache.org/ BigQuery - https://airflow.apache.org/docs/apache-airflow-providers-google/stable/operators/cloud/bigquery.html Kubernetes - https://kubernetes.io/ Thanks for listening to “The Data Flowcast: Mastering Airflow for Data Engineering & AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations. #AI #Automation #Airflow #MachineLearning…
1 Orchestrating Analytics and AI Workflows at Telia with Arjun Anandkumar 26:00
26:00 Nghe Sau Nghe Sau Danh sách Thích Đã thích26:00
Nghe Sau Nghe Sau Danh sách Thích Đã thíchT he future of data engineering lies in seamless orchestration and automation. In this episode, Arjun Anandkumar , Data Engineer at Telia , shares how his team uses Airflow to drive analytics and AI workflows. He highlights the challenges of scaling data platforms and how adopting best practices can simplify complex processes for teams across the organization. Arjun also discusses the transformative role of tools like Cosmos and Terraform in enhancing efficiency and collaboration. Key Takeaways: (02:16) Telia operates across the Nordics and Baltics, focusing on telecom and energy services. (03:45) Airflow runs dbt models seamlessly with Cosmos on AWS MWAA. (05:47) Cosmos improves visibility and orchestration in Airflow. (07:00) Medallion Architecture organizes data into bronze, silver and gold layers. (08:34) Task group challenges highlight the need for adaptable workflows. (15:04) Scaling managed services requires trial, error and tailored tweaks. (19:46) Terraform scales infrastructure, while YAML templates manage DAGs efficiently. (20:00) Templated DAGs and robust testing enhance platform management. (24:15) Open-source resources drive innovation in Airflow practices. Resources Mentioned: Arjun Anandkumar - https://www.linkedin.com/in/arjunanand1/?originalSubdomain=dk Telia - https://www.linkedin.com/company/teliacompany/ Apache Airflow - https://airflow.apache.org/ Cosmos by Astronomer - https://www.astronomer.io/cosmos/ Terraform - https://www.terraform.io/ Medallion Architecture by Databricks - https://www.databricks.com/glossary/medallion-architecture Thanks for listening to “The Data Flowcast: Mastering Airflow for Data Engineering & AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations. #AI #Automation #Airflow #MachineLearning…
1 The Role of Airflow in Finance Transformation at Etraveli Group with Mihir Samant 21:19
21:19 Nghe Sau Nghe Sau Danh sách Thích Đã thích21:19
Nghe Sau Nghe Sau Danh sách Thích Đã thíchTransforming bottlenecked finance processes into streamlined, automated systems requires the right tools and a forward-thinking approach. In this episode, Mihir Samant , Senior Data Analyst at Etraveli Group , joins us to share how his team leverages Airflow to revolutionize finance automation. With extensive experience in data workflows and a passion for open-source tools, Mihir provides valuable insights into building efficient, scalable systems. We explore the transformative power of Airflow in automating workflows and enhancing data orchestration within the finance domain. Key Takeaways: (02:14) Etraveli Group specializes in selling affordable flight tickets and ancillary services. (03:56) Mihir’s finance automation team uses Airflow to tackle month-end bottlenecks. (06:00) Airflow's flexibility enables end-to-end automation for finance workflows. (07:00) Open-source Airflow tools offer cost-effective solutions for new teams. (08:46) Sensors and dynamic DAGs are pivotal features for optimizing tasks. (13:30) GitSync simplifies development by syncing environments seamlessly. (16:27) Plans include integrating Databricks for more advanced data handling. (17:58) Airflow and Databricks offer multiple flexible methods to trigger workflows and execute SQL queries seamlessly. Resources Mentioned: Mihir Samant - https://www.linkedin.com/in/misamant/?originalSubdomain=ca Etraveli Group - https://www.linkedin.com/company/etraveli-group/ Apache Airflow - https://airflow.apache.org/ Docker - https://www.docker.com/ Databricks - https://www.databricks.com/ Thanks for listening to “The Data Flowcast: Mastering Airflow for Data Engineering & AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations. #AI #Automation #Airflow #MachineLearning…
1 Inside Ford’s Data Transformation: Advanced Orchestration Strategies with Vasantha Kosuri-Marshall 38:54
38:54 Nghe Sau Nghe Sau Danh sách Thích Đã thích38:54
Nghe Sau Nghe Sau Danh sách Thích Đã thíchData engineering is entering a new era, where orchestration and automation are redefining how large-scale projects operate. This episode features Vasantha Kosuri-Marshall , Data and ML Ops Engineer at Ford Motor Company . Vasantha shares her expertise in managing complex data pipelines. She takes us through Ford's transition to cloud platforms, the adoption of Airflow and the intricate challenges of orchestrating data in a diverse environment. Key Takeaways: (03:10) Vasantha’s transition to the Advanced Driving Assist Systems team at Ford. (05:42) Early adoption of Airflow to orchestrate complex data pipelines. (09:29) Ford's move from on-premise data solutions to Google Cloud Platform. (12:03) The importance of Airflow's scheduling capabilities for efficient data management. (16:12) Using Kubernetes to scale Airflow for large-scale data processing. (19:59) Vasantha’s experience in overcoming challenges with legacy orchestration tools. (22:22) Integration of data engineering and data science pipelines at Ford. (28:03) How deferrable operators in Airflow improve performance and save costs. (32:12) Vasantha’s insights into tuning Airflow properties for thousands of DAGs. (36:09) The significance of monitoring and observability in managing Airflow instances. Resources Mentioned: Vasantha Kosuri-Marshall - https://www.linkedin.com/in/vasantha-kosuri-marshall-0b0aab188/ Apache Airflow - https://airflow.apache.org/ Google Cloud Platform (GCP) - https://cloud.google.com/ Ford Motor Company | LinkedIn - https://www.linkedin.com/company/ford-motor-company/ Ford Motor Company | Website - https://www.ford.com/ Astronomer - https://www.astronomer.io/ Thanks for listening to “The Data Flowcast: Mastering Airflow for Data Engineering & AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations. #AI #Automation #Airflow #MachineLearning…
1 Powering Finance With Advanced Data Solutions at Ramp with Ryan Delgado 24:35
24:35 Nghe Sau Nghe Sau Danh sách Thích Đã thích24:35
Nghe Sau Nghe Sau Danh sách Thích Đã thíchData is the backbone of every modern business, but unlocking its full potential requires the right tools and strategies. In this episode, Ryan Delgado , Director of Engineering at Ramp , joins us to explore how innovative data platforms can transform business operations and fuel growth. He shares insights on integrating Apache Airflow, optimizing data workflows and leveraging analytics to enhance customer experiences. Key Takeaways: (01:52) Data is the lifeblood of Ramp, touching every vertical in the company. (03:18) Ramp’s data platform team enables high-velocity scaling through tailored tools. (05:27) Airflow powers Ramp’s enterprise data warehouse integrations for advanced analytics. (07:55) Centralizing data in Snowflake simplifies storage and analytics pipelines. (12:08) Machine learning models at Ramp integrate seamlessly with Airflow for operational excellence. (14:11) Leveraging Airflow datasets eliminates inefficiencies in DAG dependencies. (17:22) Platforms evolve from solving narrow business problems to scaling organizationally. (18:55) ClickHouse enhances Ramp’s OLAP capabilities with 100x performance improvements. (19:47) Ramp’s OLAP platform improves performance by reducing joins and leveraging ClickHouse. (21:46) Ryan envisions a lighter-weight, more Python-native future for Airflow. Resources Mentioned: Ryan Delgado - https://www.linkedin.com/in/ryan-delgado-69544568/ Ramp - https://www.linkedin.com/company/ramp/ Apache Airflow - https://airflow.apache.org/ Snowflake - https://www.snowflake.com/ ClickHouse - https://clickhouse.com/ dbt - https://www.getdbt.com/ Thanks for listening to “The Data Flowcast: Mastering Airflow for Data Engineering & AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations. #AI #Automation #Airflow #MachineLearning…
1 Exploring the Power of Airflow 3 at Astronomer with Amogh Desai 30:24
30:24 Nghe Sau Nghe Sau Danh sách Thích Đã thích30:24
Nghe Sau Nghe Sau Danh sách Thích Đã thíchWhat does it take to go from fixing a broken link to becoming a committer for one of the world’s leading open-source projects? Amogh Desai , Senior Software Engineer at Astronomer , takes us through his journey with Apache Airflow. From small contributions to building meaningful connections in the open-source community, Amogh’s story provides actionable insights for anyone on the cusp of their open-source journey. Key Takeaways: (02:09) Building data engineering platforms at Cloudera with Kubernetes. (04:00) Brainstorming led to contributing to Apache Airflow. (05:17) Starting small with link fixes, progressing to Breeze development. (07:00) Becoming a committer for Apache Airflow in September 2023. (09:51) The steep learning curve for contributing to Airflow. (16:30) Using GitHub’s “good-first-issue” label to get started. (18:15) Setting up a development environment with Breeze. (22:00) Open-source contributions enhance your resume and career. (24:51) Amogh’s advice: Start small and stay consistent. (28:12) Engage with the community via Slack, email lists and meetups. Resources Mentioned: Amogh Desai - https://www.linkedin.com/in/amogh-desai-385141157/?originalSubdomain=in%20%20https://www.linkedin.com/company/astronomer/ Astronomer - https://www.linkedin.com/company/astronomer/ Apache Airflow GitHub Repository - https://github.com/apache/airflow Contributors Quick Guide - https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst Breeze Development Tool - https://github.com/apache/airflow/tree/main/dev/breeze Thanks for listening to “The Data Flowcast: Mastering Airflow for Data Engineering & AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations. #AI #Automation #Airflow #MachineLearning…
Chào mừng bạn đến với Player FM!
Player FM đang quét trang web để tìm các podcast chất lượng cao cho bạn thưởng thức ngay bây giờ. Đây là ứng dụng podcast tốt nhất và hoạt động trên Android, iPhone và web. Đăng ký để đồng bộ các theo dõi trên tất cả thiết bị.
Daily Deals
Tương tự như The Data Flowcast: Mastering Airflow for Data Engineering & AI
BBC Radio 5 live’s award winning gaming podcast, discussing the world of video games and games culture.
…
continue reading
What is the internet doing to us? The Times tech columnist Kevin Roose discovers what happens when our lives move online. Unlock full access to New York Times podcasts and explore everything from politics to pop culture. Subscribe today at nytimes.com/podcasts or on Apple Podcasts and Spotify.
…
continue reading
Android Backstage, a podcast by and for Android developers. Hosted by developers from the Android engineering team, this show covers topics of interest to Android programmers, with in-depth discussions and interviews with engineers on the Android team at Google. Subscribe to Android Developers YouTube → https://goo.gle/AndroidDevs
…
continue reading
Africa-focused technology, digital and innovation ecosystem insight and commentary.
…
continue reading
Show notes are at https://stevelitchfield.com/sshow/chat.html
…
continue reading
The power of Data is undeniable. And unharnessed - it’s nothing but chaos. Making data your ally. Using it to lead with confidence and clarity. Host Jess Carter is solving problems in real-time to reveal what’s possible. Helping communities and people thrive. This is Data Driven Leadership, a show brought to you by Resultant.
…
continue reading
This is the audio podcast version of Troy Hunt's weekly update video published here: https://www.troyhunt.com/tag/weekly-update/
…
continue reading
Redefining AI is the 2024 New York Digital Award winning tech podcast! Discover a whole new take on Artificial Intelligence in joining host Lauren Hawker Zafer, a top voice in Artificial Intelligence on LinkedIn, for insightful chats that unravel the fascinating world of tech innovation, use case exploration and AI knowledge. Dive into candid discussions with accomplished industry experts and established academics. With each episode, you'll expand your grasp of cutting-edge technologies and ...
…
continue reading
Download This Show is your weekly guide to the world of media, culture, and technology. From social media to gadgets, streaming services to privacy issues. Each week Rae Johnston and guests take a fun, deep dive into how technology is reshaping our lives.
…
continue reading
The director’s commentary track for Daring Fireball. Long digressions on Apple, technology, design, movies, and more.
…
continue reading
Player FM - Ứng dụng Podcast
Chuyển sang chế độ ngoại tuyến với ứng dụng Player FM !
Chuyển sang chế độ ngoại tuyến với ứng dụng Player FM !
