The Role Of Airflow In Building Smarter ML Pipelines At Vivian Health With Max Calehuff The Data Flowcast: Mastering Apache Airflow ® For Data Engineering And AI podcast

Nội dung được cung cấp bởi The Data Flowcast. Tất cả nội dung podcast bao gồm các tập, đồ họa và mô tả podcast đều được The Data Flowcast hoặc đối tác nền tảng podcast của họ tải lên và cung cấp trực tiếp. Nếu bạn cho rằng ai đó đang sử dụng tác phẩm có bản quyền của bạn mà không có sự cho phép của bạn, bạn có thể làm theo quy trình được nêu ở đây https://vi.player.fm/legal.

The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI »
The Role of Airflow in Building Smarter ML Pipelines at Vivian Health with Max Calehuff

7d ago 19:30

Chia sẻ

MP3•Trang chủ episode

The integration of data orchestration and machine learning is critical to operational efficiency in healthcare tech. Vivian Health leverages Airflow to power both its ETL pipelines and ML workflows while maintaining strict compliance standards.

Max Calehuff, Lead Data Engineer at Vivian Health, joins us to discuss how his team uses Airflow for ML ops, regulatory compliance and large-scale data orchestration. He also shares insights into upgrading to Airflow 3 and the importance of balancing flexibility with security in a healthcare environment.

Key Takeaways:

00:00 Introduction.

04:21 The role of Airflow in managing ETL pipelines and ML retraining.

06:23 Using AWS SageMaker for ML training and deployment.

07:47 Why Airflow’s versatility makes it ideal for MLOps.

10:50 The importance of documentation and best practices for engineering teams.

13:44 Automating anonymization of user data for compliance.

15:30 The benefits of remote execution in Airflow 3 for regulated industries.

18:16 Quality-of-life improvements and desired features in future Airflow versions.

Resources Mentioned:

Max Calehuff

https://www.linkedin.com/in/maxwell-calehuff/

Vivian Health | LinkedIn

https://www.linkedin.com/company/vivianhealth/

Vivian Health | Website

https://www.vivian.com

Apache Airflow

https://airflow.apache.org/

Astronomer

https://www.astronomer.io/

AWS SageMaker

https://www.google.com/aclk?sa=L&ai=DChsSEwj3-fbz1tiQAxWXlKYDHXUBBVoYACICCAEQABoCdGI&ae=2&aspm=1&co=1&ase=2&gclid=Cj0KCQiA5abIBhCaARIsAM3-zFWbfj2olUvX4dqoiYNaE3q2fMf_ZifRjmbKNQCVX7D6ZMClaUXUkFkaAuwmEALw_wcB&cid=CAASQuRoMccxWhBvMq-1Uez3XOZti1ul7mTDotKvSMoDHv0q2xCsyS2FzMptO5dJf3tmfkLRu22TtD8ChTmdjvs6YetTjQ&cce=2&category=acrcp_v1_35&sig=AOD64_2xE2xolEEVbpDb56qXQluxTzs-Aw&q&nis=4&adurl&ved=2ahUKEwj7le3z1tiQAxWXcvUHHfZePbAQ0Qx6BAgUEAE

dbtLabs

https://www.getdbt.com/

Cosmos

https://github.com/astronomer/astronomer-cosmos

Split

https://www.split.io/

Snowflake

https://www.snowflake.com/en/

Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.

#AI #Automation #Airflow

82 tập

#Tech #Airflow