Chuyển sang chế độ ngoại tuyến với ứng dụng Player FM !
Orchestrating Analytics and AI Workflows at Telia with Arjun Anandkumar
Manage episode 463948888 series 2948506
The future of data engineering lies in seamless orchestration and automation. In this episode, Arjun Anandkumar, Data Engineer at Telia, shares how his team uses Airflow to drive analytics and AI workflows. He highlights the challenges of scaling data platforms and how adopting best practices can simplify complex processes for teams across the organization. Arjun also discusses the transformative role of tools like Cosmos and Terraform in enhancing efficiency and collaboration.
Key Takeaways:
(02:16) Telia operates across the Nordics and Baltics, focusing on telecom and energy services.
(03:45) Airflow runs dbt models seamlessly with Cosmos on AWS MWAA.
(05:47) Cosmos improves visibility and orchestration in Airflow.
(07:00) Medallion Architecture organizes data into bronze, silver and gold layers.
(08:34) Task group challenges highlight the need for adaptable workflows.
(15:04) Scaling managed services requires trial, error and tailored tweaks.
(19:46) Terraform scales infrastructure, while YAML templates manage DAGs efficiently.
(20:00) Templated DAGs and robust testing enhance platform management.
(24:15) Open-source resources drive innovation in Airflow practices.
Resources Mentioned:
https://www.linkedin.com/in/arjunanand1/?originalSubdomain=dk
Telia -
https://www.linkedin.com/company/teliacompany/
https://airflow.apache.org/
https://www.astronomer.io/cosmos/
https://www.terraform.io/
Medallion Architecture by Databricks -
https://www.databricks.com/glossary/medallion-architecture
Thanks for listening to “The Data Flowcast: Mastering Airflow for Data Engineering & AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.
#AI #Automation #Airflow #MachineLearning
44 tập
Orchestrating Analytics and AI Workflows at Telia with Arjun Anandkumar
The Data Flowcast: Mastering Airflow for Data Engineering & AI
Manage episode 463948888 series 2948506
The future of data engineering lies in seamless orchestration and automation. In this episode, Arjun Anandkumar, Data Engineer at Telia, shares how his team uses Airflow to drive analytics and AI workflows. He highlights the challenges of scaling data platforms and how adopting best practices can simplify complex processes for teams across the organization. Arjun also discusses the transformative role of tools like Cosmos and Terraform in enhancing efficiency and collaboration.
Key Takeaways:
(02:16) Telia operates across the Nordics and Baltics, focusing on telecom and energy services.
(03:45) Airflow runs dbt models seamlessly with Cosmos on AWS MWAA.
(05:47) Cosmos improves visibility and orchestration in Airflow.
(07:00) Medallion Architecture organizes data into bronze, silver and gold layers.
(08:34) Task group challenges highlight the need for adaptable workflows.
(15:04) Scaling managed services requires trial, error and tailored tweaks.
(19:46) Terraform scales infrastructure, while YAML templates manage DAGs efficiently.
(20:00) Templated DAGs and robust testing enhance platform management.
(24:15) Open-source resources drive innovation in Airflow practices.
Resources Mentioned:
https://www.linkedin.com/in/arjunanand1/?originalSubdomain=dk
Telia -
https://www.linkedin.com/company/teliacompany/
https://airflow.apache.org/
https://www.astronomer.io/cosmos/
https://www.terraform.io/
Medallion Architecture by Databricks -
https://www.databricks.com/glossary/medallion-architecture
Thanks for listening to “The Data Flowcast: Mastering Airflow for Data Engineering & AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.
#AI #Automation #Airflow #MachineLearning
44 tập
Tất cả các tập
×Chào mừng bạn đến với Player FM!
Player FM đang quét trang web để tìm các podcast chất lượng cao cho bạn thưởng thức ngay bây giờ. Đây là ứng dụng podcast tốt nhất và hoạt động trên Android, iPhone và web. Đăng ký để đồng bộ các theo dõi trên tất cả thiết bị.