Hot and cold data with Apache Kafka, Tiered Storage, and Iceberg

Data (R)evolution

Utilizing the true potential of data streaming is key to business success.

In this Data (R)evolution episode, we're joined by Josep Prat and Filip Yonov to dive into the transformative features of Apache Kafka and its evolving role in data architecture. They discuss the critical importance of collaboration and feedback in enhancing Kafka's capabilities, the future of "lake house" technology, exciting updates from the Open Source Program Office (OSPO), and the importance of Kafka's readiness to support evolving data formats—making it a backbone for modern data ecosystems.

Key Takeaways:

1. Community collaboration and contribution are essential for the continuous improvement and testing of Apache Kafka's capabilities
2. The evolution of Apache Kafka into a more versatile platform, combined with object storage and open table formats, can significantly enhance real-time data streaming, analytics, and the future of "lake house" technology
3. Tiered storage in Kafka facilitates more efficient and cost-effective data management by decoupling storage from computing

Resources:

• Watch the full interview on our YouTube: https://www.youtube.com/@Aiven_io
• Check out our website for more information: https://aiven.io/
• Check out Aiven AI Database Optimizer
• Want to be on our mailing list? Sign up here: https://aiven.io/resources
• Follow us on LinkedIn: https://www.linkedin.com/company/aiven/
• Sign up for our newsletter for more insights on this topic: https://aiven.io/newsletter
• Connect with Filip Yonov on LinkedIn: https://www.linkedin.com/in/filipyonov/
• Connect with Josep Prat on LinkedIn: https://www.linkedin.com/in/jlprat/

Timestamps:

[05:49] Kafka servers have theoretical storage limits

[09:29] Test storage proposal process for Apache Kafka

[17:38] LinkedIn conducted an experiment merging Xcode versions

[22:11] Data lake evolving into lake house architectures

[25:00] Broker pushes data to remote storage, plugin handles retrieval and format translation

[26:40] Kafka excels at high-speed, high-volume data

[32:18] Kafka data consumption evolving with new options

[40:19] Managing metadata for conversion on community level

[47:45] Kafka's potential as a widely used API