Collaboration & evaluation for LLM apps

Practical AI: Machine Learning, Data Science, LLM

Small changes in prompts can create large changes in the output behavior of generative AI models. Add to that the confusion around proper evaluation of LLM applications, and you have a recipe for confusion and frustration. Raza and the Humanloop team have been diving into these problems, and, in this episode, Raza helps us understand how non-technical prompt engineers can productively collaborate with technical software engineers while building AI-driven apps.

Join the discussion

Changelog++ members save 4 minutes on this episode because they made the ads disappear. Join today!

Sponsors:

• Read Write Own – Read, Write, Own: Building the Next Era of the Internet—a new book from entrepreneur and investor Chris Dixon—explores one possible solution to the internet’s authenticity problem: Blockchains. From AI that tracks its source material to generative programs that compensate—rather than cannibalize—creators. It’s a call to action for a more open, transparent, and democratic internet. One that opens the black box of AI, tracks the origins we see online, and much more. Order your copy of Read, Write, Own today at readwriteown.com
• Changelog News – A podcast+newsletter combo that’s brief, entertaining & always on-point. Subscribe today.
• Fly.io – The home of Changelog.com — Deploy your apps and databases close to your users. In minutes you can run your Ruby, Go, Node, Deno, Python, or Elixir app (and databases!) all over the world. No ops required. Learn more at fly.io/changelog and check out the speedrun in their docs.

Featuring:

• Raza Habib – Twitter, LinkedIn
• Daniel Whitenack – Twitter, GitHub, Website

Show Notes:

• Humanloop

Something missing or broken? PRs welcome!

Chương

1. Welcome to Practical AI (00:00:07)

2. Origin of Human Loop (00:00:43)

3. Types of designers (00:05:23)

4. Tech & non-tech worklfow (00:08:06)

5. What am i building? (00:11:47)

6. Sponsor: Read Write Own (00:13:19)

https://readwriteown.com/?utm_source=changelog&utm_medium=practicalai&utm_campaign=changelog

7. What can Human Loop do? (00:14:40)

8. In-production feedback (00:17:22)

9. Fine-tiuning jargon (00:18:32)

10. Fine-tuning trends (00:21:43)

11. Proliferation of open models (00:24:11)

12. Sponsor: Changelog News (00:26:49)

https://changelog.com/news

13. Different roles in the HL system / Collaborating in Human Loop (00:28:20)

14. Production framework (00:32:04)

15. Importance of evaluation (00:34:40)

16. Surprising usecases (00:38:14)

17. Exciting things that are happening (00:42:43)

18. Outro (00:45:27)