HBM146: Theodora

Here Be Monsters

How does a computer learn to speak with emotion and conviction?

Language is hard to express as a set of firm rules. Every language rule seems to have exceptions and the exceptions have exceptions etcetera. Typical, “if this then that” approaches to language just don’t work. There’s too much nuance.

But each generation of algorithms gets closer and closer. Markov chains were invented in the 1800’s and rely on nothing more than basic probabilities. It’s a simple idea, just look at an input (like a book), and learn the order in which words tend to appear. With this knowledge, it’s possible to generate new text in the same style of the input, just by looking up the probability of words that are likely to follow each other. It’s simple and sometimes half decent, but not effective for longer outputs as this approach tends to lack object permanence and generate run-on sentences. Markov models are used today in predictive text phone keyboards, but can also be used to predict weather, stock prices, etc.

There’ve been plenty of other approaches to language generation (and plenty of mishaps as well). A notable example is CleverBot, which chats with humans and heavily references its previous conversations to generate its results. Cleverbot’s chatting can sometimes be eerily human, perfectly regurgitating slang, internet abbreviations, obscure jokes. But it’s kind of a sly trick at the end of the day, and, as with Markov chains, Cleverbot’s AI still doesn’t always grasp grammar and object permanence.

In the last decade or two, there’s been an explosion in the abilities of a different kind of AI, the Artificial Neural Network. These “neural nets” are modelled off the way that brains work, running stimuli through their “neurons” and reinforcing paths that yield the best results.

The outputs are chaotic until they are properly “trained.” But as the training reaches its optimal point, a model emerges that can efficiently process incoming data and spit out output that incorporates the same kinds of nuance, strangeness, and imperfection that you expect to see in the natural world. Like Markov chains, neural nets have a lot of applications outside language too.

But these neural networks are complicated, like a brain. So complicated, in fact, that few try to dissect these trained models to see how they’re actually working. And tracing it backwards is difficult, but not impossible.

If we temporarily ignore the real risk that sophisticated AI language models pose for societies attempting to separate truth from fiction these neural net models allow for some interesting possibilities, namely extracting the language style of a large body of text and using that extracted style to generate new text that’s written in the voice of the original text.

In this episode, Jeff creates an AI and names it “Theodora.” She’s trained to speak like a presenter giving a Ted Talk. The result varies from believable to utter absurdity and causes Jeff to reflect on the continued inability of individuals, AI, and large nonprofits to distinguish between good ideas and absolute madness.

On the creation of Theodora:

Jeff used a variety of free tools to generate Theodora in the episode. OpenAI’s Generative Pre-trained Transformer 2 (GPT-2) was turned into the Python library GPT2 Simple by Max Woolf, who also created a tutorial demonstrating how to train the model for free using Google Colab. Jeff used this tutorial to train Theodora on a corpus of about 900 Ted Talk transcripts for 5,000 training steps. Jeff then downloaded the model locally and used JupyterLab (Python) to generate new text. That text was then sent to Google Cloud’s Text-To-Speech (TTS) service where it was converted to the voice heard on the episode.

Producer: Jeff Emtman
Music: Liance
Sponsor: Liance

Independent musician James Li has just released This Painting Doesn’t Dry, an album about the relationship between personal experiences and the story of humanity as a whole. James made this album while he anxiously watched his homeland of Hong Kong fall into political crisis.

Buy on Bandcamp. Listen on Spotify.