The problem with Apple's new emoji software, and how AI could solve it

This week at the Worldwide Developer Conference, Apple announced a number of fanciful updates to its iMessage platform, including the ability to write hand-scribbled notes, something called ‘invisible ink’, and one update the crowds have been clamoring for: predictive emoji.

As the pantheon of emoji grows, it has become rather cumbersome to hunt down the right one, and so a number of chat platforms have adopted a predictive model that suggests relevant emoji based on what you type. It saves you from scrolling through screens and screens of emoji, but these programs may be inadvertently limiting the way we use the cartoonish characters.

Currently, you can find predictive emoji texting on platforms like LINE and keyboard apps such as SwiftKey; they all work in a somewhat similar fashion, suggesting emoji based on what you’re typing. So if you write “I really want pi,” SwiftKey will offer up the pizza emoji.

This model of emoji prediction learns from what you type, and ostensibly gets smarter with further use. You provide the training data, and so it will never suggest anything you haven’t actually used before; it is built for convenience, not creativity.

Details on the Apple update are still scarce, but it seems as if the Apple software is based on the school of emoji use that likes to replace words with symbols. “When you tap on the emoji button, we’ll highlight all the emoji-fiable words,” said Craig Federighi, Apple’s senior vice-president of software engineering, as quoted in The Guardian. The few videos that are out there on the predictive feature seem to support the idea that the software was designed to simply substitute words with emoji counterparts.

This feels like a step towards the ‘nounification’ of emoji, software that further codifies emoji into only their literal meaning rather than more creative expression. It would be the difference between

“Did you the last night?”

vs.

“Did you see the game last night? ”

For an emergent language that could be a troubling proposition. Only using the eggplant emoji when explicitly typing the word eggplant doesn’t sound like very much fun. “Wanna get some ” is a text you’d only see see in a Dominos ad. Real human beings, meanwhile, have a more nuanced approach.

Dango, a new app released this week, understands that people don’t just want to use emoji literally, calling it a “naïve approach” that “doesn’t reflect how emoji (and language) are actually used.” Instead of tethering each emoji to a specific literal use (i.e., pizza , dog ), the Dango team built a recurrent neural network, an artificial intelligence that maps massive amounts of behavioral data in order to understand on a deeper level when people want to use particular emoji.

This kind of artificial intelligence takes a staggering amount of data to train, and Dango is no exception. The neural network was fed hundreds of millions of tweets and Reddit rants, mapping each use of emoji into a web of meaning.

Once the emoji are mapped, then the AI can predict which emoji you might want to use based on proximity to the words you have typed, or even the emoji that comes before it.

“What Dango does is take text and map it into this semantic space to understand what they’re talking about and how they were feeling,” Will Walmsley, the CEO and co-founder of Dango, told Fusion. “We don’t look [just] at what you’ve written into your text field, but at the message that’s come in from your friend, so if your friend says something positive, your response tends to have a thumbs up.”

If emoji meaning is truly determined by use, then neural networks could be the key to true emoji literacy. However, there are potential limitations with the training set. Walmsley insists that the data culled from Twitter and Reddit matches the tenor and tone of emojis used in a private conversation between friends. He has a point—younger teens do use these public social networks in ostensibly private ways.

But no matter how personal a conversation on a social platform appears, it’s still fundamentally public, which means the data they’re using to predict emoji usage is, to some degree, ineffective. The most effective way to predict which emojis you’ll use in a conversation is to pull data from that very source. As valuable as Twitter and Reddit might be, they’re not going to get you all the way there.

Take sexting, for example. As confident as Dango is in how closely the publicly available emoji data matches the private data, it seems almost impossible that people on Twitter are sending each other romantic messages with the same level of candor as they would on private text message.

Thankfully, there’s good news on that front: since Dango just launched, it should theoretically be pulling personal texting data from the users that have downloaded and used it in the last week. Already, Walmsley says that the AI is parsing meaning from words they themselves don’t understand.

“Dango is smarter than we are,” he told Fusion. “Dango understands more slang than we understand.”

And if Dango’s training set becomes reliable enough to know when to throw an eggplant emoji and a splash emoji into a conversation at the right moment in time, it’ll be the best technological solution yet for shortening the time between thinking of an emoji and having it appear in a text bubble. It will at least be the most ambitious.

Cara Rose DeFabio is a pop addicted, emoji fluent, transmedia artist, focusing on live events as an experience designer for Real Future.