This roboservant will learn how to make you coffee

In the future, we will have roboservants to do our bidding. At least that’s the promise. Right now, roombas are as close as we’ve come to having machines do our chores, and they’re a far cry from Rosie from the Jetsons.

The problem is that artificial intelligence doesn’t actually make robots very smart. They are easily confused by new things. Scientists at Cornell University are trying to change that and to make robots more versatile by teaching them how to uses classes of objects instead of individual object types. For instance, when you see a knob, you know to turn it clockwise, whether it’s attached to a door, sink or toilet. That’s because humans can make inferences about how to manipulate never-before-seen items based on their past experience. Robots aren’t very good at doing that yet, mostly because scientists have opted to hard-code robots’ abilities — programming them to do this thing to this object. That renders robots that encounter new (but similar) objects useless. So, the researchers reasoned that to make robots more functional in the real world they’d have to make them generalists.

Over the last year, the Cornell scientists have been writing software that allows robots to use different types of appliances, like juice makers, espresso machines, and sinks based on common parts. The team, led by roboticist Ashutosh Saxena and grad student Jaeyong Sung, is using “deep learning,” a breed of AI that enables better pattern recognition. The star algorithms of deep learning are so-called neural nets, software made up of layers of artificial “neurons” that can gather and analyze tons of data. For example, if researchers wanted robots to “see” a handle, they’d engineer a low-level layer to detect simple things like the edge of a particular shape. The next layer up would piece those edges together into shapes and then objects. By doing that, it helps machines build up an understanding of how things look.

All this should help robots make inferences, so the same robot could theoretically make espresso, pour beer, or use a drinking fountain, assuming the parts it needs to handle are similar. With this approach, they were able to get robots to correctly handle objects they’d never seen before. For instance, one of their robots had learned to pull down on a cereal dispenser to get cereal. When it encountered a coffee dispenser with a similar shape, but different orientation, it was able to transfer its previous experience to this new situation.

Even if it’s never seen an object before, “it should be able to figure it out, using some other knowledge that it has learned,” Sung said.

Their intelligence also relies on the collective brain of the human crowd. Because the training can be costly and time-consuming — especially if there aren’t multiple robots around — the team turned to Amazon Mechanical Turkers, an inexpensive virtual workforce, and volunteers, to gather lots of instructional data on the cheap. People can go to the Robobarista webpage and help robots get better at using a toaster or turning on a microwave, for example. That information is then fed into a database that robots anywhere can query. The hope is that by making more general models and crowdsourcing some of the training, our robot pals will be more adept at helping us around the house one day.

The project is still in the experimental phase. The robot’s espresso-making feat, which you can watch above, took about 8 hours, Sung told me — far too long for anyone who’s caffeine deprived. Plus, its dexterity leaves something to be desired. The thing is still kind of clunky. But the team’s work is a step toward building robots with better skills. And so maybe one day, Folger’s jingle will say that the best part of waking up is a robot making your cup.

Daniela Hernandez is a senior writer at Fusion. She likes science, robots, pugs, and coffee.