Robots Can Be Programmed to Follow Visual Instructions

By Ralph Birch

The days of assembling your own patio furniture or entertainment center could be over — if you have the right kind of robot, that is.

A California company has designed an operating system that allows robots to follow visual instructions and build things.

Humans — most of them, anyway — have the natural ability to look at diagrams for construction of many different items and infer what’s needed to get from one step to the next without accompanying written instructions. Robots, on the other hand, need to be programmed with the exact instructions on how to move.

Until Dileep George and his colleagues at Vicarious AI in San Francisco developed their new operating system, that is. Their artificial intelligence-based system programs robots to understand the basic ideas that are conveyed in schematic instructions and translate those ideas into actions.

“Even when you try to teach robots by demonstration, they’re just repeating the exact same motions you show them, not the concept underlying them,” said George, an artificial intelligence and neuroscience researcher.

These so-called “common-sense robots” have shown the ability to work on a wider variety of tasks under varying conditions when compared to machines that were given explicitly coded instructions or physical demonstrations of the steps needed for completion.

With the programming from George and his team, the new robotic system was able to learn more than 500 different general concepts by studying before-and-after images. When given a new set of instructions and the accompanying diagrams, the system considers all of the concepts it has learned and chooses the actions that will best help it achieve its goal.

George and his team tested the new operating system on two gripper-arm robots. The arms were tasked with moving objects across a tabletop. They examined the instructions and carried out the tasks, which included separating lemons from limes and arranging cans in rows based on color.

The arms were able to carry out the instructions under a variety of conditions, responding well when researchers switched the types of objects being moved or the color of the table where the items were placed.

For George and his team, the basis of their research was being able to create a hierarchy of abstractions — from the very basic to the most elaborate — for the robots to draw upon. The programming lets the robots use the abstractions to interpret the concepts being presented via the visual instructions and correctly apply the concepts in various situations.

All of this could lead to a future where there are a couple of robots on your deck putting together the family’s new barbecue grill instead of Mom or Dad.