Connect with us

Tech

Teaching artificial intelligence to connect senses like vision and touch

In Canadian author Margaret Atwood’s book “Blind Assassins,” she says that “touch comes before sight, before speech. It’s the first language and the last, and it always tells the truth.”

Avatar

Published

on

Teaching artificial intelligence to connect senses like vision and touch


In Canadian author Margaret Atwood’s book “Blind Assassins,” she says that “touch comes before sight, before speech. It’s the first language and the last, and it always tells the truth.”

While our sense of touch gives us a channel to feel the physical world, our eyes help us immediately understand the full picture of these tactile signals.

Robots that have been programmed to see or feel can’t use these signals quite as interchangeably. To better bridge this sensory gap, researchers from MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL) have come up with a predictive artificial intelligence (AI) that can learn to see by touching, and learn to feel by seeing.

The team’s system can create realistic tactile signals from visual inputs, and predict which object and what part is being touched directly from those tactile inputs. They used a KUKA robot arm with a special tactile sensor called GelSight, designed by another group at MIT.

Using a simple web camera, the team recorded nearly 200 objects, such as tools, household products, fabrics, and more, being touched more than 12,000 times. Breaking those 12,000 video clips down into static frames, the team compiled “VisGel,” a dataset of more than 3 million visual/tactile-paired images.

“By looking at the scene, our model can imagine the feeling of touching a flat surface or a sharp edge”, says Yunzhu Li, CSAIL PhD student and lead author on a new paper about the system.

“By blindly touching around, our model can predict the interaction with the environment purely from tactile feelings. Bringing these two senses together could empower the robot and reduce the data we might need for tasks involving manipulating and grasping objects.”

Recent work to equip robots with more human-like physical senses, such as MIT’s 2016 project using deep learning to visually indicate sounds, or a model that predicts objects’ responses to physical forces, both use large datasets that aren’t available for understanding interactions between vision and touch.

The team’s technique gets around this by using the VisGel dataset, and something called generative adversarial networks (GANs).

GANs use visual or tactile images to generate images in the other modality. They work by using a “generator” and a “discriminator” that compete with each other, where the generator aims to create real-looking images to fool the discriminator. Every time the discriminator “catches” the generator, it has to expose the internal reasoning for the decision, which allows the generator to repeatedly improve itself.

Vision to touch

Humans can infer how an object feels just by seeing it. To better give machines this power, the system first had to locate the position of the touch, and then deduce information about the shape and feel of the region.

The reference images — without any robot-object interaction — helped the system encode details about the objects and the environment. Then, when the robot arm was operating, the model could simply compare the current frame with its reference image, and easily identify the location and scale of the touch.

This might look something like feeding the system an image of a computer mouse, and then “seeing” the area where the model predicts the object should be touched for pickup — which could vastly help machines plan safer and more efficient actions.

Touch to vision

For touch to vision, the aim was for the model to produce a visual image based on tactile data. The model analyzed a tactile image, and then figured out the shape and material of the contact position. It then looked back to the reference image to “hallucinate” the interaction.

For example, if during testing the model was fed tactile data on a shoe, it could produce an image of where that shoe was most likely to be touched.

This type of ability could be helpful for accomplishing tasks in cases where there’s no visual data, like when a light is off, or if a person is blindly reaching into a box or unknown area.

Comments

Myanmar

Digital Revolution and Repression in Myanmar and Thailand

Activists have also proactively published social media content in multiple languages using the hashtags #WhatsHappeningInMyanmar and #WhatsHappeningInThailand to boost coverage of events on the ground.

Avatar

Published

on

By Karen Lee

Loading...

Following the February 1 coup, Myanmar’s netizens became the latest to join the #MilkTeaAlliance, an online collective of pro-democracy youth across Asia.

(more…)

Continue Reading

Ecommerce

How will oil prices shape the Covid-19 recovery in emerging markets?

Oxford Business Group

Published

on

How will oil prices shape the Covid-19 recovery in emerging markets?
– After falling significantly in 2020, oil prices have returned to pre-pandemic levels
– The rise has been driven by OPEC+ production cuts and an improving economic climate
– Higher prices are likely to support a rebound in oil-producing emerging markets
– Further virus outbreaks or increased production would pose challenges to price stability

Loading...

A combination of continued production cuts and an increase in economic activity has prompted oil prices to return to pre-pandemic levels – a factor that will be crucial to the recovery of major oil-producing countries in the Middle East and Africa.

Brent crude prices rose above $60 a barrel in early February, the first time they had exceeded pre-Covid-19 values. They have since continued to rise, going above $66 a barrel on February 24.

The ongoing increase in oil prices, which have soared by 75% since November and around 26% since the beginning of the year, marks a dramatic change from last year.

Following the closure of many national borders and the implementation of travel-related restrictions to stop the spread of the virus, demand for oil slumped globally.

In the wake of the Saudi-Russia price war in early 2020, Brent crude prices fell from around $60 a barrel in February that year to two-decade lows of $20 a barrel in late April, as supply increased and demand plummeted. The value of WTI crude – the main benchmark for oil in the US – fell to record lows of around $40 a barrel last year on the back of a lack of storage space.

While global demand for oil remains low, one factor credited with reversing the trend is the decision to make significant cuts to oil production, which subsequently tightened global supplies.

Read More

Continue Reading

Tech

How the Rural-Urban Divide Plays Out on Digital Platforms

It is one thing for entrepreneurs, whether urban or rural, to create and operate an online store, as some digital platforms have made it relatively easy to manage an e-store – even by using just a smartphone.

Avatar

Published

on

In the West, villages are emptying out due to the lack of economic opportunities. Consider Italy where, in a bid to attract newcomers, a handful of municipalities have turned to selling houses for €1.

Loading...
(more…)

Continue Reading

Most Viewed

Subscribe via Email

Enter your email address to subscribe and receive notifications of new posts by email.

Join 14,072 other subscribers

Latest

Trending