One day, in the spring of 2017, in a home in South Los Angeles, a robot named Kiwi stood on a table facing a child. Kiwi was a 2-foot tall green bird with felt feathers and a soft tuft of green hair. Its eyes lit up as soon as it spoke, its voice was childlike.

Across the table, a young boy played a game with the robot, travelling through space to solve math problems. A child-size tablet connected to the robot displayed the lessons. The robot tracked the boy’s gaze, giving positive feedback when he got an answer right. “Good job!” it said.

Despite the boy’s visible enjoyment, this game’s not just for fun. Along with math skills, the robot taught the boy a skill called joint attention—shared focus on an object, person, or event— that children with autism learn as part of their behavior therapy.

The child was a participant in an ambitious study, the largest of its kind, led by Professors Maja Matarić and Gisele Ragusa, which placed socially assistive robots in the homes of 17 children with autism for at least 30 days.

Each day, the children sat with their parents or other family members and played space-themed math games while the robots interacted with them.

 

The robots personalized their instruction and feedback to each child’s unique learning patterns during the interventions in real-time. This enabled the children to practice their math and social skills at home whenever they wanted, between school lessons and therapy sessions.

While the researchers anticipated some improvement in math skills, the results surpassed their expectations. At the end of the month’s intervention, all the participants demonstrated improved math skills, while 92% also improved their social skills.

In research videos, children who might previously have hidden away from interaction can be seen walking up and talking to the robot. Some children spontaneously greeted the robot: “Hey, Kiwi,” they would say when they came home after school.

“It can be hard for some kids to access social skill interventions in the home in a way that they might be comfortable,” said the parent of a child who took part in the study.

“I definitely see potential for this kind of intervention in the future, especially as an introduction for sensitive subjects like bullying and self-care related health topics.”

 

A tour de force

The study and subsequent analyses, published in conferences and the
journals Frontiers and Science Robotics, mark the culmination of eight years’ work by Matarić, Ragusa and their students in USC Viterbi’s Interaction lab, along with collaborators from Yale University and MIT.

“It took a large team and a lot of effort to create a system we could leave in homes for a month—it was a real tour de force,” said Matarić, the Chan Soon-Shiong Chair and Distinguished Professor of Computer Science, Neuroscience and Pediatrics.

A pioneer of socially assistive robotics, Matarić developed the field about 15 years ago, when she was inspired to create personalized robots that could provide support, motivation and companionship to people, focusing on those with special needs.

Since then, Matarić and her team have developed machines to help stroke patients enjoy their rehabilitation exercises and people preparing for IV injections feel less pain. Matarić imagines a future where personalized robots help children with autism practice navigating social cues and interactions with people. Realizing that future means getting research out of the lab and into people’s homes.

“There is a lot of talk about the power of machine learning, but we don’t see many companies or research labs making the effort to go into homes, collect data and see what it takes to make machine learning work with messy data from for real families,” Matarić said.

The roots of this particular study go back to 2012, when the National Science Foundation (NSF) awarded $10 million to a team that included Matarić as co-principal investigator with researchers from Yale, Stanford, and MIT.

“Our vision was to develop socially assistive robots that would help children,” Matarić said. “One of our program officers encouraged us by saying, ‘autism is the polio of our time.’ That was really a driving force for us to develop computing methods for socially assistive robots for children with autism.”

 

Early intervention

Autism, or autism spectrum disorder, is characterized by difficulties in early social interaction and communication, as well as attention. It effects one in every 59 children in the U.S.—an increase of 250 percent in the last 15 years, according to the Centers for Disease Control.

While there is no known single cause or cure for autism spectrum disorder, research suggest intensive therapy at an early stage of a child’s development can help. In-person therapy is crucial, but it may not always be available. A therapist cannot be in your house 24/7, but a socially assistive robot can.

Robots can be sent home with a program of games for the child to practice, which is helpful for those who live too far away from therapy clinics or whose caregivers’ work schedules do not allow for frequent visits. Feeling motivated and at ease with the robot, children with autism may be able to better focus on learning and practicing their skills.

At least, that was the researcher team’s goal. The big challenge was actually making it happen.

For this, Matarić enlisted the help of Ragusa, a professor of engineering education, and herself a beneficiary of early intervention. Ragusa was one of the first children in the U.S. to receive early intervention at the age of six months for cerebral palsy.

Ragusa recruited the participating families through regional centers within California’s Department of Developmental Services and local school districts. In total, the researchers selected 17 families from diverse backgrounds across L.A. County. The children were aged between 3 and 7 and formally diagnosed with autism spectrum disorder.

 

A personalized socially assistive robot

Previous research indicates that socially assistive robots can help children with autism develop and retain new skills, at least in the short term.

“When you are interacting with someone, there are social cues such as facial expressions, tone of voice and gestures, which can be overwhelming and distracting for children with autism,” Ragusa said.

“But robots are predictable and, to some extent controllable, which can make children with autism feel more comfortable while they’re learning.”

This type of therapy, however, works best if the child is excited or paying attention.

One way to keep a child a child interested is by personalizing the interaction. As such, Kiwi the robot’s feedback and the games’ difficulty were personalized in real-time to each child’s unique learning patterns. The researchers accomplished this using reinforcement learning, a rapidly growing subfield of artificial intelligence (AI).

The algorithms monitored the child’s performance on the math games. For instance, answer correctly and Kiwi would say something like, “Good job!” Get a question wrong, and Kiwi might provide some helpful tips to solve the problem, and adjust the difficulty and feedback in future games. The goal was to maximize difficulty, while not pushing the learner to make too many mistakes.

 

A constellation of symptoms

Sound straightforward enough? Think again. There’s a saying popular among people with autism and their families: “if you have met one person with autism, you have met one person with autism.”

“Autism is the ultimate frontier for robotic personalization, because, as anyone who knows about autism will tell you, every individual has a constellation of symptoms and severities of each symptom,” Matarić said.

“There is no one-size-fits all. We really have to figure out how to engage each child for the time being with a particular task and be ready to adapt as the child’s behavior, mood, skills, and preferences change, sometimes gradually and sometimes quite suddenly.”

This creates a particular challenge for machine learning, an area of artificial intelligence, which typically relies on spotting consistent patterns in huge amounts of data.

For instance, the machine learning algorithm might spot a pattern in the data and make an assumption: people who look away from a task or individual for prolonged periods are feeling distracted or disengaged.

Not so for children with autism, who frequently show atypical responses. For instance, a common characteristic of autism is difficulty making or maintaining eye contact, which doesn’t necessarily signal a lack of engagement.

“Standard artificial intelligence approaches typically fail when it comes to autism data,” Matarić said. “Machine learning methods require a lot of reasonably consistent data and that just isn’t possible with autism, where heterogeneity and inconsistency reign.”

As a result, the research team had to develop methods that “turned machine learning on its head,” Matarić added.

“Instead of gathering large amounts of data from as many people as possible, which is simply not feasible in autism due to privacy issues, and more importantly, is not necessarily useful, given the variance in the data, we focused on excavating information from the data we had, focusing on balancing the data from each child versus the data from all others,” she added.

 

Part of the family

When the experiment wrapped up, the researchers got to work assessing the success of their system and determined the robot could have autonomously detected the child’s engagement with 90% accuracy.

The researchers also tested the children on their new skills. That testing happened three times: before the experiment, on the last day of the intervention, and again 30 days after the end of the experiment ended. Not only did all the children show improvement in math and social skills, they retained these improvements one month after the study ended.

There was another important result: The robot became “part of the family.” Research videos show multiple people in the room with the child and the robot, including parents, siblings, therapists and pets.

“We found the interaction between the children and other people actually improved,” said Ragusa, who personally conducted the follow-up assessment with the children.

“They developed a social relationship with the robot and that snowballed into a social relationship with the family,” she added. “We encourage the parents to be in the background and step in when needed, so it became a triad of learning together.”

The research team stressed that the socially assistive robot is not intended to replace therapists—instead, the technology diversifies available therapies and the support available to parents—in practice, it could provide them with an opportunity for on-demand, in-home daily interventions.

 

Societally relevant challenges

While Kiwi takes a well-earned break, the research team is continuing to analyze the collected data.

One active follow-up project involves analyzing and modeling the children’s “cognitive-affective states,” including emotions such as confusion or excitement, to design socially assistive robot tutors that are even more sensitive to the emotions and moods of their users.

Another ongoing NSF-supported research project led by Mataric and Ragusa involves a telepresence robot for older children attending school remotely. The goal: to allow students who miss school due to health challenges to be physically embodied and socially embedded in the classroom with their peers.

“In our lab, we focus very specifically on compelling, societally relevant problems,” Matarić said. “We personally meet the real people behind the numbers. It’s challenging to work in the real world with vulnerable users with a broad spectrum of needs, but that is where the important challenges are—those that are really worth committing to.”