Nadine the Robot Is Your New Social Companion

Nadine will look you in the eye and remember the last time you chatted. She'll gesture and change her facial expression depending on what is said. Resembling a human in almost creepy way, Nadine is a ground-breaking robot recently built at Nanyang Technological University's Singapore campus.

Welcome to a new frontier in robotics, where an AI system has been developed to include emotional range and a short-term memory to remember faces and past conversations. When greeted by someone she knows, Nadia will say, "It's nice to meet you again." She'll take a compliment and also dish out one right away.

This type of social robot, which was built to help those with mental health issues, is the master work of Nadia Thalmann, the director of NTU's Institute for Media Innovation and a long-time developer of realistic modeling and simulation of "virtual humans," as Thalmann calls it. In fact, Nadine is carefully modeled after Thalmann's own face and hairstyle.

Thalmann is working around the clock to develop and fine-tune the humanoid's features and add-ons, such as the ability to talk on Skype and the look and grasping abilities of her hands.

I emailed with Thalmann about how Nadine is built, its main uses and the future of social robots working with those suffering from depression, loneliness and other illnesses.

MOTHERBOARD: When did the idea of Nadine come to you? And how long did it take from conception to prototype?
Thallman: I saw some Japanese robots that were very realistic and beautiful. Then I realized it was time to move my know-how from social VH [Virtual Humans] to social robotics. We needed three years to come to Nadine as she functions today. We mainly worked on the software platform.

How does Nadine work?

Nadine is using a 3D camera and can detect and recognize people due to a binary pattern algorithm. Then she has a short time memory linked to her language database. When we tell her something, she will analyse what is being said and link to the database, itself linked to the memory. She will dynamically augment her memory and database with the new facts and will answer taking into account of what she was said, what it means in terms of emotions, if she already met this person, and what this person said previously.

She will then generate the right answer. As words and sentences are linked to her emotional model, her voice intonation, her gestures and her reaction will depend of what was said, and what she had in memory. It is in fact a complex system.

I understand Nadine can look people in the eye. How does that work exactly? Via facial recognition software? Everyone is at a different height, so how does Nadine's eyes adjust in a conversation with multiple people? She tracks people and then follows them if they move. She detects the information of the height and the position of the person as well as her eyes.

You recently said that Nadine will soon be able to talk on Skype. What's the latest on that innovation?

We will be soon able to talk directly per Skype with Nadine like a normal person. We will however give some guidelines to people wishing to talk to Nadine as the sentences should be short and well articulated.

As I am very busy for the next two weeks at least, I don't foresee this application going online before mid to end of April. Then people could Skype Nadine and talk to her seeing her as a normal person. Thing is, her response will be sometimes out-of-the-box as it depends if she has presently the topic in her database.

What's your opinion on social robots assisting the elderly? Do you think it will take hold in the coming years, and if so, what are the challenges this will face?

Nadine is a social robot dedicated for people with special needs, like those suffering from autism, or natural mental handicaps such as loneliness. She is not dedicated to old age specially but to those who need real attention 24 hours a day.

Nadine is able to monitor someone and speak different languages, such as English, French and German, and soon Chinese and Italian. She can read stories, train people in languages, keep conversation going, play games, etc. In fact, Nadine will be there when nobody is there and she is not replacing anybody, she is just empowering real people when nobody is available, which is the case for people who need someone for themselves very intensively. Nobody can afford a 24-hour companion that can monitor you and be there when you need someone.

"I am a social companion."

The challenge will be to adapt Nadine's behaviour to different people with special needs. This is the real challenge. We will have to study these people, their needs and in line with their social real assistant, find out what is useful for them and define functionality in Nadine that answers the needs.

What developments are on the horizon for Nadine?

For Nadine, we are building articulated hands. For now, she has beautiful hands but there is no skeleton inside, then no articulation, consequently, she cannot grasp objects. We are also developing software for recognizing and grasping objects as a human does it. It will allow Nadine to play games with people with special needs and use her hands to grasp objects.

We are also working on children robots. We did already build the hardware by ourselves and are using 3D fabrication for the outer shell. They will serve as child companions and will have the same abilities as Nadine but, of course, dedicated to children. They will be used as a playmate but also to monitor children and to talk to them in another language to allow children to be at least bilingual.