Oh, patents! Humanoid voices (2)

Copyright © Françoise Herrmann

Lots more patented inventions for humanoids in the quest to emulate participation in human conversations! 

The Softbank Robotics family of robots are all connected robots. This makes them all technically unfinished products, considering that their functionalities might be indefinitely expanded, customized and modified with downloadable apps and extensions, designed by a community of third-party developers.   

Access to the world wide web, and to a remote database of information, greatly improves the humanoid’s capacity to parse and respond to human input in a dialogue, as it provides a way of personalizing interactions and breaking out of canned and repetitive modes of response. Indeed, even when the prior art of humanoid output is programmed to prevent repetitions, locally stored responses (even shuffled) are substantially the same for all human interlocutors. For an extreme example of stereotypical interaction, everyone has experienced interactive computer voice response (IVR) systems on the telephone, designed to lead human callers indiscriminately through a script and its options. For a humanoid robot seeking to function as a companion, this sort of indiscriminate, pre-recorded and stereotypical routine is unacceptable. The companion robot requires a bit more conversational capacity, if it is to eventually function as an assistant to daily living, and especially if it is to create engaging communication, or any kind of bond, albeit an inevitably deceptive one.

The invention recited in  US2016283465, titled Method for dialogue between a machine, such as a humanoid robot, and a human interlocutor; computer program product; and humanoid robot for implementing such a method, precisely addresses the issue of conversation ability, and in particular two aspects, termed: personalization, and progression, both within and across sessions of communication. 

The patent discloses that the humanoid robot has access to a remote database containing user profiles which may be extracted and updated with new information provided in the dialogue with a human interlocutor. Similarly, the humanoid robot is programmed to use profile information in the responses it generates, or to solicit more information to fill-in missing profile information. So, for example, the humanoid may identify an interlocutor by name, based on input data from camera sensors (not only verbal input), and then recall the previous conversation, based on stored history information in the profile, to generate a more personalized response, as in the following sequence, cited in the patent: 

[0062] Robot: Hello Jean! Thanks for coming back to see me. That makes me happy! 
[0063] How have you been since yesterday?

The profile might also be linked to additional contextual information that the humanoid can use in response to human input. The example cited in the patent concerns Europe Day, celebrated on  May 9, and both the user profile information concerning "Mother tongue"  and the missing information in the profile on "Other languages spoken". The cited dialogue prompt, generated by the robot to feed the conversation, and its human response, run as follows:

[0069] Robot: Tell me, do you know that today is Europe day? I think it's great, these countries uniting, with all the different languages and cultures. Also, I speak a number of European languages! And you, Jean, do you speak any languages other than French? [...]

[0072] Human: Yes, I speak English fluently.

The abstract for US2016283465 is included below. The patent figure drawing No. 1 is also included. The drawing shows a human interlocutor, and a humanoid robot with access, via a communication network, to a remote server containing user profiles, for use (extraction and update) in dialogue with humans.  

A method for performing a dialog between a machine, preferably a humanoid robot, and at least one human speaker, comprises the following steps, implemented by a computer: a) identifying the human speaker; b) extracting from a database a speaker profile comprising a plurality of dialog variables, at least one value being assigned to at least one of the dialog variables; c) receiving and analyzing at least one sentence originating from the speaker; and d) formulating and emitting at least one response sentence as a function at least of the sentence received and interpreted in step c) and of one dialog variable of the speaker profile. [Abstract 

US2016283465] 

This patent is part of a family, the list of which is appended below. 

• US2016283465 (A1) ― 2016-09-29 - Method for dialogue between a machine, such as a humanoid robot, and a human interlocutor; computer program product; and humanoid robot for implementing such a method
• AU2014331209 (A1) ― 2016-05-19 - Method for dialogue between a machine, such as a humanoid robot, and a human interlocutor; computer program product; and humanoid robot for implementing such a method 
• CA2925930 (A1) ― 2015-04-09 - Method for dialogue between a machine, such as a humanoid robot, and a human interlocutor; computer program product; and humanoid robot for implementing such a method  
• EP3053162 (A1) ― 2016-08-10 - Method for dialogue between a machine, such as a humanoid robot, and a human interlocutor; computer program product; and humanoid robot for implementing such a method
• FR3011375 (A1) ― 2015-04-03 - Procédé de dialogue entre une machine, telle qu'un robot humanoïde, et un interlocuteur humain, produit programme d'ordinateur et robot humanoïde pour la mise en œuvre d'un tel procédé 
• JP2016536630 (A) ― 2016-11-24 - 人型ロボット等の機械と人間話者との間の対話方法、コンピュータプログラム製品、および同方法を実行する人型ロボット
• MX2016004208 (A) ― 2017-08-16 - Method for dialogue between a machine, such as a humanoid robot, and a human interlocutor; computer program product; and humanoid robot for implementing such a method. 
• WO2015049198 (A1) ― 2015-04-09 - Method for dialogue between a machine, such as a humanoid robot, and a human interlocutor; computer program product; and humanoid robot for implementing such a method

So, does it all work? Can Softbank Robotics humanoids hold up their end of the conversation? Are Softbank Robotics humanoids likable? Lovable? Engaging? Can these humanoids be potential companions? The following video shows what happened with Nao, at three schools in the UK.

References

Softbank Robotics

https://www.softbankrobotics.com/us/ 

Softbank Robotics - Nao

https://www.softbankrobotics.com/us/nao

Oh, patents! Nao's joints

Copyright © Françoise Herrmann

Still a few more state of the art robotics patents! 

Nao’s joints arise as a safety issue in the larger focus on how to prevent unanticipated movement of a humanoid robot, and in particular, injury to a human who might get pinched in the robot’s movements or joints. The anti-pinching system disclosed in WO2015185671, titled Anti-jamming system in a humanoid-type robot, was thus designed to address problems of the prior art focused on solving issues of safety when using humanoid robots, and in particular the risks of pinching.

Indeed, the state of the art had previously explored reducing, or leaving a gap in the movement of a robot’s joints to prevent pinching, which by the same token reduced the sorts of movements that could be performed, and thus limited the anthropomorphism of the humanoid. The state of the art also attempted to limit the force deployed to actuate a joint, but this solution tended to reduce the overall capacity of the humanoid robot, for example, to lift heavy equipment.

In response, this anti-jamming invention thus discloses a passive solution that limits the pinching force of a humanoid joint. It consists in a strategically positioned, soft and flexible membrane, at any area of a humanoid joint (e.g.; hip, elbow, shoulder, knee, wrist) where a risk of pinching might be anticipated. This membrane is able to distort a certain distance (i.e.; the diameter of the round black circle (20) representing a finger in the two included patent drawings), in cases where such a finger might get caught in the humanoid joint, or area between the robot’s trunk and joint.  Thus, while the outer shell of the robot’s limbs and trunk are made of hard plastic, certain researched areas of the robot’s outer shell comprise the contiguous invention areas, made of soft and flexible silicone, or rubber, able to accommodate a finger, that could get caught, and thereby passively limiting the pinching force.

This anti-jamming invention is disclosed in a family of patents. The hyperlinked list of patents in this family appears below. The English abstract of the WIPO member of this patent family, WO2015185671, titled Anti-jamming system in a humanoid-type robot, is also included with two patent figure drawings. The two drawings respectively show a black circle (20) representing a finger potentially pinched in Nao’s elbow joint (below), and a black circle (20) representing a finger potentially caught in the space between Nao’s shoulder joint and trunk (above), with both drawings showing the anti-jamming flexible area designed to accommodates the finger, or to limit the pinching force.

The invention relates to the safety of use of a humanoid-type robot. The humanoid-type robot comprises two elements (2, 3) and an articulation (11) with at least one degree of freedom that connects the two elements, the two elements each comprising a skin (22, 23) delimiting the outer surface thereof, the articulation (11) allowing movement in a given range, a first of the two elements (2, 3) being intended to come substantially into contact with a region (25, 26) of the skin (22, 23) of a second of the two elements at one end of the range. According to the invention, the region (25, 26) is flexible such that it can be deformed by a given distance with a force less than a given force. The first element (3) is coupled to the second element (2) by passing through the flexible region (26).  [Abstract WO2015185671]

Patent Family 

• WO2015185671 (A1) ― 2015-12-10 - Anti-jamming system in a humanoid-type robot
• AU2015270477 (A1) ― 2016-12-01 - Anti-jamming system in a humanoid-type robot 
• CA2950652 (A1) ― 2015-12-10 - Anti-jamming system in a humanoid-type robot 
• EP3152013 (A1) ― 2017-04-12 - Anti-jamming system in a humanoid-type robot  
• FR3021573 (A1) ― 2015-12-04 - Système anti coincement dans un robot a caractère humanoide  
• JP2017516672 (a) ― 2017-06-22 -ヒューマノイド型ロボットにおける噛み込み防止システム
• KR20170021828 (A) ― 2017-02-28 - Anti-jamming system in a humanoid-type robot
• MX2016015823 (A) ― 2017-06-28 - Anti-jamming system in a humanoid-type robot 
• SG11201609419R (A) ― 2016-12-29 - Anti-jamming system in a humanoid-type robot
• US2017080582 (A1) ― 2017-03-23 - Anti-jamming system in a humanoid-type robot

Oh, patents! Nao's siblings (Pepper)

Copyright Françoise Herrmann

The definition of a humanoid robot according to many of the Softbank Robotics patents is the following:

A robot can be qualified as humanoid from the moment … it has certain human appearance attributes: a head, a trunk, two arms, two hands, etc. A humanoid robot may, however, be more or less sophisticated. Its limbs may have a greater or lesser number of articulations. It may control its own balance statically and dynamically and walk on two limbs, possibly in three dimensions, or simply roll over a base. It may pick up signals from the environment (“hear”, “see”, “touch”, “sense”, etc.) and react according to more or less sophisticated behaviors, and interact with other robots or humans, either by speech or by gesture. [Extracted from US20170197311A1] 

Thus, it probably comes as no surprise that Nao is also part of humanoid family that includes a couple of humanoid siblings. Pepper is Nao’s first humanoid sibling. This Softbank Robotics humanoid robot measures 4 ft in height. Pepper’s trunk is humanoid, with two arms, two hands and a head. The lower half of her body is designed as a skirt comprising wheels to ensure mobility. Pepper is equipped with a 3D camera to detect people and their emotions, as well as her own surroundings. The little robot interacts via a tablet and voice. Her hands, fingers and forearms together display 20 degrees of freedom for gestures, specifically designed to enhance communication.  

Pepper was designed as a business companion, in particular for retail and services. She is the ideal host, fielding reception of clients in hotel lobbies, at airports, and in buildings. She might also be found in stores, where customers might query her about goods or inventory. Pepper is also connected, using apps that can be downloaded to increase, or customize, functionalities. She was also designed with a community of developers in mind, who would provide her with endless new possibilities of use and functionality.

The following video pitches Pepper to developers:

Pepper embodies numerous patented inventions, some of which are found in all the humanoid robots manufactured by Softbank Robotics. For example, see posts referring to the inventions covering Nao's hands and Nao's skull. However, the following two patent families are each respectively specific to Pepper’s mobility, and her ability to dock independently into a recharging base:

Patent Family I (Pepper’s mobility)

• US20170144299A1 - Omnidirectional wheeled humanoid robot based on a linear predictive position and velocity controller
• AU2015248711 (A1) ― 2016-11-03 - Omnidirectional wheeled humanoid robot based on a linear predictive position and velocity controller  
• EP2933069 (A1) ― 2015-10-21 - Omnidirectional wheeled humanoid robot based on a linear predictive position and velocity controller  
• CA2946049 (A1) ― 2015-10-22 - Omnidirectional wheeled humanoid robot based on a linear predictive position and velocity controller 
• HK1216406 (A1) ― 2016-11-11 - Omnidirectional wheeled humanoid robot based on a linear predictive position and velocity controller 
• JP2017513726 (A) ― 2017-06-01 - Omnidirectional wheeled humanoid robot based on a linear predictive position and velocity controller
• KR20170030078 (A) ― 2017-03-16 - Omnidirectional wheeled humanoid robot based on a linear predictive position and velocity controller
• SG11201608204Q (A) ― 2016-10-28 - Omnidirectional wheeled humanoid robot based on a linear predictive position and velocity controller 
• WO2015158885 (A2) ― 2015-10-22 - Omnidirectional wheeled humanoid robot based on a linear predictive position and velocity controller 

Patent family II (Pepper’s ability to dock herself into her recharging base)

• US20170080816A1 - Battery charging base and recharging method implementing such a base
• AU2015270600 (A1) ― 2016-12-01 - Battery charging base and recharging method implementing such a base
• CA2951060 (A1) ― 2015-12-10 - Battery charging base and recharging method implementing such a base 
• EP3152080 (A1) ― 2017-04-12 - Battery charging base and recharging method implementing such a base
• FR3021914 (A1) ― 2015-12-11 - Battery charging base and recharging method implementing such a base 
• JP2017518195 (A) ― 2017-07-06 - Battery charging base and recharging method implementing such a base 
• KR20170026441 (A) ― 2017-03-08 - Battery charging base and recharging method implementing such a base 
• MX2016015828 (A) ― 2017-06-28 - Battery charging base and recharging method implementing such a base
• SG11201609422X (A) ― 2016-12-29 - Battery charging base and recharging method implementing such a base 
• WO2015185525 (A1) ― 2015-12-10 - Battery charging base and recharging method implementing such a base

References

Softbank Robotics – Pepper

https://www.softbankrobotics.com/us/pepper

Wikipedia – Pepper (Robot)

https://en.wikipedia.org/wiki/Pepper_(robot)

Oh, patents! Nao's spine

Copyright © Françoise Herrmann

Beyond Nao’s unique and patented appearance in US design patent USD774148S1, this little robot embodies far more inventions. Beginning with Nao’s spine, or central nervous circuitry, each of the little robot’s humanoid features and functions are disclosed in utility patents, granted in the US, in Europe, as well as globally by WIPO.

For example, US2015122073 (A1) titled Spine for a humanoid robot discloses the invention that invokes Nao’s circumscribed spinal capacity for extension, flexion and rotation. This invention is one of several that gives the little robot, a specific set of features that mimic human function. A set of mimetic features that, in turn, and beyond representation, perhaps also offer a laboratory (or mirror) for better understanding certain aspects of human function.

Indeed, US2015122073 (A1) discloses a flexible spine with just two degrees of freedom that attaches to the pelvis at the lower end, and to the neck of the little robot, at the upper end. For practical design and production purposes, the invention spine only bends on two horizontal axes (i.e; right to left, and forward and back). Torque or vertical axis rotations were designed separately from the invention spine, into the little robot’s neck.  

The humanoid robot's spine thus comprises just two double acting linear actuators attached to the base of the neck and to the base of the pelvis, which together, or separately, control the forward/backward, right/left movements of the spine, comprising otherwise a flexible rod, and ball joints with just two degrees of freedom, to prevent the flexible rod from buckling. Far greater specification is included in the disclosure of this invention that includes 14 claims. 

US2015122073 (A1) is part of a family that includes 5 more patents disclosing Nao’s spine:

• WO2013178772 (A1) ― 2013-12-05 - Spinal column for a humanoid robot 
• JP2015523221 (A) ― 2015-08-13 - Spinal column for a humanoid robot
• FR2991221 (A1) ― 2013-12-06 - Spinal column for a humanoid robot
• EP2855104 (A1) ― 2015-04-08 - Spine for a humanoid robot 
• DK2855104 (T3) ― 2017-10-16 - Spinal column for a humanoid robot 

The abstract for US2015122073 (A1) is included below with one of the patent drawings showing the design of Nao’s spine. 

The invention concerns a spinal column for a humanoid robot, the column (20) comprising a lower base (11) intended to be fixed to a pelvis of the robot (10) and an upper base (13) intended to be fixed to a neck (14) of the robot (10), the spinal column (10) allowing two rotations of the upper base (13) relative to the lower base (11), a first taking place about a sagittal axis (21) and a second taking place about a transverse axis (22). According to the invention, the column (20) comprises a flexible band (25) and linear actuators (26, 27), the band (25) being embedded in a first of the ends (28, 30) thereof at a point (29) in a first of the bases (11, 13) and at least guided at a point (31) in a second of the bases (11, 13), the actuators (26, 27) both being anchored between the two bases (11, 13) at anchoring points (32, 33, 34, 35). For each of the bases (11, 13), the anchoring points (32, 33, 34, 35) of the two actuators (26, 27) and the embedding or guide point (29, 31) of the band (25) are spaced apart.

A video is also included below to show Nao’s range of movements in dance, and for you to witness the mimesis.

Video - 5 Nao robots programmed by Valentin Bertrand to dance to Via Continum by Ez3kiel. 

References

Ez3kiel

https://www.ez3kiel.com/ 

Oh, Patents! Nao

Copyright © Françoise Herrmann

Developed by Softbank Robotiks, in partnership with the French Center for Integrated Robotics - Criif (Centre de Robotique Intégré de l’Ile-de-France – Bureau d’étude robotique sur mesure et objets connectés), Nao is one of three well known French robots. The Softbank Robotiks troika includes Nao, Pepper and Romeo.

Nao is a short little humanoid companion robot (about 2 feet tall), which means that the little robot has a body designed to resemble a human. In particular, Nao displays  25% humanoid range of motion. Nao is also equipped with speech recognition for 20 languages. Nao is connected to the Internet, via wi-fi or ethernet, which means that it is also customizable on your computer, with downloadable Nao designed apps, using Softbank Robotics' own NAOqi OS platform with Choréographe C that supports SDK Python, C++, Java and Java Script languages for programming and animation.

Use of Nao in the classroom comes with a curriculum, developed by teachers for teachers (e,g,; STEMRobotics). Special education programs have also already been developed for autistic children by Autistes Without Borders in France.

The US design patent USD774148S1 titled Humanoid Robot was filed by Softbank Robotiks on Sept. 1, 2015, and awarded on Dec. 13, 2016 to Robert Hong.

As a reminder a US design patent is different from a US utility patent in that:

“a “utility patent” protects the way an article is used and works (35 U.S.C. 101), while a "design patent" protects the way an article looks (35 U.S.C. 171)” (USPTO) 

One of the patent drawings is included above, and a video is included showing NAO R&D for a new docking station, at Aldebaran, a Softbank Robotics company, 

References

CRIIF - Centre de Robotique Intégré de l’Ile-de-France – Bureau d’étude robotique sur mesure et objets connectés

http://criif.fr/ 

Softbank Robotikks

https://www.ald.softbankrobotics.com

Autistes sans frontières
http://www.autistessansfrontieres.com/

Stem Robotics

https://www.michigan.gov/documents/mde/SES_Robotics_Brochure_Revised_and_Edited_414088_7.pdf