Oh, patents ! Origami robots aka robogamis

Copyright © Françoise Herrmann

According to Jamie Paik, Director of the Reconfigurable Robotics Laboratory (RRL) at the Federal Polytechnic School of Lausanne (EPFL- École Polytechnique Fédérale de Lausanne), in Switzerland, origami robots, also termed robogamis, are part of a paradigm shift in robotic design. Indeed, they are so different and new that they form a framework for robotic design. A framework that extends, for example, to soft robotics, haptics and modular design (EPFL1, EPFL 2, EPFL3). In contrast to traditional, anthropomorphic robots with a single memetic form, robogamis morph. They transform from one form to another, considering that in the terms of mathematics, any 3D shape can be obtained from folding a 2D surface.  

The video below shows the MIT Computer Science and Artificial Intelligence Lab (CSAIL) origami robot, and how this little robot is programmed to morph (CSAIL 1, CSAIL 2). According to Daniela Rus, Director of MIT CSAIL, and of the origami robot project, manufacturing the origami robot is also an innovation, as the MIT origami robots are printed flat (CSAIL 3). A manufacturing process that is fast, inexpensive and convenient. Succinctly, the MIT CSAIL origami robot has a body comprising three layers. The middle layer is heat-reactive, causing the material to shrink (and bend) under the effect of heat. A process that is controlled, for angle degree, via gaps cut-out in the two outer structural layers. Thus, once printed, the robot actually self-folds using a self-folding compiler.

In particular, for example, the MIT CSAIL origami robot was further researched and bench-tested as an ingestible device (Hardesty, 2016). In this simulated application, the MIT CSAIL origami robot is first ingested in a medium that dissolves (e.g., ice). The MIT CSAIL origami robot, once released, then unfolds like an accordion inside a simulated stomach medium, where it is guided via a programmable magnetic field to find small ingested objects such as a button battery. The MIT CSAIL origami robot then attaches to the object via a magnet, dislodging the battery from where it is embedded in the simulated lining of the stomach or esophagus.  Thus, the origami robot would fulfill its mission to prevent risks of serious organ ulceration, resulting from ingested button batteries that are stuck. The MIT CSAIL origami robot then might disintegrate, or fracture, under the effect of gastric fluids, so that it can also be expelled through the GI tract. In future versions, the MIT CSAIL origami robot might search and retrieve small ingested objects autonomously using algorithm-driven sensors and cameras, or it might perform different endoscopic interventions, such as delivering medicine or patching wounds, using its own origami structure.

The MIT CSAIL, ingestible, endoscopic, origami robot invention is recited in the US patent application US20200038061A titled Origami robots, systems, and method of treatment. The abstract of the invention is included below, together with the Figure 1 drawing of the patent application. The Figure 1 drawing depicts a magnified view of the origami robot deployed inside the stomach of a patient. The origami robot invention is intended to resolve issues of the prior art of endoscopic devices, as it is a non-invasive procedure, invoking no surgery that relies on a tethered endoscope. Likewise, the origami robot is intended to resolve issues of the prior art of pill-cam endoscopes, devices that are unguided, once ingested. 

Specifically, the Figure 1 drawing depicts a patient 130, and an origami robot 103 that is encapsulated by biocompatible material 101, in the shape of a capsule or pill 100. The biocompatible material 101 is meltable or degradable, once ingested into the patient’s GI tract 132. The origami robot 103 comprises a foldable body portion 102, comprising actuation means for unfolding. The foldable body portion 102 initially appears folded 110, inside the stomach, once the encapsulation has disintegrated/melted. Then, the body portion 102, is also depicted unfolded 120. A magnet 104, designed to retrieve a lodged button battery, is also depicted in this embodiment of the origami robot 103. Finally, an area 104 is also shown. The area 104 corresponds to a wound site that the endoscopic origami robot is designed to treat.  

 Origami robots, and associated systems, methods of treatment, and methods of manufacture are provided. A system includes an origami robot encapsulated for ingestion by a patient, such as in a biocompatible material that is dissolvable or meltable within the gastrointestinal tract. A method of treatment includes delivering an origami robot in a folded position into a gastrointestinal tract of a patient, causing the origami robot to unfold within the gastrointestinal tract, and directing the origami robot to a site requiring treatment in the gastrointestinal tract.

 [Abstract US20200038061A]

 References

EPFL – Reconfigurable Robotics Laboratory - École Polytechnique Fédérale de Lausanne.   https://www.epfl.ch/labs/rrl/

EPFL (1) – RRL - Modular Origami Robots.   École Polytechnique Fédérale de Lausanne.  https://www.epfl.ch/labs/rrl/research-2/research-origami/mori/

EPFL (2) – RRL – Artificial skin could help rehabilitation and enhance virtual reality. École Polytechnique Fédérale de Lausanne. https://actu.epfl.ch/news/artificial-skin-could-help-rehabilitation-and-enha/

EPFL (3) – RRL – Spinoffs- FOLDAWAY – Ultra-portable haptic interface. https://www.epfl.ch/labs/rrl/spin-offs/

Hardesty, L. (May 12, 2016) Ingestible origami robot.  https://news.mit.edu/2016/ingestible-origami-robot-0512

MIT – Computer Science and Artificial Intelligence Laboratory (CSAIL).   https://www.csail.mit.edu/

MIT CSAIL (1) (Aug. 7, 2014) Origami robot folds itself, walks away.   https://www.csail.mit.edu/news/origami-robot-folds-itself-walks-away

MIT CSAIL (2) (June 12, 2015) Origami robot self folds, crawls, swims, self-destructs.   https://www.csail.mit.edu/news/origami-robot-self-folds-crawls-climbs-swims-self-destructs

MIT CSAIL (3)  Daniela Rus (Director of CSAIL)  – Research Project: Self-folding robots.   http://danielarus.csail.mit.edu/index.php/2015/09/lorem-ipsum-2/

Oh, patents! The Skypod (2)

 Copyright © Françoise Herrmann

Autonomous Exotec Skypods dance their 3D way around warehouses, climbing up and down shelves to towering heights, swiveling around as if mounted on ball-bearings. The following video (viewable only on Youtube) offers a Skypod choreography to the tune of Edvard Grieg’s (1875) orchestral piece for Scene 6, Act II of Henrik Ibsen’s (1867) play Peer Gynt (1).

https://www.youtube.com/watch?v=_HoEFcZOiuY

Copyright © Exotec - A three dimensions dance performed by order fulfillment robots : the Skypod

The Skypods also move around with a visible stability, regardless of the presence, absence or position of the load transported in the payload crate. The US family member utility patent US2019263463A1, titled Automatically guided trolley for transporting and/or handling a load, discloses an invention that precisely addresses the load-bearing issue, relative to equilibrium, speed and energy-efficiency, including the corollary issue of swiveling-capacity.

In a nutshell, the invention disclosed replaces the typical 4-wheel drive suspension system of a vehicle. The invention system comprises two frames: one receiving the payload, and the other in contact with the ground, supporting the payload-receiving frame. The supporting frame further comprises two connected walking beams at the front and rear of the Skypod autonomous vehicle (termed automated trolley), connected to two idler wheels and two drive wheels. Translation of the two walking beams is designed to distribute the payload weight, received by the main frame, evenly between all wheels, idle and drive, at the front and rear of the vehicle, as well as left and right. The walking beam system is also much lighter, thus promoting enhanced vehicle performance. both in terms of speed and energy efficiency.

For those skilled in the art, the abstract of this patent, hyperlinked to the complete disclosure of the invention, is included below, together with the patent Figure 1, showing an exemplary embodiment of the invention. In particular, the patent Figure 1 depicts the Skypod vehicle (guided trolley)  10,  comprising a supporting frame 12 and main load-receiving frame 11, idle wheels 18 and driver wheels 16, engine 17, and the two walking beams 13 and 14, connected to the supporting frame 12, with translation arms (132, 142), and (131, 141), contributing to evenly distribute a payload weight to all the wheels. In sum, a new mechanical load-bearing-and-distributing system, enabling the visible stability, and calculated efficiency, of the autonomous Skypod vehicle.

An automated guided trolley for transporting and/or handling a load. The trolley has a main frame for receiving the load and a supporting frame having two walking beams extending respectively towards the front and the rear of the trolley. The walking beams are mounted rotatably with respect to the main frame respectively about a first axis and a second axis and such a walking beam includes elements for securing one arm of one walking beam to one arm of the other walking beam and for supporting the first of these arms on the other arm. [Abstract US2019263463A1]

___________

(1) Grieg, Suite No 1 Op. 46:IV - In the Hall of the Mountain King. 

Reference

Exotec (website): www.Exotec.com 

Oh, patents! The Skypod (1)

Copyright © Françoise Herrmann

Within the context of booming eCommerce, resolving issues of warehouse logistics arises as another pandemic-aggravated priority. A priority with many different robotic solutions. The Exotec Skypod solution is a towering, three-dimensional solution, primarily addressing order processing and storage space, including issues of retrieval and accelerated speed. The video below shows the Skypods in action.

The essential components of the Exotec Skypod system are stackable bins and a fleet of autonomous robots, able to navigate the stacks in three dimensions (up, down and across). The Skypods travel 4 meters per second (approx., 13ft/s) to retrieve and deliver merchandise to the operator stations. The robots' activity is synchronized using a software package called ASTAR. Stackable bins, available in 2 sizes, carry up to 30 kilos (approx., 66 lbs) of merchandise. Bins are stackable up to 10 meters high (approx., 33 feet). Thus, operators are able to work at ergonomically-designed stations. They no longer have to walk an estimated 15 km per day (approx. 10 miles/day) in the stacks, where pedestrian accidents occur. Space is optimized, due to vertical stacking. Adding racks, and stacks, does not interrupt the existing flow of activity. The energy footprint of the system is also very small, due to the robots’ recharging capacity.  In sum, the system is super scalable, which also makes the Exotec Skypod system super nimble, customizable, and modular.

Exotec, a French company launched in 2014, has now opened offices in North America. The various mechanical (e.g., robot climbing system), logistics (eg., sorting system) and navigation aspects of the Skypod invention are patented in a series of patent families, including the following French patent family members (with English patent titles in parenthesis):

• FR3072371 A1 - Système de stockage et de transport d'objets entreposés dans des rayonnages d'un entrepôt. (System for storing and transporting objects stored in the racks of a warehouse.)
• FR3065208A1 -  Chariot à guidage automatique pour le transport et/ou la manutention d'une charge. (Automatically guided trolley for transporting and/or handling a load.)
• FR3057258 A1 - Système de préparation de commandes. (Order picking system.)
• FR3048238A1 - Système de tri d'articles et procédé de tri correspondant. (Article sorting system and corresponding sort method.)
• FR3045895 A1 -  Système de preparation de commandes et procédé de préparation d'au moins une partie d'une commande correspondant. (System for preparing orders and corresponding method for preparing at least one portion of an order.)  

The Exotec Skypod system is 3D-warehousing, brought to you from the northern city of Lille, in France. A solution that integrates with existing Warehouse-Management Systems (WMS). 

 References

Skypod– Exotec solutions 

https://www.exotec.com/  (FR)

https://www.exotec.com/en/solutions (ENG)

Oh, patents! The Robomart™

Copyright © Françoise Herrmann

Back to robots, their expanded role within the context of the pandemic and re-opening of economies. The Robomart™ is a driverless autonomous mini-market on wheels. Already in operation, prior to the pandemic, the Robomart™ offered a mobile-driven mini-market, wherever customers ordered it to drive by. The Robomart™ was successful because it was not just saving customers time, going to the grocery store, it raised the bar on the experience of online shopping, allowing customers to see, and choose the produce they were buying, wherever they were located. Advantages that became amplified during the pandemic, when everyone was confined, and the demand for online shopping skyrocketed. 

Indeed, the Robomart™ became especially attractive, since it offered contactless delivery of produce, via a robotic operator that never got sick. As an autonomous vehicle, the Robomart™ delivery van is equipped with all the standard AI self-driving technology, such as LiDAR, radar and cameras, CAN motion-control system, robotic route-planning and obstacle-avoidance software, enabling various levels of autonomy from a human operator. 

The video below shows The Robomart™ in action. After viewing the video you might legitimately wonder how it is that the Robomart™ knows which merchandise has been selected, “checked-out”, and debited to your credit card. This question has a patented answer.

The Robomart invention recited in US20180349872 is titled One tap/command grocery ordering via self-driving mini-marts and seamless checkout-free technology. The abstract, together with the patent Figure 3, are included below.

The present disclosure generally relates to an application for obtaining groceries or other merchandise and, more particularly, a one tap/command grocery ordering via self-driving mini marts and seamless checkout-free technology. The method is implemented in a computer infrastructure having computer executable code tangibly embodied on a computer readable storage medium having programming instructions and is operable to: deploy an autonomous vehicle which carries consumer merchandise to a consumer; determine which merchandise has been taken from the autonomous vehicle by the consumer; calculate a cost for the merchandise taken from the autonomous vehicle by the consumer; and provide a receipt to the consumer for the merchandise taken from the autonomous vehicle by the consumer. [Abstract US20180349872]

 

US20180349872

The invention recites a system that advantageously combines the convenience of online delivery with the comfort of picking out produce yourself. An invention that also promises cost savings, passed on to customers, resulting from the fully automated system. The patent further discloses that an entire mini store is brought to the consumer vs. just an order of groceries. Thus, the Robomart™ is effectively able to compete with delivery bots and drones, as it offers a much larger selection of produce and groceries. 

According to the disclosure of the invention, the user taps on the Robomart™ app to dispatch the closest Robomart™ van 100, from a store, to their location. On its way, the Robomart™ van 100 will be tracked by the user. When the Robomart™ van 100 arrives, the user opens the compartment doors and selects items. A seamless check out system is invoked as tagged selections are recorded by sensors 105, on the racks, while prices and costs are displayed on a smart glass panel 110 of the van compartment doors, together with the customer’s name. Once the purchase is completed, debited using POS (Point-of-Sale) means from the user’s saved credit card, time-stamped according to the opening and closing of the van compartment doors, and processed using the Stripe platform on the backend, the Robomart™ van 100 moves on to the next customer, or to a replenishment center for refilling the racks, all of which can be tracked variously by users and managers, equipped with the Robomart™ app.

Receipt of purchased items might be itemized in various ways, depending on the embodiments of the invention. For example, items purchased might be listed, or they might include images. The information displayed on the smart glass compartment 110 might also be displayed simultaneously on the Robomart™ app.

The patent additionally recites various levels of self-driving vehicle technology i.e., radar and LIDAR; other cameras or backup Radio-Frequency Identification (RFID) systems for computation of items removed or added to shelves; other sensors (motion, proximity and infrared); modems for connectivity; a speaker for communication with the customer, and a cooling system for refrigeration. The patent further includes a description of the system architecture, as well as a description of real-time mapping, and tracking technology, the restocking depots and vehicle communications technology, and the on-demand summoning technology.

References

Robomart (website) https://robomart.co/

Tarantola, A. (Jan 9, 2018) Robomart autonomous bodegas will deliver produce to your door https://www.engadget.com/2018-01-09-robomart-autonomous-bodega-deliver-produce.html

Oh, patents! Starship robots (3) Low-light navigation

Copyright © Françoise Herrmann

One of the main advantages of using delivery robots, to solve the last mile logistics of transporting goods, is that robots can operate 24/7 without mandatory resting periods, or extra pay. Last mile robotic deliveries thus already appeared as quite an atractive solution, considering the surge of ecommerce and online deliveries, even before the pandemic,  a forciori during the pandemic, and the gradual re-opening of economies.

However, for robots to moonlight (pun intended) at no extra costs, they also have to be able to navigate in low-light conditions. Easier said, than done.  How does a camera sensor capture the image of an objet that is no longer visible? How can terrain be mapped accurately and efficiently at night? The StarshipTechnologies patent WO2019086465A1, titled Visual localization and mapping in low light conditions, precisely adresses this issue.

 The patent discloses a SLAM (Simultaneous Localization and Mapping) method where the robot can estimate its own position on a map, while simultaneously creating the map, enabling it to continue its route autonomously. The method relies on data collected by a wide variety of sensors, such as GPS, Lidar, gyroscope,  cameras,  odometer, accelerometer and magnetometer. At sundown, when the sun is astronomically positioned between 0 and 6 degrees below the horizon, a twilight map is created. The twighlight map has the advantage of having both daytime features (e.g. ; straight lines) and night time features (e.g.,urban lights) features.  Thus, the twilight map, in fact, bridges the visibility gap by mapping the position of nighttime features onto a daytime model. In turn, position relative to visibility is triangulated with data incoming from other sensors, and mapping is adjusted accordingly. Otherwise captured images might also be downsized to bring blurry or jagged lines into sharper focus, during mapping.  Likewise roads and buildings might be tagged relative to daytime and night time features to facilitate localization.

The abstract of the invention is included below, together with the patent Figure 4 showing a twighlight map with night time visual features (e.g., urban lights) 2T, and day time visual features (e.g., lines) T1,  extracted during twighlight time, when the sun was positionned astronomically  between 3 and 8 degrees below the horizon.

The present invention relates to a method comprising generating a map comprising day-time features and night-time features, wherein the position of night-time features relative to the day-time features is determined by at least one image captured during twilight. The present invention also relates to a corresponding processing unit configured to execute such a method. [Abstract WO2019086465A1]

Most of the time, all goes well. The 99% autonomous Starship robots fullfill their missions, delivering goods at extended hours, seven days a week, to happy customers. For example, according to the Youtube video incuded below, Starship Robots fulfilled 2500 deliveries during their first week of operation at the Univesrity of Houston, TX, in 2019. However, on occasion the robots get stuck. The following Youtube video shows how a Starship robot was rescued by a University of Houston student, in the middle of the night. Equipped with voiced interaction routines, the Starship robot even gratefully thanked the student, after being rescued.  

Reference

Starship Technologies

https://www.starship.xyz/company/

Oh, patents! Starship robots (2) The mothership

Copyright © Françoise Herrmann

 The Mercedez-Benz robovan has been called the Starship robot mothership (Vincent, 2016). An adapted Mercedez-Benz Sprinter van, the Mercedez-Benz robovan ferries eight Starship robots for 99% autonomous parcel delivery, directly to clients, on an algorithm-optimized route. Eight Starship robots enter the van from the rear, using a ramp, and exit the van from the side, also using a ramp. The Starship robots deliver goods within a 2-mile radius. The goods, stored in robovan bins above the robots, are manually loaded into the robot’s payload compartment. After delivery of the goods, directly to customers, the Starship robots return back autonomously to the mothership, where they can dock to recharge, and be refilled with a new payload. The robovan mothership, together with its fleet of eight Starship robots, makes an estimated 400 deliveries per 9-hour day, which solves the last-mile logistics of delivering goods, efficiently and cost-effectively (Burgess, 2017). Especially in response to the surge in demand for online and contactless deliveries, during and post, pandemic.

 The YouTube video below shows the Mercedez-Benz robovan, together with the Starship delivery robots.

 The robovan mothership invention is recited in a family of three patents.  

• WO2018024851A1, titled Vehicle (1) 
• GB2573382A, titled Vehicle (2) 
• DE102016009572A1, titled  Fahzeug (3) 

The patents reciting the Starship mothership vehicle invention are surprisingly wider in scope than the Mercedez-Benz robovan embodiment. Looking at the British patent, for example, the definition of the term “vehicle” is extended to:

 “a passenger car, rail vehicle, watercraft (e.g. ship), underwater vehicle or aircraft.” [0006]

 Likewise, the definition of the term “delivery robot” has a much wider definition, understood in particular to mean: 

a self-driving delivery robot, a self-flying delivery robot (drone), a self-controlling floating vehicle, etc..” [0007]

Otherwise, the delivery robots are described as autonomous vehicles, able to charge or refuel autonomously inside the mothership hold. The robots are also described as preferably fully autonomous. Thus, the robots are equipped with navigation and positioning means, as well as a robot-guidance system, a 2D/3D guidance route, and sensors to collect recordings of the environment, for evaluation in regards to existing obstacles. Equipment ultimately designed to enable the delivery robots to locate a customer’s address and to return to the mothership, without the assistance of a human operator.   

The mothership vehicle is equipped with automated mechanical clamping means to secure the robots in place within the vehicle’s hold. Such automated mechanical clamping means are also described as partially inflatable, in order to accommodate different design contours or angles of the individual delivery robots, loaded into the vehicle hold. 

Communication between the vehicle and the robots is Bluetooth® enabled. For example, communication to monitor loading and unloading of the robots, to transmit the recipients’ delivery addresses to the robots and instructions for remitting the payloads to the recipients, to launch automated routines, as well to transmit information to a communication center and/or to the vehicle driver, for monitoring and oversing the condition and activity of both the mothership and the robots.

The mothership vehicle is also equipped with accumulators, able to interface with each individual delivery robot, for the purposes of charging or refueling, particularly during vehicle travel. Advantageously, charging might be designed contactless via induction. In any event, charging occurs without human intervention. 

To optimize the vehicle’s automated handling of the delivery robots, the cargo hold is also equipped with sensors, able to communicate information in regards to the number of robots in the hold and their position. Likewise, the vehicle is also equipped with means to record and perform the loading and unloading of the robots into the vehicle hold, whether loading and unloading invoke ramps, platforms, and/or a docking interface. 

The British patent abstract of the Starship vehicle invention is included below, together with the patent Figure drawings 1a & 1b of the mothership vehicle, loaded with delivery robots. In Figure 1a, the vehicle (1) is further depicted with a non-inflated fixing device (12) for securing the robots (50) in the hold (10). In Figure 1b the fixing device (12) is shown inflated, and actively securing the delivery robots in place, in the cargo hold (10). 

The invention relates to a vehicle (1) for accommodating a number n ≤ N of delivery robots (50) in a cargo compartment (10) of the vehicle (1), where N is the maximum number of delivery robots (50) which can be accommodated in the cargo compartment (10) and n is the number of delivery robots (50) currently in the cargo compartment (10). The vehicle (1) has the following: - a fixing device (12) for the automatic individual fixing of N delivery robots (50) in the cargo compartment (10), - a communication interface (14) for communication between the vehicle (1) and the n delivery robots (50), and - a number N of charging interfaces (16) for the individual automatic charging of energy stores of the n delivery robots (50) in the cargo compartment (10) [Abstract GB2573382A].

____________________

Notes 

(1) A Patent Cooperation Treaty [PCT], United Nations World Intellectual Property Organization [WIPO] patent, filed in German, by StarshipTechnologies.
(2) A British patent, filed in English, by Starship Technologies
(3) A German patent, filed in German, by Daimler AG

References 

Burgess, M. (Sept. 7, 2016) Mercedes vans filled with swarming delivery bots could be heading to your hometown – Wired Mag.  https://www.wired.co.uk/article/mercedes-starship-drones-delivery-van

Daimler.com (Jan. 13, 2017)  Mercedes-Benz invests in Starship Technologies, the world's leading manufacturer of delivery robots.  https://media.daimler.com/marsMediaSite/en/instance/ko/Mercedes-Benz-Vans-invests-in-Starship-Technologies-the-worlds-leading-manufacturer-of-delivery-robots.xhtml?oid=15274799

Starship Technologies - https://www.starship.xyz/

Vincent, J. (Sept. 6, 2016) Mercedes Benz has made a ‘mothership’ van for six-wheeled delivery robots. The Verge. https://www.theverge.com/2016/9/7/12830298/delivery-bot-van-mercedes-starship-technologies

Oh, patents! Humanoid voices (1)

Copyright © Françoise Herrmann

A humanoid robot without a voice would really have missed the point of emulating humans!

So, each of the Softbank Robotics humanoid robots – Nao, Pepper and Romeo, are not only equipped with voices (i.e.; speech synthesizer and speech recognition with a natural language interface), they also have actuators generating coordinated body language that animates interactions to make them more mimetic -- plus much more in terms of personalized and synergetic interactive capacity.

The many Softbank Robotics R&D partnerships in human/machine interaction, at major academic robotics research labs, in France and Europe, are both part of the humanoids' development, and users of the robotics platforms to further their own research agendas.  

One such project, led by Devillers (2017) at L'IMSI (affiliated with France's National Center for Scientific Research), seeks to model not only the verbal components of human/machine interaction, but also the non-verbal or paralinguistic aspects of interaction. The assumption is that the quality of human/machine interactions might increase when more aspects of the interactions are modeled and detected. It is not only what is said that matters, but how it is said, for example with intonation or facial expression. Detecting that an interlocutor might be annoyed, or angry, perplexed or unsure, will greatly enhance the quality of the interactions. 

Ultimately, the goal is for the robot companion to please, so that more satisfying human/machine interactions might arise, perhaps creating conditions for a warm relationship to take root, even if it is going to be a very deceptive one. Thus, the project also includes an ethical component, designed to limit the ways in which such potentially deceptive interactions play out with vulnerable populations, such as children, the elderly and handicapped. 

Finally, the project seeks to define machine humor, also for the purposes of making the machine more likable. If the machine is bound to make mistakes, then the machine’s own error detection might be transformed into humor. Otherwise, humor might also be modeled as a form of response to the detection of certain emotional states. 

The detection and modeling of non-verbal input in human/machine interaction, for the purposes of enhancing human/machine interactions, is a patented Softbank Robotics invention.  US2017148434 titled Method of performing multi-modal dialogue between a humanoid robot and user, computer program product and humanoid robot for implementing said method, discloses the acquisition of input from at least a sound sensor and a motion or image sensor, for the purposes of interpreting such linguistic and paralinguistic aspects of human/machine interaction as utterances, intonation, gestures, facial expressions and body posture. In turn, the interpretation of multi-modal input is designed to invoke a humanoid response that also includes both linguistic and paralinguistic features, such as an utterance, intonation, gestures, facial expression, and body posture!  

Thus, the humanoid response is also animated. The patent figure drawing 5b shows the syntactic analysis of the utterance: I agree with you. for the purpose of determining the insertion point(s) of the mechanical actuation that will animate the robot's response. 

The abstract of this patent is included below:

A method of performing dialogue between a humanoid robot and user comprises: i) acquiring input signals from respective sensors, at least one being a sound sensor and another being a motion or image sensor; ii) interpreting the signals to recognize events generated by the user, including: the utterance of a word or sentence, an intonation of voice, a gesture, a body posture, a facial expression; iii) detennining a response of the humanoid robot, comprising an event such as: the utterance of a word or sentence, an intonation of voice, a gesture, a body posture, a facial expression; iv) generating, an event by the humanoid robot; wherein step iii) comprises detemlining the response from events jointly generated by the user and recognized at step ii), of which at least one is not words uttered by the user. A computer program product and humanoid robot for carrying out the method is provided. [Abstract US2017148434] 

This invention is disclosed in a whole family of patents listed, and hyperlinked, below. 

• US2017148434 (A1) ― 2017-05-25 - Method of performing multi-modal dialogue between a humanoid robot and user, computer program product and humanoid robot for implementing said method 
• AU2015248713 (A1) ― 2016-11-03 - Method of performing multi-modal dialogue between a humanoid robot and user, computer program product and humanoid robot for implementing said method 
• CA2946056 (A1) ― 2015-10-22 - Method of performing multi-modal dialogue between a humanoid robot and user, computer program product and humanoid robot for implementing said method 
•  EP2933067 (A1) ― 2015-10-21 - Method of performing multi-modal dialogue between a humanoid robot and user, computer program product and humanoid robot for implementing said method 
• HK1216405 (A1) ― 2016-11-11 - Method of performing multi-modal dialogue between a humanoid robot and user, computer program product and humanoid robot for implementing said method
• JP2017520782 (A) ― 2017-07-27 - Method of performing multi-modal dialogue between a humanoid robot and user, computer program product and humanoid robot for implementing said method  
• KR20170003580 (A) ― 2017-01-09 - Method of performing multi-modal dialogue between a humanoid robot and user computer program product and humanoid robot for implementing said method
• MX2016013019 (A) ― 2017-05-30 - Method of performing multi-modal dialogue between a humanoid robot and user, computer program product and humanoid robot for implementing said method. 
• SG11201608205U (A) ― 2016-10-28 - Method of performing multi-modal dialogue between a humanoid robot and user, computer program product and humanoid robot for implementing said method 
• WO2015158887 (A2) ― 2015-10-22 - Method of performing multi-modal dialogue between a humanoid robot and user, computer program product and humanoid robot for implementing said method

The following video will give you a glimpse of how well Pepper performs, in an interview, with a human. 

NB. Aldebaran Robotics is the former Softbank Robotics.

References

Devillers, L. (2017) Rire avec les robots pour mieux vivre avec. Le Journal du CNRS 9-02-2017 

https://lejournal.cnrs.fr/billets/rire-avec-les-robots-pour-mieux-vivre-avec

CNRS - National Center for Scientific Research

http://www.cnrs.fr/en/aboutcnrs/overview.htm

LIMSI - Laboratoire  d'Informatique pour la Mécanique et les Sciences de l'Ingénieur

https://www.limsi.fr/fr/

Projet romeo – Partenaires

https://projetromeo.com/partenaires/

Softbank Robotics

https://www.softbankrobotics.com/us/

Oh, patents! Nao’s sibling (Romeo)

Copyright © Françoise Herrmann

Nao’s youngest and largest sibling is Romeo, a humanoid robot designed as a companion for the elderly. Thus, Romeo is being developed with both specific automated physical capacities and artificial intelligence for cognitive assistance and communication with an elderly population. The robot’s cognitive assistance module includes, for example, breakfast, lunch, nap and dinner routines, and reminders to drink fluids. The conversation module, for example, enables patients to query the robot for time of day, date and the latest news, or for certain tasks, such as “turn on the lights”, “show me a movie”, “answer the phone”, “bring me my glasses”, “follow me”, “let’s exercise:”, “hold this”, “empty the dishwasher”, plus much more, since this is a connected robot, whose functionalities are, in principle, indefinitely expandable with new apps, new extensions, and upgrades. 

Indeed, the Romeo platform was actually tested with a consortium of 16 top industrial and academic robotics research partners, such as the CNRS (France’s National Center for Scientific Research) and INRIA (France’s Institute for Research in Computer Science and Automation), for more than 4 years. This mega-collaboration has given rise to many patented inventions, some of which inform the design of  Nao second-generation robots and the design of Pepper, or alternatively, some of which were tested on Nao before becoming available on Romeo. Such R&D research projects have, for example, focused on hand and wrist operation, gestures to enhance communication, enhanced visual recognition or coordination of visual recognition with gestures in communication, robot safety, and collision avoidance. The following video shows just one such R&D project, focused on Romeo pouring a glass of water, while still tethered to a computer workbench.

The below-listed patent family, related to safe use of a humanoid robot, discloses one of the important inventions, connected to the Romeo project. This is an invention that addresses the issue of preventing damage to surroundings, in case the robot falls, and loses control of its movements. In fact, this invention, titled Safety of a humanoid-type robot in US2017072560, discloses an emergency stop button. 

The emergency stop button is triggered when a certain force is exerted that exceeds a given threshold, for example, in case of impact, or if the robot falls, for one reason or another. The button can also be actuated by an operator who wants to take the robot out of service because of some observed malfunction. Otherwise, the force exerted that exceeds a given threshold is preferably exerted by the robot’s movements, and in particular, the robot's head on its trunk, as shown in the appended patent Figure 4, below.

The following is a list of the patents belonging to the patent family disclosing the emergency stop button for the purpose of safely using humanoid robots. The abstract for the US member of the patent family is also included below with Figure 4 of the patent showing the robot's head actuating the switch on the robot's trunk. 

A humanoid-type robot comprises two elements and an articulation with at least one degree of freedom linking the two elements, the articulation allowing a travel in a given range in operational operation, a first of the two elements being intended to come into contact with an abutment belonging to a second of the two elements at the end of the range. According to the invention, the robot further comprises at least one switch. The switch is configured to actuate an electrical contact when a force exerted by the first element against the abutment exceeds a given force. [Abstract US2017072560]

• US2017072560 (A1) ― 2017-03-16 - Safety of a humanoid-type robot
• AU2015270476 (A1) ― 2016-12-01 - Safety of a humanoid-type robot 
• CA2950660 (A1) ― 2015-12-10 - Safety of a humanoid-type robot
• EP3152008 (A1) ― 2017-04-12 - Safety of a humanoid-type robot
• FR3021572 (A1) ― 2015-12-04 - Sécurité d'un robot à caractère humanoïde
• JP2017516671 (A) ― 2017-06-22 - ヒューマノイド型ロボットの安全性
• KR20170021800 (A) ― 2017-02-28 - Safety of a humanoid-type robot
• MX2016015822 (A) ― 2017-06-28 - Safety of a humanoid-type robot
• NZ726224 (A) ― 2017-09-29 - Safety of a humanoid-type robot  
• SG11201609420R (A) ― 2016-12-29 - Safety of a humanoid-type robot
• WO2015185670 (A1) ― 2015-12-10 - Safety of a humanoid-type robot 

References

Softbank Robotics

https://www.ald.softbankrobotics.com/en/robots

Softbank Robotics - Nao

https://www.ald.softbankrobotics.com/en/robots/nao

Softbank robotics - Pepper

https://www.ald.softbankrobotics.com/en/robots/pepper

Sofbank Robotics - Romeo

https://www.ald.softbankrobotics.com/en/robots/romeo 

CNRS

http://www.cnrs.fr/index.php

INRIA

https://www.inria.fr/en/

Oh, patents! Nao's siblings (Pepper)

Copyright Françoise Herrmann

The definition of a humanoid robot according to many of the Softbank Robotics patents is the following:

A robot can be qualified as humanoid from the moment … it has certain human appearance attributes: a head, a trunk, two arms, two hands, etc. A humanoid robot may, however, be more or less sophisticated. Its limbs may have a greater or lesser number of articulations. It may control its own balance statically and dynamically and walk on two limbs, possibly in three dimensions, or simply roll over a base. It may pick up signals from the environment (“hear”, “see”, “touch”, “sense”, etc.) and react according to more or less sophisticated behaviors, and interact with other robots or humans, either by speech or by gesture. [Extracted from US20170197311A1] 

Thus, it probably comes as no surprise that Nao is also part of humanoid family that includes a couple of humanoid siblings. Pepper is Nao’s first humanoid sibling. This Softbank Robotics humanoid robot measures 4 ft in height. Pepper’s trunk is humanoid, with two arms, two hands and a head. The lower half of her body is designed as a skirt comprising wheels to ensure mobility. Pepper is equipped with a 3D camera to detect people and their emotions, as well as her own surroundings. The little robot interacts via a tablet and voice. Her hands, fingers and forearms together display 20 degrees of freedom for gestures, specifically designed to enhance communication.  

Pepper was designed as a business companion, in particular for retail and services. She is the ideal host, fielding reception of clients in hotel lobbies, at airports, and in buildings. She might also be found in stores, where customers might query her about goods or inventory. Pepper is also connected, using apps that can be downloaded to increase, or customize, functionalities. She was also designed with a community of developers in mind, who would provide her with endless new possibilities of use and functionality.

The following video pitches Pepper to developers:

Pepper embodies numerous patented inventions, some of which are found in all the humanoid robots manufactured by Softbank Robotics. For example, see posts referring to the inventions covering Nao's hands and Nao's skull. However, the following two patent families are each respectively specific to Pepper’s mobility, and her ability to dock independently into a recharging base:

Patent Family I (Pepper’s mobility)

• US20170144299A1 - Omnidirectional wheeled humanoid robot based on a linear predictive position and velocity controller
• AU2015248711 (A1) ― 2016-11-03 - Omnidirectional wheeled humanoid robot based on a linear predictive position and velocity controller  
• EP2933069 (A1) ― 2015-10-21 - Omnidirectional wheeled humanoid robot based on a linear predictive position and velocity controller  
• CA2946049 (A1) ― 2015-10-22 - Omnidirectional wheeled humanoid robot based on a linear predictive position and velocity controller 
• HK1216406 (A1) ― 2016-11-11 - Omnidirectional wheeled humanoid robot based on a linear predictive position and velocity controller 
• JP2017513726 (A) ― 2017-06-01 - Omnidirectional wheeled humanoid robot based on a linear predictive position and velocity controller
• KR20170030078 (A) ― 2017-03-16 - Omnidirectional wheeled humanoid robot based on a linear predictive position and velocity controller
• SG11201608204Q (A) ― 2016-10-28 - Omnidirectional wheeled humanoid robot based on a linear predictive position and velocity controller 
• WO2015158885 (A2) ― 2015-10-22 - Omnidirectional wheeled humanoid robot based on a linear predictive position and velocity controller 

Patent family II (Pepper’s ability to dock herself into her recharging base)

• US20170080816A1 - Battery charging base and recharging method implementing such a base
• AU2015270600 (A1) ― 2016-12-01 - Battery charging base and recharging method implementing such a base
• CA2951060 (A1) ― 2015-12-10 - Battery charging base and recharging method implementing such a base 
• EP3152080 (A1) ― 2017-04-12 - Battery charging base and recharging method implementing such a base
• FR3021914 (A1) ― 2015-12-11 - Battery charging base and recharging method implementing such a base 
• JP2017518195 (A) ― 2017-07-06 - Battery charging base and recharging method implementing such a base 
• KR20170026441 (A) ― 2017-03-08 - Battery charging base and recharging method implementing such a base 
• MX2016015828 (A) ― 2017-06-28 - Battery charging base and recharging method implementing such a base
• SG11201609422X (A) ― 2016-12-29 - Battery charging base and recharging method implementing such a base 
• WO2015185525 (A1) ― 2015-12-10 - Battery charging base and recharging method implementing such a base

References

Softbank Robotics – Pepper

https://www.softbankrobotics.com/us/pepper

Wikipedia – Pepper (Robot)

https://en.wikipedia.org/wiki/Pepper_(robot)