Oh, patents! Hololens gestures (3)

Copyright © Françoise Herrmann

Did you notice how Microsoft Hololens users resort to gestures to control the holograms? Actually, they use a combination of gaze and gestures, and sometimes voice and the Hololens clicker.

Several gestures are commonly used in combination with a gaze. 

- The Bloom gesture is the Home gesture. 

- To move the virtual cursor, users move their heads and gaze at a holographic app, a holographic menu item or hologram. 

- To select a holographic item (i.e.; to "click" on it), users combine gaze and Air Tap, swiftly tapping their finger in the air. 

- To scroll, pin/unpin, drag and resize a holographic item, users combine gaze, Air Tap, hold and movement. For example, to scroll, users gaze, Air tap and hold a scroll tool, and then move their hand up or down.  To drag, users gaze, Air tap and hold a drag tool, and then move their hand right or left.

Easy! But how does it work since everything is virtual, and the user’s fingers are not touching anything? Traditional modes of input invoke various sorts of controllers, joysticks, mice, keyboards, haptics, or voice. Gesture input to control mixed reality, using the Hololens, is read and interpreted via the head-mounted Hololens camera sensor. When the gesture is performed within the camera’s field of view, the cursor is ring-shaped, when the gesture is performed outside the camera’s field of view, the cursor is dot-shaped.  Once the gesture is captured by the camera, it is then extracted from the flow of images, identified as a particular gesture, and then converted to input for the holographic app, or for legacy programs (i.e.; programs that were not designed to receive input in the form of gestures).

Microsoft Hololens gestures within the larger context of user input, invoke several patented inventions, the following three in particular:

• US20100306714 - Gesture shortcuts
• US20150363005 - Techniques for using gestures for gesture-unaware programs
• US20170144067 - Assigning gesture dictionaries

The Microsoft YouTube video below shows uses of the Hololens AirTap gesture to select and manipulate various holographic items.

References

Microsoft Hololens – Use gestures
https://support.microsoft.com/en-us/help/12644/hololens-use-gestures

Microsoft Hololens

https://www.microsoft.com/en-us/hololens

Oh, patents! Hololens (2)

Copyright © Françoise Herrmann

Are you wondering how the Hololens possibly works --beaming in avatars or enabling collaborators to walk around a virtual object that is superimposed on the real world? Any pair of 3D glasses lets you see virtual objects, but walking around them to see them from all angles, including from the rear, is a Hololens feature.

Moving about to visualize the back of a hologram is a feature of the Hololens recited in US20150317831(A1) titled Transitions between body-locked and world-locked augmented reality. In a nutshell, the virtual object appearing on the see-through Hololens display as a volumetric hologram may be body-locked in 2D video mode, as it moves with the user around the room, or it may be anchored in the spatial mapping of the real-world in 3D, enabling the user to walk around it, the transitions from one state to another occurring via animation, and according to certain events. Thus, in the body-locked position, the hologram moves relative to the real world. In the real world-locked position, the user moves relative to the object.

The abstract of this invention is included below, together with patent figures 5 and 6, showing the two locked positions of holographic objects, respectively real-world locked in Fig. 5 and body-locked in Fig. 6.  

Various embodiments relating to controlling a see-through display are disclosed. In one embodiment, virtual objects may be displayed on the see-through display. The virtual objects transition between having a position that is body-locked and a position that is world-locked based on various transition events.

The Orinox-AvevaYoutube video included below shows how the Hololens is used to visualize an engineering plant in holographic 3D, Enlarged, and Full-scale sizes, including labels with datasheet details. 

References

Microsoft Hololens

https://www.microsoft.com/en-us/hololens

Interlude - The Kanneh Masons

Copyright © Françoise Herrmann

How about a family of 7 kids, each one a music prodigy! 

The Kanneh Masons are British, hailing from Sierra Leone. The children are all musicians, classical musicians. Their names are Isata (21 – pianist), Braimah (20 – violinist), Sheku (19 – cellist), Konya (17 - pianist and violinist), Jeneba (15 – pianist and cellist), Aminata (12 - violinist and pianist) and Mariatu (8 – cellist and pianist). 

They are making waves.  Here is a video of their performance at Britain's 2017 Classroom Heroes.

References
The Kanneh-Masons
http://www.kannehmasons.com/

Oh, patents! Microsoft Hololens – EPO Award Finalist

Copyright © Françoise Herrmann

In the holographic - augmented reality (AR) domain of invention, Microsoft’s HoloLens is competing for a 2018 European Patent Award, in the non-EPO (European Patent Organization) country category.  The HoloLens is a holographic AR computer (running Windows 10) that looks like a pair of head-mounted smart glasses. Using the HoloLens allows users to interact with holograms that are superimposed onto the real world. 

Introduced in 2015, the growing Microsoft BUILD developer community has already developed many HoloLens applications. For example, applications already exist, such as:

• holographic instruction manuals, including expert assistance, that pop-up next to complex machinery, superimposing the real-world parts
• holographic models of the human body, and of medical equipment, for medical instruction purposes
• the design of mechanical engineering projects such as planes and automobiles using holograms that can be manipulated in 3D
• visualization and manipulation of construction project holograms, in real space and time
• real people (e.g.; experts, politicians, speakers, doctors) might also be “beamed in" as avatars for real-time participation in meetings, consultations or other situations

 The video below stages various uses of the Hololens for the design of a new shoe product display. The video illustrates how teams can collaborate beaming in avatars, building in 3D with holographic tools, and visualizing a project before its implementation in the real world. 

The Hololens invention includes many aspects, which are all patented. In particular,  patents have been filed for the HoloLens hardware -- the sound, visual and movement sensors, the Holographic Processing Unit (HPU), designed to process the terabytes of digital data frames per second produced by the sensors, and the transparent lenses through which both the real world and the holograms are mixed and visualized, Patents also cover the AI algorithms invoked to both translate the HPU-processed sensor data, and to merge the 3D holographic animation, in real time and space, so that the holographic data can be manipulated (moved, altered or viewed) in 3D and real time. Indeed, about 250 patents are returned, linked to various aspects of this invention, granted to the Hololens inventor, Alex Kipman and teams.

Included below, a hyperlinked list of selected patents covering the HoloLens invention in what Microsoft calls “mixed reality. These patents are listed, in addition to the competing European patent, EP3028121, titled Mixed reality graduated information delivery.

• US2010302253 (A1) ― 2010-12-02 - Real-time retargeting of skeletal data to game avatar 
• US2010306714 (A1) ― 2010-12-02 - Gesture Shortcuts
• US2011025689 (A1) ― 2011-02-03 - Auto-generating a visual representation
• US2012158755 (A1) ― 2012-06-21 - Granular metadata for digital content
• US2012194645 (A1) ― 2012-08-02 - Living room movie creation
• US2013050833 (A1) ― 2013-02-28 - Adjustment of a mixed reality display for inter-pupillary distance alignment 
• US2013147687 (A1) ― 2013-06-13 - Displaying virtual data as printed content
• US2013147838 (A1) ― 2013-06-13 - Updating printed content with personalized virtual data
• US2013169683 (A1) ― 2013-07-04 - Head mounted display with iris scan profiling 
• US2014152558 (A1) ― 2014-06-05 - Direct hologram manipulation using imu 
• US2014168261 (A1) ― 2014-06-19 - Direct interaction system mixed reality environments
• US2014320508 (A1) ― 2014-10-30 - Systems and methods for applying animations or motions to a character 
• US2014347391 (A1) ― 2014-11-27 - Hologram anchoring and dynamic positioning
• US2014368534 (A1) ― 2014-12-18 - Concurrent optimal viewing of virtual objects
• US2014368532 (A1) ― 2014-12-18 - Virtual object orientation and visualization 
• US2014368537 (A1) ― 2014-12-18 - Shared and private holographic objects
• US2015370528 (A1) ― 2015-12-24 - Interactive content creation
• US2014375680 (A1) ― 2014-12-25 - Tracking head movement when wearing mobile device
• US2015317831 (A1) ― 2015-11-05 - Transitions between body-locked and world-locked augmented reality
• US2015363005 (A1) ― 2015-12-17 - Techniques for using human gestures to control gesture unaware programs 
• US2016086382 (A1) ― 2016-03-24 - Providing location occupancy analysis via a mixed reality device 
• US2016171779 (A1) ― 2016-06-16 - Opacity filter for display device 
• US2016210789 (A1) ― 2016-07-21 - Mechanism to give holographic objects saliency in multiple spaces  
• US2016292850 (A1) ― 2016-10-06 - Personal audio/visual system
• US2016321841 (A1) ― 2016-11-03 - Producing and consuming metadata within multi-dimensional data
• US2017080342 (A1) ― 2017-03-23 - Game browsing
• US2017069143 (A1) ― 2017-03-09 - Indicating out-of-view augmented reality images
• US2017144067 (A1) ― 2017-05-25 - Assigning gesture dictionaries 
• US2017178412 (A1) ― 2017-06-22 - Mixed reality graduated information delivery
• US2017177082 (A1) ― 2017-06-22 - Stabilization plane determination based on gaze location
• US2017216718 (A1) ― 2017-08-03 - Device for identifying and tracking multiple humans over time
• US2017236332 (A1) ― 2017-08-17 - Reality mixer for mixed reality

References

Microsoft BUILD Live - Microsoft Developers

https://developer.microsoft.com/en-us/events/build/content

Microsoft HoloLens

https://www.microsoft.com/en-us/hololens

Oh, patents! Concours Lépine 2018 – First Prize for MEDPACK

Copyright © Françoise Herrmann

Every year, since 1901, the Concours Lépine is hosted in Paris (France). Originally organized by Louis Lépine, the Paris Police Commissioner, to protect small artisan toymakers from growing competition, the modern-day version of the exhibit and contest is now hosted under the auspices of the Paris Faire (Foire de Paris), held every year in Spring.

 In 1901, the toymakers flocked to the exhibit because they were given free patents. Today, no one gets a free patent for their participation, but everyone submitting to the Concours Lépine contest is an inventor! The many prizes awarded are all reminiscent of the administrative roots of the contest. For example, the first prize is called the President of the Republic Award, the second prize is the Senate Award, the third prize is The National Assembly Award etc., through to the Police Commissioner Awards and the City of Paris Awards. Competition is quite stiff with some 500 contestants participating in 2018.

The 2018 President of the Republic Award was awarded to Samuel Mercier, fireman, first responder and Emergency Medical Technician (EMT), for the MEDPACK, a medical ER workstation. 

Drawing on his experience as a fireman EMT, eight years of ergonomic R&D and the idea that the iconic "doctor's bag" needed serious upgrading, Mercier invented this emergency first response workstation for use in difficult emergency situations. Thus, the workstation is designed to resolve a series of very specific issues arising in ER situations, such as portability for the management of space, light, hygiene, waste, safety, time and staff. In particular, the design incorporates a very precise ergonomic analysis of the EMT’s gestures and actions as demonstrated in the video (in French) included below.

This invention is recited in a French patent, FR3016513, titled Station de travail médicale extra-hospitalière. A US patent application was also filed, US2016338792(A1), titled Extra-hospital medical workstation.

The patent discloses such aspects as:

• a telescoping mast equipped with a powerful light
• three articulated and telescoping legs adaptable to uneven ground 
• a drawer and a nested tray, functioning as  horizontal work surfaces when deployed
• means for carrying the workstation
• means for dispensing different sorts of tape
• means for securing, organizing, and protecting supplies from wind, rain, sand, and snow
• means for sorting and safely disposing of needles and vials
• means for opening vials
• means for preparing, setting up and starting an IV
• removable tray for tracheal intubation performed on the ground
• means for loading fresh trays and drawers containing new supplies
• means for portability of the whole workstation.

The abstract of this invention is included below, together with a patent drawing of the deployed, and ready-to-use, workstation. 

The invention relates to a medical equipment structure intended to be used chiefly by medical or paramedical personnel in an extra-hospital environment, comprising at least a work tray (500), as well as a raised hook-up system and attachment means allowing care waste storage accessories to be attached, characterized in that said structure further comprises a platform (10) having a top surface, a bottom surface, and several lateral surfaces, the platform (10) being connected at its bottom surface to folding legs and, by a pivot means, to at least one work tray (500), in that said platform (10) has a mast (300) passing through it, and in that it comprises a means (600) for holding, all of this allowing the collection of these various elements to be transported and set up quickly, stably and level both on flat ground and on uneven ground.

References 

Concours Lépine

http://www.concours-lepine.com/

Foire de Paris - Concours Lépine

https://www.foiredeparis.fr/Decouvrez-Foire-de-Paris/Concours-Lepine

Oh, patents! The Concours Lépine. Posted at Patents on the soles of your shoes, May 13, 2017

https://patentsonthesolesofyourshoes.blogspot.com/2017/05/oh-patents-concours-lepine.html

Paris emergency medic pack wins Concours Lépine 2018 (May 16, 2018)

https://www.connexionfrance.com/French-news/Paris-emergency-medic-pack-wins-2018-Concours-Lepine

Whoops! Cinco de Mayo

Copyright © Françoise Herrmann

Please get this straight for 2019!...

Cinco de Mayo is the celebration of Mexico’s victory against France, in the Battle of Puebla, in 1861, and not an equivalent of the Fourth of July, south of the border, which would be Mexico’s Independence from Spain in 1821.

Despite all the criticism surrounding the celebration, north of the border, elders in Mexico celebrate with La danza de los viejitos.  

In the YouTube video below, the Orquestro Flor de Dalia de Jarachuaro de Michoacan is accompanying the dancers.

References

Cabrera, C & L. Lucero II (May 5, 2018) What is Cinco de Mayo? – NY Times

https://www.nytimes.com/2018/05/05/business/cinco-de-mayo-facts-history.html

Young, C. (May 5, 2018) The real history of Cinco de Mayo and how its celebrated around the world
http://www.businessinsider.com/cinco-de-mayo-history-2018-4

Battle of Puebla

https://en.wikipedia.org/wiki/Battle_of_Puebla

Bataille de Puebla

https://fr.wikipedia.org/wiki/Bataille_de_Puebla

Mexican War of Independence

https://en.wikipedia.org/wiki/Mexican_War_of_Independence

Guerre d'indépendance du Mexique

https://fr.wikipedia.org/wiki/Guerre_d%27ind%C3%A9pendance_du_Mexique

Oh, patents! Lego® Mindstorms® – EPO Award Finalist

Copyright © Françoise Herrmann

Sixty years later, and more than 1000 patients after the first Lego® brick patent was filed in 1958 by Godtfred Kirk Christiansen, the Danish Lego® Mindstorm® inventors, Gaute Munch, Erik Hansen and team, are competing for a 2018 European Patent Office Award in the Industry category. In the interim also, The Lego Group Toy Company has also become the third largest toy manufacturer in the world.

Lego® Mindstroms® brings together Lego® brick playworlds and coding for robotics, according to the seminal theories of education and computer use, expressed in the 1980 book, Mindstorms: Children, Computers and Powerful Ideas, written by MIT Professor Seymour Papert, one of the founders of the MIT Media Lab. In other words,  Lego® Mindstorms®, launched in 1998, is the culmination of an almost 40-year partnership between the industrial Lego Group R&D and groundbreaking research focused on constructivist approaches to learning and the development of programming languages for children, carried out at the MIT Media Lab. 

In the year 2018, the partnership involves the Lego Foundation and the Lifelong Kindergarten Group at the MIT Media Lab, where the Lego Foundation has endowed three graduate student fellowships and the Lego Papert Professorship of Learning Research, in honor of Seymour Papert.

The Lego® Mindstroms® inventors, Gaute Munch, Erik Hansen and team, contending for the 2018 EPO Award in the Industry category, produce the robotics kits that have “ignited classrooms” everywhere.  An ignition that is fueled by competition since the use of the kits has also promoted the inter-classroom FIRST Lego® League (FLL) tournaments, complete with local, national and international rounds. The teams compete according to elaborate First Lego® League (FLL) tournament rules -- investigating a real-world problem such as food safety or recycling, building a solution, designing and programming a Lego® Mindstrom® robot that competes on a tabletop, and developing team participation skills and ethics. 

Thus, participation in the First Lego® League (FLL) tournament program, using Lego® Mindstroms® invokes intensive application and development of STEM (Science, Technology, Engineering and Math) skills in the classroom, as well as imagination and much creativity, in a fun and energizing context. To date, more than 250,000 students have participated, in 32,000 teams, building 32,000 robots, at 1464 events,  in 88 countries!

The Lego® Mindstorm®  inventions, enabling children to build and program robots using their computers and/or tablets, remotely connected to an intelligent brick, sensors, beacons and motors,  are recited in a series of patents, four of which are cited below:

• EP1146941 (A1) ― 2001-10-24 - A remote-controlled  toy 
• EP1148921 (A1) ― 2001-10-31 - A programmable toy with communication means
• EP2217341 (A1) ― 2010-08-18 - A toy construction system
• EP2032227 (A1) ― 2009-03-11 - A toy building system

The video below shows the accelerated construction of a third-generation Lego® Mindstorm® EV3 robot.

References

Barr, J. (Dec. 19, 2017) MIT’s Lego Legacy: Iconic toymaker supports learning through play

https://www.technologyreview.com/s/609588/mits-lego-legacy/

First Lego® League

http://www.firstlegoleague.org/

Lego Mindstorm

https://wwwsecure.us.lego.com/en-us/mindstorms/products/mindstorms-ev3-31313 

Papert, S. (1980) Mindstorms: Children, Computers and Powerful Ideas.New York, NY: Basic Books.

Papert, S. & I. Harel, Eds (1991) Constructionism: research reports and essays 1985 - 1990 by the Epistemology and Learning Research Group, the Media Lab, Massachusetts Institute of Technology, Ablex Pub. Corp, Norwood, NJ.

Papert, S. (1993) The Children’s Machine: Rethinking School in the Age of the Computer. New York, NY:  Basic Books.
Papert, S. (1995) The Connected Family: Bridging the Digital Generation Gap. New York, NY: Basic Books.

Oh, patents! Lego® (1)

Copyright © Françoise Herrmann

Sixty years ago, the Danish inventor  Godtfred Kirk Christiansen, son of the Interlego AG Founder, Ole Kirk Christiansen,  filed the first Lego® brick patent. 

Indeed, the US patent  US3005258,  titled Toy Building Brick, was filed at the USPTO, on July 28, 1958, and granted on Oct. 24, 1961. The invention offered a novel way of assembling, and most importantly, re-assembling interlocking toy building blocks, alternatively called bricks. The open-faced, rectangular parallelopiped Lego® toy building brick invention thus comprises (to date) interlocking means in the form of protrusions and cavities enabling a clamping engagement of the bricks (see red bricks below). A clamping engagement that can be repeated indefinitely, and in different ways, for the purposes of generating a great many different sorts of play structures and constructions. The bricks were otherwise originally made of plastic and produced using an inventive injection molding process.

The first patent drawing page, containing six of the 12 original patent figures, is included below, respectively showing for :  

• Fig. 1 - the protrusions (21 and 22), on both sides of the base plate (10), comprising the interlocking system; 
• Fig 2 - a plan view of a hollow, open-faced rectangular brick, showing how the circular protrusions (21 and 22) fit together within the side walls (11 and 12); 
• Fig. 3 - a cross-section of the Fig. 2 brick, showing the protrusions on both sides of the base plate; 
• Fig. 4 - two stacked bricks assembled lengthwise;  
• Fig. 5 - a cross-sectional view of the Fig. 4, showing how two bricks are stacked and connected lengthwise, and
• Fig. 6 - two bricks assembled sideways.

An image of two red bricks, showing the interlocking protrusions on each side of the plate base, is also included below the patent drawings. 

Note: The name Lego®,  short for leg godt meaning  “Play well” in Danish, was used from the onset. 

References

Lego® USA

https://www.lego.com/en-us/

BrickiPedia: Lego® Wikis in other languages

http://lego.wikia.com/wiki/Brickipedia:LEGO_Wikis_in_Other_Languages

Ole Kirk Christiansen and the history of Lego®

https://www.thoughtco.com/ole-kirk-christiansen-lego-1991644