Oh, patents! Microsoft Hololens – EPO Award Finalist

Copyright © Françoise Herrmann

In the holographic - augmented reality (AR) domain of invention, Microsoft’s HoloLens is competing for a 2018 European Patent Award, in the non-EPO (European Patent Organization) country category.  The HoloLens is a holographic AR computer (running Windows 10) that looks like a pair of head-mounted smart glasses. Using the HoloLens allows users to interact with holograms that are superimposed onto the real world. 

Introduced in 2015, the growing Microsoft BUILD developer community has already developed many HoloLens applications. For example, applications already exist, such as:

• holographic instruction manuals, including expert assistance, that pop-up next to complex machinery, superimposing the real-world parts
• holographic models of the human body, and of medical equipment, for medical instruction purposes
• the design of mechanical engineering projects such as planes and automobiles using holograms that can be manipulated in 3D
• visualization and manipulation of construction project holograms, in real space and time
• real people (e.g.; experts, politicians, speakers, doctors) might also be “beamed in" as avatars for real-time participation in meetings, consultations or other situations

 The video below stages various uses of the Hololens for the design of a new shoe product display. The video illustrates how teams can collaborate beaming in avatars, building in 3D with holographic tools, and visualizing a project before its implementation in the real world. 

The Hololens invention includes many aspects, which are all patented. In particular,  patents have been filed for the HoloLens hardware -- the sound, visual and movement sensors, the Holographic Processing Unit (HPU), designed to process the terabytes of digital data frames per second produced by the sensors, and the transparent lenses through which both the real world and the holograms are mixed and visualized, Patents also cover the AI algorithms invoked to both translate the HPU-processed sensor data, and to merge the 3D holographic animation, in real time and space, so that the holographic data can be manipulated (moved, altered or viewed) in 3D and real time. Indeed, about 250 patents are returned, linked to various aspects of this invention, granted to the Hololens inventor, Alex Kipman and teams.

Included below, a hyperlinked list of selected patents covering the HoloLens invention in what Microsoft calls “mixed reality. These patents are listed, in addition to the competing European patent, EP3028121, titled Mixed reality graduated information delivery.

• US2010302253 (A1) ― 2010-12-02 - Real-time retargeting of skeletal data to game avatar 
• US2010306714 (A1) ― 2010-12-02 - Gesture Shortcuts
• US2011025689 (A1) ― 2011-02-03 - Auto-generating a visual representation
• US2012158755 (A1) ― 2012-06-21 - Granular metadata for digital content
• US2012194645 (A1) ― 2012-08-02 - Living room movie creation
• US2013050833 (A1) ― 2013-02-28 - Adjustment of a mixed reality display for inter-pupillary distance alignment 
• US2013147687 (A1) ― 2013-06-13 - Displaying virtual data as printed content
• US2013147838 (A1) ― 2013-06-13 - Updating printed content with personalized virtual data
• US2013169683 (A1) ― 2013-07-04 - Head mounted display with iris scan profiling 
• US2014152558 (A1) ― 2014-06-05 - Direct hologram manipulation using imu 
• US2014168261 (A1) ― 2014-06-19 - Direct interaction system mixed reality environments
• US2014320508 (A1) ― 2014-10-30 - Systems and methods for applying animations or motions to a character 
• US2014347391 (A1) ― 2014-11-27 - Hologram anchoring and dynamic positioning
• US2014368534 (A1) ― 2014-12-18 - Concurrent optimal viewing of virtual objects
• US2014368532 (A1) ― 2014-12-18 - Virtual object orientation and visualization 
• US2014368537 (A1) ― 2014-12-18 - Shared and private holographic objects
• US2015370528 (A1) ― 2015-12-24 - Interactive content creation
• US2014375680 (A1) ― 2014-12-25 - Tracking head movement when wearing mobile device
• US2015317831 (A1) ― 2015-11-05 - Transitions between body-locked and world-locked augmented reality
• US2015363005 (A1) ― 2015-12-17 - Techniques for using human gestures to control gesture unaware programs 
• US2016086382 (A1) ― 2016-03-24 - Providing location occupancy analysis via a mixed reality device 
• US2016171779 (A1) ― 2016-06-16 - Opacity filter for display device 
• US2016210789 (A1) ― 2016-07-21 - Mechanism to give holographic objects saliency in multiple spaces  
• US2016292850 (A1) ― 2016-10-06 - Personal audio/visual system
• US2016321841 (A1) ― 2016-11-03 - Producing and consuming metadata within multi-dimensional data
• US2017080342 (A1) ― 2017-03-23 - Game browsing
• US2017069143 (A1) ― 2017-03-09 - Indicating out-of-view augmented reality images
• US2017144067 (A1) ― 2017-05-25 - Assigning gesture dictionaries 
• US2017178412 (A1) ― 2017-06-22 - Mixed reality graduated information delivery
• US2017177082 (A1) ― 2017-06-22 - Stabilization plane determination based on gaze location
• US2017216718 (A1) ― 2017-08-03 - Device for identifying and tracking multiple humans over time
• US2017236332 (A1) ― 2017-08-17 - Reality mixer for mixed reality

References

Microsoft BUILD Live - Microsoft Developers

https://developer.microsoft.com/en-us/events/build/content

Microsoft HoloLens

https://www.microsoft.com/en-us/hololens