backtop


Print


  (Source: Microsoft/Guide Dogs UK)
Bone conductive headphones and Bluetooth beacons are paired a Windows phone app, powered by Cortana and Bing Maps

Microsoft Corp. (MSFT) has completed the design and first test trials of a "proof of concept" wearable that can help the blind navigate around a city.  Consisting of a pair of bone conductive headphones and a smartphone app that ties various Microsoft data troves together, this is arguable a one-of-a-kind accessiblity assistance device. 

And soon, Microsoft hopes it may be available to the 39 million people around the world living with severe vision impairment.

I. Bold Vision From a Man Who Lost His Sight

The idea traces back to one of its own -- Microsoft manager Amos Miller.  Today Mr. Miller is seemingly on top of the world.  Growing up in a kibbutz in Israel he took naturally to the software field.  He graduated with honors from the Israel Institute of Technology (Technion) and then jetted off to the UK, where he earned a Master's degree in business administration (MBA) from London Business School.  After stops in the IT industry, he joined Microsoft in 2007.  After a series of promotions he now is serving as Microsoft's director of Asian enterprise strategy.

Indeed, today Mr. Miller seemingly has it all -- a wife, a high-paying career, and a young daughter.  But for all his success, Mr. Miller still grapples daily with a tough challenge faced by an estimated 39 million+ people worldwide -- finding his way around in the world without sense of vision.



Mr. Miller has retinitis pigmentosa (RP), a hereditary degenerative vision disorder.  Early in life doctors made it unclear to expect the unexpected -- RP leaves many patients blind, but its progression varies greatly from patient to patient.

Retina Pigmentosa
Retinitis Pigmentosa
[Image Source: Bio1020 Blogspot / Natural Cure Clinics / Retinal Microscopy]

Mr. Miller recalls:

I started university fully sighted, but finished not being able to read anything. Not the white board, not the books, not the computer screen.  I actually did go through some therapy at university to start calling myself a blind person. That wasn’t the purpose of it, but it was one of the results. I learned that life could be much better and easier once you accept it, and it did become much better.
 
With the help of his seeing eye dog, which he adopted in 2007, he has for nearly a decade navigated with relative confidence through the streets of London, his adopted home.  But he is well aware that navigating busy city streets is very challenging for a vision impaired person -- even one with a seeing eye dog.  After all, dogs can't tell you where the closest spot to grab a bite to eat is.

Microsoft Amos Miller
Microsoft manager Amos Miller [Image Source: ThirdSector.co.uk]

A smartphone might tell you that.  But even they fall short. A local public transit authority might offer vision impaired auditory clues, for example, and/or release a well-integrated smartphone app that tells riders exactly what bus is arriving when.  But many city services lack these kind of accessibility efforts.  And even those that have them, may be unable to save a vision impaired person from getting on the wrong bus, if the scheduled bus on the route is running late, and a different route is running early.

Mr. Miller was passionate about giving back.  In 2008 he won a seat on the board of The Guide Dogs for the Blind Association, a group who is not only the UK's largest seeing eye dog trainer, but also the trainer who gave Mr. Miller his guide dog.  He took the task of strengthening the group as seriously as his day job.  By the time he left in 2012, the group was much more financially stable and active.

Guide Dogs for the Blind

But he stil felt he could do more.  One thing that struck him was how many of the 80-year-old charity's 8,000 customers were relying more on guide dogs and classic white canes -- pre-internet era tools -- rather than a digital solution.  Part of the problem was that there was no comprehensive digital solution to guide the blind in cities.  Mr. Miller wondered -- could Microsoft possibly put its software prowess to work, improving the lives of millions of vision impaired people worldwide?

It was worth a shot.

II. Dreams of a New World

So he approached Richard Leaman, CEO of Guide Dogs, and the group's research director, Jenny Cook, offering them a vision of a bold collaboration.  Guide Dogs jumped at the opportunity.

Mr. Miller recruited a close friend at Microsoft -- Indian-born UK immigrant Jarnail Chudge.  Together the two Microsoft managers brainstormed their pitch, formulating many of the ideas that would go into the finished wearable.

Janrnail Chudge
Microsoft coworker Jarnail Chudge backed helped Amos Miller make his vision a reality.

To sell their idea, they created a 15 minute concept video entitled "A Family Day Out".  The short film depicts a special day in the life of an impaired family man -- inspired by Mr. Miller's own trials and trevails.  In the video the man finds in posession of a wearable headset which gives him new confidence to explore his city.  Able to navigate fully for the first time he goes ont to enjoy a day at the museum with his family.

Microsoft's senior director for accessiblity, Jenny Lay-Flurrie received the video and was inspired by its message.  In the U.S. 65 percent of people with vision loss are unemployed.  Could Microsoft do something to help?



The video was submitted to an annual Microsoft employee competition, called the Ability Summit held this April.  

New Microsoft CEO Satya Nadella presented the pair of employees with a special award.  As he walked off, Mr. Chudge tracked him down and convinced him to consider a pitch for a wearable device for the blind.  After an email to Mr. Nadella elaborating on the vision, several Microsoft superstars were assigned to the project.

Microsoft's CEO assigned one of his close friends to the team -- Dave Campbell, chief technology officer for Microsoft’s Cloud and Enterprise.  Mr. Campbell recruited a couple of digital wizards from Microsoft Research for the effort -- chief scientist Henrique (Rico) Malvar, who had a key role in designing Microsoft's 3D audio and Bing Maps technology, and Bill Buxton, one of multi-touch's true inventors from the late 1970s.

Mr. Buxton, in particular, is well know for his work at the University of Toronto where he helped design multi-touch.  In 1985 [PDF] he had proposed a "Multi-Touch Three Dimensional Touch-Sensitive Tablet".  Mr. Buxton was perhaps the first to propose a capacitive multi-touch tablet similar to the hardware which would come more than two decades later.  Mr. Buxton was a contemporary of Myron Krueger (Univ. of Conneticut) -- founder of the "Videoplace" [PDF], a user interface lab where pinch-to-zoom and other multitouch gestures were invented.



Together the pair are known as some of the true "inventors" of multi-touch -- a fact which may come as a bit of a shock for those who believe that late Apple, Inc. (AAPL) CEO Steve Jobs "invented" the tablet, multi-touch hardware, or multi-touch gestures.

Both Mr. Campbell and Mr. Buxton loved the idea of an altruistic wearables project.  Mr. Campbell enthuses:

The purpose and meaning of this project meant it was so incredibly easy to enlist people — it just sort of tugged at them.  I've been on this design empathy kick for close to 10 years, and I think it's really important to the company right now.

We were born as a company of engineers building products for other engineers and geeks. That's no longer the case. This notion of design empathy, or being able to build something for someone who is not like you — it's much, much harder. You really need to adopt a perspective other than your own to pull it off.

After so many years of pushing the boundaries of algorithms and user interface research, why not give it a go once more?

III. Software Makes Off-the-Shelf Design Feel Purpose-Built

Now the project was getting serious.

The goal was to create a proof of concept system that would guide the blind through the city.  The system would be a high-tech supplement to a guide dog. And for people that are unable to afford a seeing eye dog (which have a lifetime cost of around $80,000 USD), it could eventually offer some mild independence, as it matures.  The World Health Organization (WHO) estimates that 90 percent of the blind living in poverty, so that would be a prettty big deal.

Mr. Learman comments what a difference the technology could make, remarking:

In the U.K., there are about 180,000 people who rarely if ever get out and about. That’s a massive social issue in its own right.  That was really the heart of our new strategy. Supporting blind people with 5,000 guide dogs is really not scratching the surface.

Introducing new services, like the use of technology, is a vital ingredient to change. If our major cities in the U.K. had sensor-enriched zones where this capability worked, I think it would be an incredible achievement. Beyond that, I expect this to go global, as I’m sure major cities around the world will want the same. It’s like an awakening.

Guide Dogs UK and Microsoft worked together to develop a prototype of this unprecedented piece of wearable electronics.  Microsoft would come to dub the effort "Cities Unlocked".  The project uses relatively cheap off the shelf hardware to create a package that feels purpose-built.



At the heart of the system is a Windows smartphones running an app which ties together Microsoft's navigation (Bing Maps) and location aware services, alongside data from local businesses and transit firms cooperating with the project.

Microsoft Cities Unlocked -- Windows Phone
[Image Source: Guide Dogs UK/Flikr]

The software for Cities Unlocked was test on Microsoft Lumia 1520 smartphones, 6-inch devices which pair Qualcomm, Inc. (QCOMSnapdragon 800 SoCs with 2 GB of LPDDR3 DRAM.  This device retails for $585 USD, and packs Windows 8.1 Update 1.  The Cities Unlocked app leverages Microsoft's advanced mobile OS, integrating Bing Maps and Microsoft's voice assistant Cortana to allow the users to search for and select a location using natural language input.  

Lumia 1520
Bing Maps and Cortana running in Windows Phone 8.1 Update 1 on the Nokia Lumia 1520.

More fine-grain information is enabled via a variety of third party partners, who use various rich data technologies to offer information such as real time location details on city buses and trains.

Among the partners on the project are: The partners worked together to demonstrate how the technology could enable the blind to more easily navigate about neighborhood or city sidewalks, utilize public transit, and pick up essentials like groceries.

IV. The Father of Multi-Touch Debuts 3D SoundScape

Perhaps the most crucial part of the project demonstration was its use of a brand new kind of auditory feedback technology being matured by Bill Buxton.  His lab is just one of several labs at Microsoft Research exploring novel new sound feedback technologies [PDF] for the vision impaired.

The feedback mechanism, as well as more traditional voice turn-by-turn directions and conversative menu navigation is supplied by the Cities Unlocked app.  The app "talks" to the user via an open ear, bone conductive headband, which feeds sound into the users jaw.  

Aftershokz headphones

The headset appears to be a slightly modified and retrofitted version of the Bluez Bluetooth headset, which retails for $85 on Amazon.com, Inc. (AMZN).

Microsoft's Aftershokz

This special headset is popular among gamers and outdoor athletes (e.g. cyclists) alike as it's light weight, rugged, and packs terrific sound.  The biggest perk, is that it relies on bone conductive sound, which allows users to carry on conversations.  For the blind, Microsoft felt this feature would be critical both from a safety and a social perspective.



Microsoft uses its Kinect depth perception technology to map users faces and optimize the sound input [PDF] through the headphones whose vibrations are carried via the the users' upper jaw bones.

Lastly, there's a bit more Microsoft special sauce that turns this good piece of hardware into greatness.  The Microsoft app uses a technology called 3D SoundScape, which uses direction sensors and navigation information to add audio cues to the spoken turn by turn directions.

3D SoundScape is driven by a combination of GPS for coarse grain positioning, and Bluetooth beacons for finer grain positioning/routing.  Microsoft, Apple, Inc. (AAPL), and Google Inc. (GOOG) are among the companies racing to stock tomorrow's cities with smart Bluetooth beacons.  

3D Bluetooth Beacon
Microsoft installed Bluetooth beacons all over a neighborhood in Reading.

Microsoft and the Bluetooth Standards International Group recently announced that Windows 10 would pack support for Bluetooth Smart beacons, hot on the heels of Apple's promotion of its "iBeacon" technology.  Microsoft also sells "Treasure Tag" Bluetooth beacons to customers, which interface with a Windows Phone app which helps users find lost keys or other items.  Treasure Tags are available from Amazon for $30 USD.

The 3D SoundScape navigation technology perhaps traces back to the work [PDF] of University of California, Santa Barbara UCSB) Psychology Professor Jack Loomis and University College of London student Ethan L. Smith.

3D Sound Navigation
UCSB Psychology Professor explored 3D navigation for the vision-impaired via sound feedback in a landmark 2006 study. [Image Source: ACM/UCSB]

A click-clack noise will tell you you're off course.  You turn your head until you hear the click-clack subside into a cheery ping.  That's the direction you need to head in.  The stereo feedback technology is to hearing what multi-touch is to touch, expanding the user's senses to allow surprising new interactions.

Unsurprisingly, the advanced auditory feedback firmware is being developed by the father of multi-touch, Bill Buxton, at Microsoft Research.  He describes:

The best technology is invisible. It just lets me get on with my life.  When an interaction with technology is as it should be, the user is not there as an equipment operator, but as a human being. They are not walking down the street operating technology, but with the intent to go to work, or get fresh air, or exercise.

[3D SoundScape is like a pair of shoes.]  I never think about my shoes as mobility devices, but it’s the same kind of thing. They just enable me to do what I want without getting in the way.

But would this "new shoe" find a fit amongst its target audience?  Microsoft was about to find out.



The device was put to the test in a trial Microsoft called "Independence Day".  Microsoft selected a diverse group of 8 testers for the project.  Some testers were blind (having become blind (some from birth, others late in life), others were people with vision who were blindfolded for the test.

The demo took testers through Reading and London, cities in the UK which are roughly an hour and a half apart.  Reading was a natural choice as it's the location of Microsoft's UK headquarters.

Microsoft HQ in Reading
Microsoft's UK Headquarters in Reading, UK [Image Source: Cylon]

The device drew rave reviews from users with vision, who said the experience was so natural and compelling it could be popular among sighted, as well as vision impaired users.

Among these sited users were a couple sighted members of the media.  BBC technology correspondent Rory Cellan-Jones took the device out for a spin and offered cautious praise, writing:

At first I found it distracting rather than helpful, a clippety-clop sound echoing around my head plus a repeated ping to say I was on track. But as I hesitantly made my way down a residential street, across a road and to a bus stop, the instructions - "parked cars and overhanging trees ahead" - gave me added confidence and the 3D sound provided me with a somewhat better feel for my surroundings.

For me it was all about feeling a little less scared - but for the visually impaired people who have been testing the technology for some weeks, it seemed to have increased their confidence in taking new routes up to a new level.


Tom Warren from The Verge was another tester.  He writes of the experience:

Microsoft blindfolded me and led me around several streets before leaving me on the ledge of a freezing cold canal in London yesterday. Not in the interest of torturing me, or product secrecy; it was simply the best way to transport me to a reality that 39 million people face daily: blindness.

...

If you’re on a train or a bus [the device] will tell you when you’re about to reach a station or stop using GPS or nearby beacons, but it’s blindly walking around environments where it’s especially impressive.

The first step is positioning your head towards a click-clack sound until you hear a small constant pinging noise. That’s the trigger to let you know you’re facing in the right direction for navigation, thanks to a compass embedded in the back of the headset. At first, blindfolded and relying on the headset and cane, I still felt anxious. Within minutes, though, the combination of a white cane and this subtle guide helped me walk straight, around corners, and even cross roads with assistance. If the pinging noise cut out I felt lost, but as soon as I aligned myself again I was back on track. It felt natural, like a guide was placing me in the right direction, and adding the white cane allowed me to sense any obstacles in the way.

...

This technology feels like a demonstration of what Microsoft is capable of, and a new sign of collaboration across groups within the company. There's an attention to detail here and Microsoft is trying to take technology out of the way to focus on the experience, making this a particularly interesting experiment to see where the company is heading in future.

Microsoft's chief storyteller -- Jennifer Warnick -- was another sighted user who was able to experience the device in London, UK.  She describes:

Because the headphones didn't cover my actual ears, I could also listen for environmental noises. I'd never before had an audio experience like this — its richness helped me visualize the neighborhood around me while its immersiveness gave me more confidence with every step.

Eventually, about halfway through the walk, I relaxed enough to carry on a conversation. By the time I reached the bus stop, I was chatting away with the team from Microsoft and Guide Dogs, even as the headset beamed route updates and points of interest to my inner ear.

I was so excited to get the hang of it that I was reluctant to remove the blindfold and headset once we reached Reading Station. Parker and Yates said this is a common reaction from people who have made the journey, visually impaired and sighted alike.

I'm not sure if I could make my way across my own living room blindfolded, at least not without some bruising, and yet I'd just traveled across an unfamiliar city relying primarily on a cane and a few well-placed, 3D sounds. Where I anticipated feeling vulnerable and anxious in the blindfold, I ended up feeling strangely super-powered wearing the headset, like some sort of dry-land dolphin.

"Man," I thought to myself as the train to London pulled away from Reading Station, "there hasn’t been this much magic in the British suburbs since Harry Potter was dropped at 4 Privet Drive."

Other users were vision impaired.  Amos Miller himself was among the first to test the device his vision had materialized into.

Microsoft Independence Day
Microsoft's Amos Miller tests his dream device.

He remarks:

This is Version 0.1.  Someday I want the language we use to be rich and creative — to use metaphors and prose to help me build a picture of what’s around me.

[The realization that technology can be crafted for the disabled] actually gives me hope that technological products down the line are going to be fun for me to use, and not a drag, because I love technology.

Jennifer Bottom, a software tech support worker, was another tester.  

Jennifer Bottom
Jennifer Bottom, blind from birth, was among those to test the device.  She was impressed.

Blind from birth, she said the technology was a nice supplement to her seeing eye dog Archie.  She remembers initial skepticism recalling:

There are quite a few technology offerings out there which certainly claim they’re going to change the lives of blind people — that it will be the same as if you could see, which of course isn’t true.

I’m quite a cynical person. It took me a few minutes to get used to the technology and to trust the feedback.  Overall, I was very comfortable with it. You can definitely tell that visually impaired people have been consulted, not just a bunch of executives.

Kristin Grice, another vision impaired tester told the BBC:

We want to live like normal people. We don't always want to plan ahead to see if we can get community transport or a taxi or something, we want to be able to just jump on a bus and go somewhere and have that freedom.

Kristin Grice tries the device

Kristin Grice arrives at bus stop

Kristin Grice
Tester Kristin Grice navigates through a neighborhood, arrives at a bus stop, and boards a bus, with help from the device. [Image Source: Guide Dogs UK/Flikr]

The device was also well received by Terry Brewell, 69, a retired phone company employee who slowly lost his sight 40 years ago.  He states:

It makes you feel more confident. You get advance warning of what’s ahead, or what your surroundings are.  One of the hardest parts is being confronted by unexpected situations. Like where I live they’re rejuvenating the town center, and it’s quite a mess.

Terry Brewell
Terry Brewell prepares to board a train with some help from the Microsoft device.

Microsoft's results are impressive.  It's build a solid technology demonstration, with a price of around $635 USD in off-the-shelf parts (the $550 Lumia 1520 and $85 Aftershokz Bluez headset).

Now the hard work to advance the product to commercial use begins.  Microsoft is determined to bring its Cities Unlocked vision all across the globe, ensuring millions of vision impaired people worldwide get to experience an "Independence Day" of their own.

Sources: Microsoft, Guide Dogs [UK], Catapult -- Future Cities, The Verge





"Paying an extra $500 for a computer in this environment -- same piece of hardware -- paying $500 more to get a logo on it? I think that's a more challenging proposition for the average person than it used to be." -- Steve Ballmer













botimage
Copyright 2017 DailyTech LLC. - RSS Feed | Advertise | About Us | Ethics | FAQ | Terms, Conditions & Privacy Information | Kristopher Kubicki