Computer “Vision” for the Visually Impaired: I have a dream

The Shock

  • Unfortunately, technology is not effective in helping them doing their daily basic tasks
  • The mobile phone is not the best medium/technology to support blind people, it is not practical and it was not designed for that
  • Most of the mobile apps i tried were not effective
  • Many support software (like JAWS) and mobile apps are commercial and expensive

Their Needs

  • To Recognize objects around them
    Case: If something fell on the ground, they can’t find it easily. Also they may forget the location of their belongings (the keys or example).
  • To Recognize people and faces
    Case: when a visually impaired person enters a room they don’t know who is inside and they have to wait for the other people to greet and identify themselves which is really frustrating.
  • Fast text reader and color recognizer
    Case 1: Imagine you are blind and you want to buy milk, how would you check the expiry date, how would you make sure you didn’t pass that date later.
    Case 2: You go shopping, you can touch the cloth to know the quality of the material, how would you know the color, size and the price.
  • Support navigating, avoiding obstacles and crossing the street
    One of the major problems — even in top world cities
    Technology support is needed to help the blind walk fast and safe to his/her target destination, specifically help is needed to give warning if you are about to hit an obstacle and need to step down the curb. Another problem is knowing when to cross the street. Maybe technology can help detecting the color of the traffic light and the surrounding movements to support the blind pedestrian decision. Finally indoor navigation is needed.
  • Better experience interacting with mobile
    Current accessibility and voice guiding features are good, but not good enough. I believe navigation should be more intuitive and driven by natural conversation between the Human and the Mobile system.
  • Scene Explainer
    Ever thought if you are blind how would you see images and videos, for those who lost their sight in their mid life and know what is a sky and how does a river look like, it would be very nice for a system to be able to explain scenes.

My Dream

  • Recognize Faces and Objects using a “deep-learning system” — Convolutional Neural Networks (CNN) like GoogLeNet — The system can recognize faces and will ask for a name for each face to associate with so next time it can tell you who is in the room. The system is also capable of recognizing objects indoor and outdoor and is able to change modes when triggered by some objects.
  • Read Text and Recognize Colors
    The user is in the store, he/she can read product names, expiration dates and choose their preferred product color. Another use case is triggering the “street-cross mode” after detecting the traffic light object and then give advice to pass when signal turns to green.
  • Navigation (Indoor and outdoor) and Obstacle Warning
    The system utilize both GPS, camera input and object detection and maybe laser distance measurement to give advice on outdoor navigation as well as indoor. An example: when walking in a park and the footpath is curving and you don’t own a guide dog (an actual case that really happened) the system will give warning and produce stereo sound in the direction of the curve as well as warning for obstacles and curb edges.
    Think about a Google Self Driving System for Humans.
  • Scene Explainer
    A blind girl is walking in the street and hears some noise which sounds like a street show. The girl ask the device to describe the scene using voice command; the system respond with “a man juggling balls”. An illustration of such functionality can be found in this facebook video. You can imagine so many other cases.
  • Modelling 3D space using sound
    This is something i don’t have experience in and not sure if it could even work. The first case was illustrated in the “Navigation” feature above to help the user feel curves using sound. The question now: can we model the whole 3D space using sound waves to give the blind person a new dimension of moving things that does not produce sounds ?
    If you close your eyes and a Bee passes from your left to your right you can see the direction although you don’t use your eyes, right ?
    Now, can a blind guy play Table Tennis ? as illustrated in the following links Holophonics Holophonic 3D Sound Compilation using such technology we can convert the movement of the ball to sound waves so that the brain can workout its location and the blind user can return the ball back ! I hope it is possible …
    (UPDATE: https://www.youtube.com/watch?v=I0lmSYP7OcM)
  • This device shouldn't be commercial, it should be given for free or at least at production price. I believe organization interested in the welfare of blind people and philanthropic foundations such as Bill & Melinda Gates Foundation and Chan Zuckerberg Initiative can help in that
  • The device should work offline and online
  • The device is processing intensive so it should be supported by a processing unit as well as a centralized cloud system
  • Strong battery
  • It should be a personalized learning system
  • Multilingual

Conclusion

That was my dream. I am looking forward to that day when i can see it happening, a day where i can obtain an early version of that device and give it as a personal gift to my dear friend and see the effect of technology on her life.

So far i can think of 2 companies capable of creating such device: Google and Facebook.

--

--

--

Freelance Consultant — Data & Product. Writing about Data, Entrepreneurship and Life. https://karim.ouda.net

Love podcasts or audiobooks? Learn on the go with our new app.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Karim Ouda

Karim Ouda

Freelance Consultant — Data & Product. Writing about Data, Entrepreneurship and Life. https://karim.ouda.net

More from Medium

Coloring Black and White Images

Remove shadows and uneven lighting from photographs of writing on paper

iMaterialistic(Fashion)2019 at FGVC6

PyTorch to CoreML: Writing custom layers with Metal