Sight Impaired Mobility Assistance

Our goal is to create a hardware accessory to give the visually impaired a greater level of autonomy. From finding a glass of water to navigating a busy street, we hope to provide ample audio feedback to give the user a complete "image" of what's around them.

Problem Description
Our problem is two part. Part one involves creating a virtual environment implemented in software. This environment is intended to represent the real world and give the user audio feedback. Part two of the problem is applying this knowledge to a real-world environment. This entails receiving input from a depth camera and allowing someone who is visually impaired to navigate a given environment which previously would have been impossible.


 * 1) Use or design a 3D environment which represents the real world
 * 2) Test systems of audio feedback to the user and determine which is most understandable.
 * 3) Package our software so that it is easily distributed.
 * 4) Port the software to work with our depth camera hardware and perform testing in the real world.

Concept Development
We are currently determining what tools to use for the front and back end of the project. For the front end 3D simulator, we have a simple navigable 3D world implemented in the popular 3D game engine Unity. Our main criteria for the 3D engine are ease of use, ability to procure depth data from the camera, and distributability to end users.

For the back-end of the project, we are evaluating ways to process the received depth data and relay it to the wearer as sound. One of the main goals of our project is to determine how to optimally relay the maximum amount of depth data to our users while allowing them to locate important features of their surroundings and distinguish as much detail as possible, without overwhelming the user.

We are currently experimenting with OpenAL for use as a sound generation environment.

Design


In order to work out the best method for translating spacial data into auditory feedback, we are developing a simulator instead of an actual headset. This program will allow the user to navigate around a virtual environment with both visual and audio feedback. The simulator is split into two major sections: a front end which manages the virtual world and creates packets of spacial data, and a back end which consumes packets of spacial data and generates audio feedback. Splitting the simulator in this manner allows for the back end to be used, with minimal alteration, with real data from the physical world in a future iteration of the project.

Front End
The front end of the simulator is responsible for the virtual environment and collecting sensor data. We are using the Unity engine, as it provides us with a simple and robust tool for managing a 3D virtual world. Custom rendering shaders are then used to get depth data from the visible scene, which is then placed in shared memory for use by the back end.

Back End
The back end of the simulator is responsible for generating aural feedback from the depth data captured in the front end. It is built in C++, using the OpenAL library for sound generation and the OpenCV library for processing of the depth data.

Document Archive
All meeting minutes, client discussion notes, and meeting agendas can be found on our team Google Drive.