Visual Depth and Velocity Mapping

This project provides a sight substitution system for the visually impaired. Information about the surrounding environment is gathered by stereo vision cameras and used to generate disparity maps. These maps are then used to calculate the depth and velocity of surrounding objects. The depth and velocity values are expressed as vectors, which are then translated into audio signals of varying pitch and synchronization.

=Problem Definition=

Although there are extensive resources available for most disabilities, there are very few aids for the visually impaired. This project aims to write software to map visual stimuli to an auditory field using Real Time Disparity Mapping (RTDM) with a dual camera system and simple processor. The unit should be wearable and functional in the sense that somebody could walk around campus with it on their head. By November 27th we expect to have our first prototype running and functional.

Background
It has been demonstrated that those who are visually impaired are able to learn how to navigate via echo location. Thus, this project is based on the intuition that it is possible to learn how to translate variations of an audio signal into meaningful information which can then be used to ascertain the spatial relationship of objects in the surrounding environment.

Deliverables
The deliverables for this project include the following:

1. Functioning hardware (ZED stereo camera integrated with Nvidia GPU processing board)

2. Proprietary modifications to SIFT and SNAP disparity mapping algorithms

3. Portfolio containing documentation

Specifications


User Interface Requirements:



A set of headphones and a camera shall be attached to a helmet or hat.

The device should be secured to the body in a configuration suitable for walking and sitting

The device should not impede user's ability to interact with objects in front or to either side of him or her

The device should not expose the user to any severe electric or mechanical hazards (electric shock or burning)  

What the product should do:

 This product will relay 3D position and velocity information in an audio format to the user. The resolution and accuracy of the data should be sufficient for traversing a room without the assistance of eyesight.</li> </ul> </li>

Physical Requirements



Maximum weight: 10 lbs</li>

Power: Board 12.6 W</li>

Camera 3.8 W</li>

Framerate 15 30 60 FPS</li>

Depth of field (max): 15 20 30 m</li>

</ul>

</li> </ul>

=Design Considerations= In order to create a Real Time Disparity Mapping system that could be used in a mobile setup, it was necessary to find hardware that was small enough to be worn on the user's body and yet powerful enough to process images. The first prototype that we proposed was a Tara Stereo vision camera that would be connected to a Raspberry Pi 3.0 B+, which would be powered by a battery pack.

However, after discussing this option with our client, we determined that the Raspberry Pi was not powerful enough to do the image processing that we needed and that the viewing angle of the Tara camera was not large enough. After more research, we found another camera called ZED produced by StereoLabs which had the desired viewing angle and depth perception. Also, we found another processing board called the Nvidia Jetson TX2 which had sufficient computational capabilities to perform the tasks that we needed.

Another problem that our team deliberated over was how to place the hardware on the user's body. We initially thought of placing the camera on the user's chest, but then decided to have it fixed to their head with the jetson board stored in a backpack along with a battery pack.

Another discovery that our team made was that we need to develop the code for the project on a custom development kit produced by Nvidia instead of Open CV. Links to each of these development environments are given below:



Open CV: </li>

Nvidia Jetpack: </li> </ul>

=Project Learning=

=Final Design=

=Validation=

=Team Members=

=Additional Documentation=

Project Schedule

Spectralink schedule.pdf

Meeting Minutes

Presentations



Client Interview