Sightless Navigation and Perception (S.N.A.P)

Team SONAR’s goal is to invent a device for the visually impaired that can create a highly detailed, acoustic picture of their surroundings granting them the ability to efficiently navigate their environment.

Problem Description
SNAP leverages modern robotic vision systems to produce augmented echolocation used for sightless perception of the surrounding environment. This system aims to provide those who are visually impaired with a means of perceiving their environment in real-time, and at a resolution never before accomplished.

The success of SNAP relies heavily on our innate ability to locate objects in 3D space. This ability called “Sound Localization”, is achieved through binaural hearing. Much like binocular vision, which grants us depth perception, binaural hearing lets us compare incoming sounds as it is heard by each ear to triangulate the origin.

OpenAL
Since our project deals heavily with being able to manipulate audio, we needed a powerful Audio library, which is why we are using OpenAL. OpenAL is a very powerful, 3D audio library which will allow us to communicate to the user how far away objects through sound.

OpenCV
We also needed a library that will be able to convert our depth data we recieve from our Intel RealSense hardware into actual "Mat" objects that will be able to communicate with our multiple audio sources from OpenAL. For this purpose, we will be using OpenCV which is a library for real-time computer vision.

Unity
Unity will allow us to perform tests on our backend programming so that we aren't having to constantly rely on working with our Intel RealSense hardware at all times. By creating a first-person module of a camera moving by multiple objects, this will allow us to perform tests on our backend without having to use our hardware. Our backend program will be able to work seamlessly between our Unity tests and the physical hardware.

Map Creation
One important component to our Unity test bed, is our unique testing maps. Our goal to complete our project with three different testing maps that each provide a separate challenge to the user. In order to create these maps, there was a learning curve in how to create 3-D environments in Unity that implement dynamic models and a basic physics engine.

Main Menu
Along with our testing maps, we wanted the main menu and user interface to also be done through Unity. To incorporate this, there was a large amount of research on how to allow users to switch between different scenes, load in different backend audio configuration files, and be able to create basic tutorials from the main menu.

Documentation/Learning Current Codebase
The previous group's backend file has little documentation. When looking through the backend file we were confused on how certain functions work. On top of learning about the previous groups own function's, we are also trying to learn about how certain OpenAL and OpenCV functions work. We decided it would be best if we took some time to look through the backend file and start documenting what we think each function does. So far, we have a write up of what some of the functions in the backend file do. We are planning on meeting with Collin who was a student who worked on the project last year. If he can walk us through how the backend file operates and answer some of our questions, we can start documenting more and make some more forward progress.

Agile Tools
Our client Dan Schneider plans on having future capstone groups continue the work that we have done. He even eventually wants to create a company based on this software. For that reason we want the project to be set up the right way. That is, the future groups will be able to quickly and easily get up to speed and understand the intricacies of the system well enough to make meaningful contributions. For this to be possible we will be using a few different tools that are available on Github.

Visual Audio Engine
Allow OpenAL to be configurable. Maximize source resolution. Optimize shared memory block communication. Keep everything compatible with hardware.

Configuration
Create intuitive configuration menu. Sandbox configuration mode. Create a config file format that can be saved and shared.

Logging Component
Keep info from each session such as # of collisions, time taken to complete, configurations used, map/test used, etc. Allow for easy data export and analytics on test data.

Menu System
Handle switching between unity scenes and submenus. File Selection Dialogs for loading config files. Tutorial and manual pages.

Character Controller/Headset Simulator
Robust first person controller that can simulate human navigation. Keep headset in sync with hardware.

Installation and Distribution
Everything needed to run the simulation must be packaged together in one download file. Must include an easy installation\build script that minimizes the need for user input and installs all required dll files. (Essentially a single “Install” button.) Ideally would have a GUI installation process. Must Find an easy way to host and distribute the Installation file.

Current State of the Code
The previous group created what they called "The Backend" in c++ using OpenCV to analyze a depth map frame and from that information use OpenAL to create 3D audio sound sources around the user to inform them about objects in front of them.

New Name
We made the decision to rename The Backend to the Visual Audio Engine (VAE) to avoid confusion with common web development terms 'frontend' and 'backend'.

More Documentation
Very little of the code is actually documented and much of it is nontrivial and hard to figure out. This leads to a very steep learning curve when new developers attempt to improve upon it.

Modular Class Structure
One of our core requirements for this project is to be able to easily modify every aspect of the visual audio algorithms that we use to figure out the best possible way to translate visual information into audio information. For this to be possible we need the VAE functionality to be modular. To accomplish this we plan on breaking the algorithms and functionality out of the "main" function and into a more modular class hierarchy that we can utilize to accomplish the level of customization that our client requires.

Some key aspects are the InputModule that allows for either a camera or shared memory space to get frame data. The OpenALModule that abstracts out all of the complicated openAL boilerplate to allow for generating multiple sound algorithms quickly. The ConfigModule uses a c++ JSON library to read in configs in JSON format. And finally the SoundAlgorithm which is the basic building block for an algorithm we can inherit from it to create all the different sound algorithms we need.

Maps
We will be creating three distinct testing maps for our users, with each one provided a different obstacle to replicate a different type of real world environment. Our first map we have designated as our "Hallway Map," because the main focus for the user is to navigate through a hallway of static objects. Our second map will incorporate moving objects, to allow for users to see how well a specific audio configuration works with dynamic objects. Our third map will incorporate the y-axis, so users will have to be able to maneuver through obstacles such as a staircase.

Configurations
The configuration menu allows users to edit the test bed audio configurations. Users can alter variables such as the frequency range the audio sources can output and how the time interval of each scan.

Logs
The logs menu is where you can filter through all the log files from test runs.

Map Settings
Map settings allows users to alter variables about the map such as how many random obstacles will spawn and the sizes of these random obstacles.

Project Timeline