Sightless Navigation and Perception (S.N.A.P)

Team SONAR’s goal is to invent a device for the visually impaired that can create a highly detailed, acoustic picture of their surroundings granting them the ability to efficiently navigate their environment.

Problem Description
SNAP leverages modern robotic vision systems to produce augmented echolocation used for sightless perception of the surrounding environment. This system aims to provide those who are visually impaired with a means of perceiving their environment in real-time, and at a resolution never before accomplished.

The success of SNAP relies heavily on our innate ability to locate objects in 3D space. This ability called “Sound Localization”, is achieved through binaural hearing. Much like binocular vision, which grants us depth perception, binaural hearing lets us compare incoming sounds as it is heard by each ear to triangulate the origin.

OpenAL
Since our project deals heavily with being able to manipulate audio, we needed a powerful Audio library, which is why we are using OpenAL. OpenAL is a 3D audio library which will allow us to communicate to the user how far away objects are via sound.

OpenCV
OpenCV is a real-time computer vision library that will convert our depth data we receive from Unity or our Intel RealSense hardware into actual "Mat" matrix objects that will communicate with our multiple audio sources from OpenAL.

Unity
Unity will allow us to perform tests using our Visual Audio Engine so we don't have to rely on working with our Intel RealSense hardware at all times. Using a first-person camera will allow us to gather frames and pass them to our Visual Audio Engine as depth images. Our Visual Audio Engine will be able to work seamlessly between our Unity tests and the physical hardware.

Map Creation
One important component to our Unity test bed is our unique testing maps. Our goal to complete our project with two different testing maps that each provide a separate challenge to the user. In order to create these maps, there was a learning curve in how to create 3-D environments in Unity that implement dynamic models and a basic physics engine.

Main Menu
Along with our testing maps, we wanted the main menu and user interface to be done through Unity. To incorporate this, there was a large amount of research on how to allow users to switch between different scenes, load in different visual audio configuration files, and store logging information from completed tests.

Documentation/Learning Current Codebase
The previous group's "backend" file has little documentation. When looking through the "backend" file we were confused on how certain functions work. On top of learning about the previous groups own function's, we are also trying to learn about how certain OpenAL and OpenCV functions work. We decided it would be best if we took some time to look through the "backend" file and start documenting what we think each function does. So far, we have a write up of what some of the functions in the "backend" file do. We meet with Collin who was a student who worked on the project last year. He walked us through how the "backend" file operates and answered some of our questions.

Agile Tools
Our client Dan Schneider plans on having future capstone groups continue to work on the project. He even eventually wants to create a company based on this software. For that reason we want the project to be set up the right way. That is, the future groups will be able to quickly and easily get up to speed and understand the intricacies of the system well enough to make meaningful contributions. For this to be possible we will be using a few different tools that are available on Github.

Visual Audio Engine
Allow OpenAL to be configurable. Maximize source resolution. Optimize shared memory block communication. Keep everything compatible with hardware.

Configuration
Create intuitive configuration menu with multiple configurable visual audio components. Create a config file format that can be saved, loaded and shared.

Logging Component
Keep info from each test session such as number of collisions, time taken to complete a test, and map used for testing. Create logging menu with filter system to quickly access past logging data.

Menu System
Handle switching between unity scenes and submenus. File selection dialogs for loading config files. Logging menu to view logging data. Map settings menu.

Character Controller/Headset Simulator
Robust first person controller that can simulate human navigation. Keep headset in sync with hardware.

Installation and Distribution
Everything needed to run the simulation must be packaged together in one download file. Must include an easy installation\build script that minimizes the need for user input and installs all required dll files. (Essentially a single “Install” button.) Ideally we would have a GUI installation process. Must Find an easy way to host and distribute the Installation file.

Current State of the Code
The previous group created what they called "backend" in c++ using OpenCV to analyze a depth map frame and used OpenAL to convert the depth map frame into 3D audio sound sources around the user.

New Name
We made the decision to rename The Backend to the Visual Audio Engine (VAE) to avoid confusion with common web development terms 'frontend' and 'backend'.

More Documentation
Very little of the code is actually documented and much of it is nontrivial and hard to figure out. This leads to a very steep learning curve when new developers attempt to improve upon it.

Modular Class Structure
One of our core requirements for this project is to be able to easily modify every aspect of the visual audio algorithms that we use to figure out the best possible way to translate visual information into audio information. For this to be possible we need the VAE functionality to be modular. To accomplish this we plan on breaking the algorithms and functionality out of the "main" function and into a more modular class hierarchy that we can utilize to accomplish the level of customization that our client requires.

Some key aspects are the InputModule that allows for either a camera or shared memory space to get frame data. The OpenALModule that abstracts out all of the complicated openAL boilerplate to allow for generating multiple sound algorithms quickly. The ConfigModule uses a c++ JSON library to read in configs in JSON format. And finally the SoundAlgorithm which is the basic building block for an algorithm we can inherit from it to create all the different sound algorithms we need.

Maps
We will be creating two distinct testing maps for our users, where each one provides a different obstacle to replicate a different type of real world environment. Our first map we have designated as our "Random Hallway Map," because the main focus for the user is to navigate through a hallway of random objects. Our second map will incorporate moving objects, to allow for users to see how well a specific audio configuration works with dynamic objects, such as people walking.

Configurations
The configuration menu allows users to edit the test bed visual audio configurations. Users can save their configurations to their computer for future use. The configuration files are saved in JSON format, which allows for easy readability and distribution.

Logs
The logs menu allows users to view their past test results. There is a filter on the right side where you can filter through logs based on map type.

Map Settings
Map settings allows users to alter variables about the map such as how many random obstacles will spawn and the sizes of these random obstacles.

Project Timeline




Document Archive
Presentations
 * Design Review 1 [[File:Design_Review_Presentation.pdf]]
 * Design Review 2 [[File:Design_Review_2_snap.pdf]]
 * Expo Technical Presentation [[File:Expo_presentation_snap.pdf]]

Expo
 * Poster [[File:expo_poster_snap.pdf]]

Github Repository
 * https://github.com/projectSNAP/SNAP-test-bed