Sightless Navigation and Perception (S.N.A.P)

From Mindworks
Jump to: navigation, search
Sightless Navigation and Perception
Main Menu created in Unity
Team Name SONAR
Sponsor Daniel Schneider
Faculty Advisor
  • Bruce Bolden
  • Duration Fall 2017 - Spring 2018

    Team SONAR’s goal is to invent a device for the visually impaired that can create a highly detailed, acoustic picture of their surroundings granting them the ability to efficiently navigate their environment.

    Problem Description[edit]

    SNAP leverages modern robotic vision systems to produce augmented echolocation used for sightless perception of the surrounding environment. This system aims to provide those who are visually impaired with a means of perceiving their environment in real-time, and at a resolution never before accomplished.

    The success of SNAP relies heavily on our innate ability to locate objects in 3D space. This ability called “Sound Localization”, is achieved through binaural hearing. Much like binocular vision, which grants us depth perception, binaural hearing lets us compare incoming sounds as it is heard by each ear to triangulate the origin.


    Headphones Depth Camera
    Audio Technica ATH-m50x Intel RealSense
    • Low Cost
    • Studio Quality
    • Frequency Responce: 15 - 28,000 Hz
    • Realsense r200 depth camera
    • Small, low power, light weight camera


    Game Engine 3D Sound Engine Depth Data Interpreter
    Unity 3D Engine OpenAL Sound Library OpenCV Library
    • Free
    • Easy to learn
    • Included depth grabbing tools
    • Free
    • Contains the tools we require
    • Uses C++, which we are familiar with
    • Free
    • Works well with OpenAL
    • Also uses C++

    Project Learning[edit]


    Since our project deals heavily with being able to manipulate audio, we needed a powerful Audio library, which is why we are using OpenAL. OpenAL is a very powerful, 3D audio library which will allow us to communicate to the user how far away objects through sound.


    We also needed a library that will be able to convert our depth data we recieve from our Intel RealSense hardware into actual "Mat" objects that will be able to communicate with our multiple audio sources from OpenAL. For this purpose, we will be using OpenCV which is a library for real-time computer vision.


    Unity will allow us to perform tests on our backend programming so that we aren't having to constantly rely on working with our Intel RealSense hardware at all times. By creating a first-person module of a camera moving by multiple objects, this will allow us to perform tests on our backend without having to use our hardware. Our backend program will be able to work seamlessly between our Unity tests and the physical hardware.

    Map Creation[edit]

    One important component to our Unity test bed, is our unique testing maps. Our goal to complete our project with three different testing maps that each provide a separate challenge to the user. In order to create these maps, there was a learning curve in how to create 3-D environments in Unity that implement dynamic models and a basic physics engine.

    Main Menu[edit]

    Along with our testing maps, we wanted the main menu and user interface to also be done through Unity. To incorporate this, there was a large amount of research on how to allow users to switch between different scenes, load in different backend audio configuration files, and be able to create basic tutorials from the main menu.

    Documentation/Learning Current Codebase[edit]

    The previous group's backend file has little documentation. When looking through the backend file we were confused on how certain functions work. On top of learning about the previous groups own function's, we are also trying to learn about how certain OpenAL and OpenCV functions work. We decided it would be best if we took some time to look through the backend file and start documenting what we think each function does. So far, we have a write up of what some of the functions in the backend file do. We are planning on meeting with Collin who was a student who worked on the project last year. If he can walk us through how the backend file operates and answer some of our questions, we can start documenting more and make some more forward progress.

    Agile Tools[edit]

    Agile Tools
    Agile Tools

    Our client Dan Schneider plans on having future capstone groups continue the work that we have done. He even eventually wants to create a company based on this software. For that reason we want the project to be set up the right way. That is, the future groups will be able to quickly and easily get up to speed and understand the intricacies of the system well enough to make meaningful contributions. For this to be possible we will be using a few different tools that are available on Github.

    Project Components[edit]

    Project Components


    Allow OpenAL to be configurable. Maximize source resolution. Optimize shared memory block communication. Keep everything compatible with hardware.


    Create intuitive configuration menu. Sandbox configuration mode. Create a config file format that can be saved and shared.

    Logging Component[edit]

    Keep info from each session such as # of collisions, time taken to complete, configurations used, map/test used, etc. Allow for easy data export and analytics on test data.

    Menu System[edit]

    Handle switching between unity scenes and submenus. File Selection Dialogs for loading config files. Tutorial and manual pages.

    Character Controller/Headset Simulator[edit]

    Robust first person controller that can simulate human navigation. Keep headset in sync with hardware.

    Installation and Distribution[edit]

    Installation and Distribution

    Everything needed to run the simulation must be packaged together in one download file. Must include an easy installation\build script that minimizes the need for user input and installs all required dll files. (Essentially a single “Install” button.) Ideally would have a GUI installation process. Must Find an easy way to host and distribute the Installation file.

    Design Goals[edit]

    Visual Audio Engine Design[edit]

    Current State of the Code[edit]

    The previous group created what they called "The Backend" in c++ using OpenCV to analyze a depth map frame and from that information use OpenAL to create 3D audio sound sources around the user to inform them about objects in front of them.

    Areas of Improvement[edit]

    New Name[edit]

    We made the decision to rename The Backend to the Visual Audio Engine (VAE) to avoid confusion with common web development terms 'frontend' and 'backend'.

    More Documentation[edit]

    Very little of the code is actually documented and much of it is nontrivial and hard to figure out. This leads to a very steep learning curve when new developers attempt to improve upon it.

    Modular Class Structure[edit]

    One of our core requirements for this project is to be able to easily modify every aspect of the visual audio algorithms that we use to figure out the best possible way to translate visual information into audio information. For this to be possible we need the VAE functionality to be modular. To accomplish this we plan on breaking the algorithms and functionality out of the "main" function and into a more modular class hierarchy that we can utilize to accomplish the level of customization that our client requires. BackEndUML.jpg

    Some key aspects are the InputModule that allows for either a camera or shared memory space to get frame data. The OpenALModule that abstracts out all of the complicated openAL boilerplate to allow for generating multiple sound algorithms quickly. The ConfigModule uses a c++ JSON library to read in configs in JSON format. And finally the SoundAlgorithm which is the basic building block for an algorithm we can inherit from it to create all the different sound algorithms we need.



    We will be creating three distinct testing maps for our users, with each one provided a different obstacle to replicate a different type of real world environment. Our first map we have designated as our "Hallway Map," because the main focus for the user is to navigate through a hallway of static objects. Our second map will incorporate moving objects, to allow for users to see how well a specific audio configuration works with dynamic objects. Our third map will incorporate the y-axis, so users will have to be able to maneuver through obstacles such as a staircase.

    Configuration Menu
    Configuration Menu


    The configuration menu allows users to edit the test bed configurations.

    Configuration Menu


    The logs menu is where you can filter through all the log files from test runs.

    Logs Menu

    Map Selection[edit]

    This is a simple menu for selecting different test maps.

    Map Select Menu

    Project Timeline[edit]

    Timeline Semester 1
    Timeline Semester 2

    Team SONAR[edit]

    Picture Bio Discipline
    Dustin Fox
    Dustin Fox: Dustin hails from the beautiful city of Emmett, Idaho. He is currently a member of the Phi Gamma Delta Fraternity on campus. In his free time, Dustin enjoys Skiing and playing guitar. He has a passion for technology and artificial intelligence. Computer Science
    Dylan Carlson
    Dylan Carlson: Dylan was born in the temperate city of Plano, Texas. He later moved to attend high school and college at the University of Idaho. Dylan is currently a member of the Phi Delta Theta Fraternity on campus. Computer Science
    Andrew Rose
    Andrew Rose: Andrew grew up in Nampa, Idaho. He came to the University of Idaho in 2014 and is majoring in Computer Science. In his free time he enjoys listening to music and laughing with friends. Computer Science

    Document Archive[edit]