Sight Impaired Mobility Assistance

From Mindworks
Jump to navigation Jump to search
Sight Impaired Mobility Assistance
Simulation enviornment
Team Name SONAR
Sponsor Daniel Schneider
Faculty Advisor
  • Brian Dyre
  • Duration Fall 2016 - Spring 2017

    Our goal is to create software that represents a virtual environment to the user through audio feedback. The result should be that the user should have a complete “image” of their soundings, based on what they hear. Thus, they should be able to navigate rooms, pick up objects, and complete goals completely through listening.

    Problem Description

    The current methods in place to assist the visually impaired in navigating their environment are inadequate. We seek to address this issue by translating visual data into audio data to allow users to comprehend the world around them through sound. If we can prove that this can be done in a simulation, then future groups can prove that this process could work in the real world. The steps to achieve this simulation are the following

    1. Design a 3D environment which represents the real world
    2. Test systems of audio feedback to the user and determine which is most understandable.
    3. Package our software so that it is easily distributed.

    Project Details


    Headphones Depth Camera Headtracking System
    Audio Technica ATH-m50x Intel RealSense Wii Remote
    • Low Cost
    • Studio Quality
    • Frequency Responce: 15 - 28,000 Hz
    • Realsense r200 depth camera
    • Small, low power, light weight camera
    • Easily accessible
    • Prebuilt Unity package
    • A lot of users already have the equipment


    Game Engine 3D Sound Engine Depth Data Interpreter
    Unity 3D Engine OpenAL Sound Library OpenCV Library
    • Free
    • Easy to learn
    • Included depth grabbing tools
    • Free
    • Contains the tools we require
    • Uses C++, which we are familiar with
    • Free
    • Works well with OpenAL
    • Also uses C++


    Project Design

    The design for our simulation includes a front end, and a back end. The front end which manages the virtual world, takes user input, and creates packets of spacial data. The back end consumes packets of spatial data, and generates audio feedback, which is then relayed to the user. Splitting the simulator in this manner allows for the back end to be used, with minimal alteration, with real data from the physical world in a future iteration of the project.

    Front End

    The front end of the simulator is responsible for the virtual environment and collecting sensor data. We are using the Unity engine, as it provides us with a simple and robust tool for managing a 3D virtual world. A Unity tool of depth mapping is used to get depth data from the visible scene, which is then placed in shared memory for use by the back end. The front end also includes a head tracking system. This will allow the user to look around their enviornment by moving their head. Our current headtracking system uses a Wii-Remote hat that the user wears. We chose this model as its cheap, and most users who may download our software may already have a Wii-Remote

    Back End

    The back end of the simulator is responsible for generating aural feedback from the depth data captured in the front end. It is built in C++, using the OpenAL library for sound generation and the OpenCV library for processing of the depth data.

    Virtual Environment

    Virtual Environment. Several rooms and roaming robots give plenty of testing opportunities.

    For our simulated environment, we plan on having several different scenes for testing. Currently, we have a map that randomly generates rooms, hallways, and walls of varying heights. This way, we can offer a unique experience every time. We also plan on developing a simple room where the user stays stagnant, and we move several different objects of differing shape and sizes around the player. Then we can ask them to gauge each objects distance and direction to the player. Lastly, we programmed some complex objects in the environment. For example, the user can pick up a green bottle from a table, and set it down elsewhere. Also objects, like cars, which locate and move towards the player. We plan on seeing if we can have a user avoid objects or find and move objects as way of testing our system.

    Audio Feedback Model

    Representation Of Environment With Audio

    One of our biggest challenges is how we will represent the virtual environment through sound. We plan on having several different models of how we represent the environment with audio feedback. During testing, we will test all these models and see which one works best for users. Our main model is shown to the right. It represents height with pitch, depth with sound pulses, the angular size of a viewed object with volume, and actual size of that object with timbre.

    Alternative Designs

    We considered several different designs before settling on our current one. They are listed below.

    Front End

    We considered several different game engines for the front end. The Unreal Engine was considered, as its a powerful game engine with a powerful 3D sound engine. However, we found that the sound engine wasn't powerful enough, and the game engine was very tasking so that not all our team members could run it. We also considered using Blender as it was easy to use. However, we wanted something with more tools included. So we settled on the Unity Engine.

    A major issue with our front end was how we were to grab environment depth data in the Unity engine. First, we tried ray casting. This was very slow so we tried other methods. Next, we tried a complex method of reading moving textures on the map. Finally, we settled on a built in Unity function of depth mapping. This was much faster and easier than the previous two methods.

    Back End

    The back end needed a powerful 3D sound engine in order for us to give accurate audio feedback. First, we tried the 3D sound engine included in Unity. It was lacking in certain tools we needed though. We are now currently using OpenAL

    Audio Feedback Models

    Original Audio Feedback Model

    Our first step in creating Audio Feedback models was doing some research. We found a similar study to ours at the University Of St. Andrews. For their study, they represented height with pitch and depth with volume. This model is seen to the right. Original St.Andrews study

    We then consulted a professor who specialized in audio perception of the brain. We learned from him that this was not an optimal model, as the human ear has a hard time distinguishing pitch and volume just by itself. In order to interpret pitch, one must compare it to a different pitch. Same with volume. So we designed a sound pulsing model. This model has a constant volume and pitch, followed by the differing sound pulse that represents the environment. This way, the user can relate the two, and hear their environment more clearly.

    Team SONAR

    Picture Bio Discipline
    Matt Daniel
    Matt Daniel: Matt is a citizen from Nampa, Idaho. He is a member of FarmHouse Fraternity and the Student Alumni Relations Board on campus. Computer Science
    Mason Fabel
    Mason Fabel is a senior in computer science at the University of Idaho. Growing up in Portland, Oregon, he was introduced to computers through LEGO Mindstorms, and later FIRST robotics. Outside of technology, Mason enjoys playing guitar, reading about history, taking long walks, and a variety of board and card games. Computer Science
    Eric Marsh
    Eric Marsh: From Boise, Idaho. Eric started programming his freshman year at the University of Idaho. Since then, he has made several video games and projects. He likes to make stir frys, read music reviews, and play Team Fortress 2. Computer Science
    Colin Pate
    Colin Pate: Colin is an Electrical Engineering major from Kirkland, WA. He enjoys working on engineering projects and going to Costco in his free time. Electrical Engineering
    John Snevily
    John Snevily is a senior in Computer Science at the University of Idaho. He enjoys rugby, Unity projects and anything made with Python/Pygame. Computer Science

    Document Archive

    All meeting minutes, client discussion notes, and meeting agendas can be found on our team Google Drive.