Speech Therapy Application

From Mindworks
Jump to: navigation, search

The goal of this project is to create a speech visualization tool that can analyze speech and provide visual feedback as well as an error report. This should have a modular design that can be easily added to in the future.

Sponsor Micron
Mentors Bruce Bolden & Feng Li
Team Name Smooth Talkers
Duration Fall 2017 - Spring 2018

Problem Definition[edit]

Using CMUSphinx, an open-source ASR toolkit, we will analyze speech files to detect errors, and we will develop a user interface in Java.


There have been many projects in recent years having to do with analyzing pronunciation errors. Many of these have used CMUSphinx. Our project is different in two ways. First, our project will be developed with a specific aim toward working with child speech. Second, we aim to take the current state of the art and improve on it.


Metric Importance (1 = More, 5 = Less) Units Marginal Value Ideal Value
Real time feedback 3 Seconds 60 5
Textual error response 1 Coverage Common Errors All Errors
Audio error response 2 Coverage Common Errors All Errors
Child speech samples for testing database 4 Samples 100 200
Child speech samples for training database 4 Samples 400 800
Working acoustic model 1 Error Rate 20% <15%
GUI Interface 1 x x x
Cross system compatability 5 OS Windows Windows, Mac, Linux
Handles low ambient noise 4 decibel 40 65
Spectrum Analysis (FFT) 1 x x x
Waveform Visualization 1 x x x
Additional speech analysis 2 Graphs Few Many

Project Learning[edit]


CMUSphinx is an ASR toolkit which we are using for this project. It uses the most state-of-the-art algorithms based on decades of CMU research.

Speech Recognition[edit]

Speech recognition consists of multiple steps. First there is feature extraction, forming a feature vector. Then, this vector is matched to the acoustic model.


The Blue Snowball Mic

The choice of a microphone took many factors into account. This program’s eventual goal is to be used in schools or people’s homes. Knowing that majority of users wouldn’t have access to expensive audio equipment the team decided to set a soft budget of $50 for a microphone. Using an inexpensive microphone will allow the team to test the program using audio quality that would be reasonable for a user to achieve. The final decision was to use the Blue Snowball Microphone.

The Snowball offers a great balance of audio quality and price. A cardioid recording mode is also included which only records audio directly in front of the microphone, useful to prevent unwanted sounds during recording. USB connectivity ensures that the mic should work on nearly every computer. The mic also comes with a small tripod, making it much more portable than a microphone requiring a boom arm mounting system.

This setup will allow for a portable audio recording system that provides quality audio recordings to train and test the programs acoustic model.

Speech disorders[edit]

Speech Sound Disorders, or SSDs, affect 10-15% of preschoolers and 6% of school-aged children. The majority of these have unknown causes.

There are two major categories of speech disorders: phonological disorders and motor speech disorders. In phonological disorders, patients are physically capable of producing the correct sounds. In motor speech disorders, patients physically cannot produce the correct sounds and must be shown how. Our software will be more helpful to the former category.

Relevant data for diagnosis includes omission and distortion of phonemes, stress errors, speed of speech, and consistency.

It's important to test phonemes alone, in syllables, in phrases, and in spontaneous speech. The results often differ between these contexts, and that information can be valuable to diagnosis.


We originally used Pocketsphinx with Python, but then discovered that Sphinx4 with Java is 200x faster, most likely due to optimizations implemented after the release of Pocketsphinx.



The Graphical User Interface will be written in Java and will look something like this:


Team Information[edit]

Member Biography Discipline
Simon Barnes Simon Barnes is a senior Computer Engineer at the University of Idaho. Interest in the project comes from his mother's involvement working with children who have speech disabilities as well as the programming involved for speech recognition. His hobbies include cars, technology, and Super Smash Bros Melee for the Nintendo GameCube. Computer Engineer
Emma Bateman Emma Bateman is a senior Computer Science major at the University of Idaho. She is interested in machine learning and wants to learn more about speech recognition. Computer Science
Joshua Bonn Joshua Bonn is a senior Computer Science major at the University of Idaho. He is interested in this project because of his own issues with speech and his learning through speech therapy as well as an interest in machine learning. His interests include video games and music. Computer Science