Raspberry Pi Image Processing

The goal of this project is to integrate machine learning with the Raspberry Pi 4, leveraging the computational power of the Intel Neural Compute Stick 2 in order to get sufficient speeds to identify patches of weeds in farm plots.

=Problem Definition= Current computers suffer from being heavy and using high amounts of electricity. This prevents them from being carried by drones, which means that any sort of aerial image recognition is impossible.

=Background= With the advent of artificial intelligence, computers are now able to manage computationally-heavy tasks, such as facial recognition and self-driving vehicles. However, machine learning is typically limited by the processing speed of the computer, and is usually limited to bulky computers that are stationary or draw large amounts of power. The purpose of this project is to demonstrate the viability of machine learning on smaller machines, such as the Raspberry Pi, which could then be used for crop identification via drones. Since big computers are too heavy to be carried by a drone, the extremely light-weight Raspberry Pi would eliminate the problem of weight as well as the problem of high power draw.

=Deliverables=
 * Develop a benchmark for speed comparisons between the Raspberry Pi with the NCS2, and a standard computer with a modern processor and graphics card.
 * Implement a pre-trained artificial neural network with the Raspberry Pi.
 * Use a standard image processing library to process images in real-time.
 * Compare processing times between the computer and the Raspberry Pi.

=Value Proposition Statement= Most modern farmers use small, Kerosene-fueled airplanes to spray pesticides onto their fields. Our goal is to instead, use a small computing system atop a drone to detect weeds and trigger a spraying mechanism in order to save farmers time and money. If successful, modern farmers won’t need to use small airplanes that are dangerous, use fossil fuels, take up space, and pollute the atmosphere. Our weed spraying drone will be easier to store, cost-effective, more accurate, and require brief human interaction. This could improve the quantity as well as quality of foods. Our object detection will allow us to target exact locations that need spraying, so it will reduce the amount of toxic chemicals going into the soil. Our project will be using a raspberry pi to explore the benefits of using smaller computers to do computationally heavy tasks with computer weight limits.

=Design Considerations=
 * Develop a sufficiently light-weight and low-power system.
 * Train a neural network to accurately identify different types of weeds, specifically dandelions, clovers, and grass.
 * Develop a program to interface with the Raspberry Pi remotely.
 * Ensure our model runs quick enough for real-time computations.

=Project Learning=
 * Utilize pre-trained models on the TensorFlow Github, which allows us to get a prototype up and running quicker.
 * See the models here: Models
 * Incorporating the NCS2 with the Raspberry Pi is well-documented and supported by the Raspberry Pi operating system.
 * This tutorial is what we used to get the NCS2 working: OpenCV, NCS2, and the Raspberry Pi


 * Video of the system in action can be found here.

=Quickstart Guide=

Foreward
This guide will walk you through the entirety of the steps we took to make this project work. Over the course of this guide, you will learn to set up TensorFlow, install Ubuntu, install the Raspbian operating system on a Raspberry Pi, train a custom machine learning model, optimize the model for the NCS2, and finally run the model on your Raspberry Pi with the help of the NCS2.

One of the biggest issues with this project was the immense dependence on 3rd-party libraries. Between TensorFlow, OpenVINO, and the Raspbian operating system, we ran into numerous dependency issues due to libraries being updated, halting development until we could find a version of the offending package that worked with the project. TensorFlow itself is the worst example of this, where the newest version of the library is actually less supported than the older versions. Our suggestion is to install the exact same versions of libraries that we have when listed, which will guarantee that the project will work for you even if the libraries are older. If exact versions cannot be found, the closer to the target version the better.

Requirements

 * This section will cover hardware and software requirements and where to get them.
 * Raspberry Pi 4
 * Micro SD card for the operating system, the larger the better.
 * External battery or power source.
 * Intel Neural Compute Stick 2
 * Raspberry Pi Camera Module
 * Computer with Windows installed, for TensorFlow usage.
 * NVIDIA graphics card suggested for faster training, TensorFlow GPU is NOT compatible with AMD cards.
 * Computer with Linux installed for OpenVINO model optimizer usage.

Installing TensorFlow on Windows

 * This section will describe setting up our version of TensorFlow on Windows.
 * The first component necessary for this process to work is Anaconda. Anaconda will allow you to create separate Python development environments, and more importantly, install the specific versions of libraries that we need for our training purposes. It is not recommended to proceed without utilizing Anaconda.
 * Once Anaconda is installed, open the start menu and search for "Anaconda Prompt". Run this program as administrator.
 * In this prompt, run the command "conda create --name tfgpu". This will create a new Anaconda environment named tfgpu. The name doesn't matter, but for future training you'll need to remember it in order to enable your development environment.
 * Type "conda activate tfgpu" to open your development environment. The far left side of the prompt should now list (tfgpu) instead of (base).
 * Next, we'll install TensorFlow.
 * If training with your CPU, run TODO.
 * Otherwise, run the command "conda install -c conda-forge tensorflow-gpu=1.15".
 * Note: yIt's absolutely vital that you install version 1.15, or you risk issues with optimizing the model later on.
 * TensorFlow will install a variety of libraries, if any fail to install you'll need to find the exact version that TensorFlow 1.15 requires.

Setting up the Linux Installation

 * This section will guide you through installing Ubuntu for the model optimizer usage.

Setting up the Raspberry Pi Installation

 * This section will teach you how to download and install Raspbian, and also to enable the camera module on the Raspberry Pi.

Training a Custom Model

 * Gathering Training Data
 * Find 100+ images of each object class
 * Use labelImg to annotate each image with bounding boxes around the objects
 * Split the images and labels into a training (80%) and testing set (20%)
 * Training a Model
 * Use a pretrained model from TensorFlow model zoo
 * Edit the .config file to point to the training and testing data as well as the .pbtxt file with class names
 * Run the training script (train.py) with the path to the config file
 * Train until the loss reaches an accepted (low) value
 * Freezing a Trained Model
 * Run export_inference_graph.py with the path to the training dir, config file, and the desired model.ckpt file
 * Frozen model will be in the form of a .pb file
 * Use the frozen model in object detection tutorial scripts to test the model performance on various images

Setting up the Model Optimizer

 * To use the model optimizer
 * install openVINO on Windows using this guide: https://docs.openvinotoolkit.org/latest/openvino_docs_install_guides_installing_openvino_windows.html
 * install openVINO on Linux using this guide: https://docs.openvinotoolkit.org/latest/openvino_docs_install_guides_installing_openvino_linux.html
 * Download openVINO on a PC. Make sure it has the model_optimizer folder (if RPi install is used, it will not include this folder)
 * In order to run models on the Raspberry Pi, they need to be in a .bin .xml format
 * https://docs.openvinotoolkit.org/latest/openvino_docs_MO_DG_Deep_Learning_Model_Optimizer_DevGuide.html
 * To optimize a frozen model (trained using TensorFlow and SSD MobileNet)
 * Go to deployment_tools/model_optimizer and run: mo_tf.py --input_meta_graph path_to/model.ckpt.meta --tensorflow_object_detection_api_pipeline_config path_to/pipeline.config --tensorflow_use_custom_config \openvino\deployment_tools_model_optimizer\extensions\front\tf\ssd_support.json
 * use ssd_v2_support.json if trained with newer TensorFlow versions
 * this command will output .bin and .xml files
 * If using Windows, you must open cmd as administrator

Raspberry Pi and NCS2 Integration

 * This section will finally set up the NCS2 on the Raspberry Pi, and will run a simple script to show that the model is running correctly.

=Final Design=

=Validation=

=Team Members=

Victoria Gehring
Computer Science Student

Hometown: Meridian, ID

Hobbies/Interests: My professional interests include object detection, machine learning, and cyber security. In my free time I enjoy volleyball, camping, and PC building.

Plan for the Future: My plan is to further my understanding of machine learning and object detection to optimize and automate processes.

Email: gehr1898@vandals.uidaho.edu

Jon Gift
Computer Science Student

Hometown: Bend, OR

Hobbies/Interests: I enjoy working with Python and C#, and I have a significant amount of experience with both Raspberry Pi's and the Unity engine. I also like to dance, rock climb, and play guitar.

Plan for the Future: My goal is to work at Intel in Portland and eventually teach computer science someday.

Email: gift7380@vandals.uidaho.edu

Isabel Hinkle
Computer Science Student

Hometown: Coeur d’Alene, Idaho

Hobbies/Interests: My professional interests include all things cybersecurity. I am most interested in the topic of Digital Forensics and filesystem analysis. In my free time I like to create art, listen to music, and hang out with my friends.

Plan for the Future: My goal is to work for a Federal Executive agency pursuing a career in a cybersecurity-related field.

Email: hink0402@vandals.uidaho.edu

Oshan Karki
Computer Science Student

Hometown: Kathmandu, Nepal

Hobbies/Interests: I like everything related to AI and Machine Learning. In leisure, I also like to play soccer and go fishing.

Plan for the Future: My goal is work as a Machine Learning Engineer.

Email: kark6037@vandals.uidaho.edu

=Additional Documentation=

Project Schedule

Gantt Chart

Meeting Minutes

Meeting Minutes

Presentations

Snapshot

Meeting Agendas

Agendas