Real-time object detection by combination of depth camera and VPU, implementation of high-speed transparentation and distance measurement

Katsuya Hyodo

Nagoya, Aichi

6 0

RaspberryPi3(Raspbian Stretch) or Ubuntu16.04/UbuntuMate + Neural Compute Stick(NCS/NCS2) + RealSense D435(or USB Camera or PiCamera) + MobileNet-SSD(MobileNetSSD) - MultiProcess - MultiThread - MultiRequest - MultiModel (VOC + FaceDetection) - MultiClustering ...learn more

Project status: Under Development

RealSense™, Internet of Things, Artificial Intelligence

Groups
Internet of Things, DeepLearning, Movidius™ Neural Compute Group

Intel Technologies
OpenVINO, Movidius NCS

Code Samples [1]Links [5]

Overview / Usage

Overview
Measure the distance to the object with RealSense D435 while performing object detection by MobileNet-SSD(MobileNetSSD) with RaspberryPi3 boosted with Intel Neural Compute Stick.
"USB Camera mode" can not measure the distance, but it operates at high speed.
And, This is support for MultiGraph and FaceDetection, MultiProcessing, Background transparentation.
And, This is support for simple clustering function. (To prevent thermal runaway)
Performance measurement result each number of sticks. (It is Detection rate. It is not a Playback rate.)
The best performance can be obtained with QVGA + NCS1 x5 Sticks or NCS2 x2 Sticks.
However, It is important to use a good quality USB camera.
　
Environment
- RaspberryPi3 + Raspbian Stretch (USB2.0 Port) or RaspberryPi3 + Ubuntu Mate or PC + Ubuntu16.04
- Intel RealSense D435 (Firmware Ver 5.10.6) or USB Camera or PiCamera
- Intel Neural Compute Stick v1/v2 x１piece or more
  - OpenCV 3.4.2 (NCSDK)
  - OpenCV 4.0.1-openvino (OpenVINO)
- VFPV3 or TBB (Intel Threading Building Blocks)
- Numpy
- Python3.5 (Only MultiStickSSDwithRealSense.py is multiprocessing enabled)
- NCSDK v2.08.01 (It does not work with NCSDK v1.　v1 version is here)
- OpenVINO R5 2018.5.445
- RealSenseSDK v2.16.5 (The latest version is unstable)
- HDMI Display
  　
Usage
https://github.com/PINTO0309/MobileNet-SSD-RealSense#firmware-update-with-windows-10-pc
https://github.com/PINTO0309/MobileNet-SSD-RealSense#1ncsdk-ver-not-compatible-with-ncs2
https://github.com/PINTO0309/MobileNet-SSD-RealSense#2openvino-ver-corresponds-to-ncs2
　
Execute the program
https://github.com/PINTO0309/MobileNet-SSD-RealSense#execute-the-program
　
Future tasks
- Docker compatible
- multiple models
- Individual identification

Methodology / Approach

RaspberryPi3(Raspbian Stretch) or Ubuntu16.04/UbuntuMate + Neural Compute Stick(NCS/NCS2) + RealSense D435(or USB Camera or PiCamera) + MobileNet-SSD(MobileNetSSD)