Overview / Usage

It is a virtual assistant which can perform the different task without any physical intervention with keyboard and mouse. It will use both facial expression and hand gestures for performing the different task. We can control different activities by using hand gestures and facial expressions.

For system:
These are some activities:
1.Swapping different application
2. Creating shortcuts on the basis of gestures
3. Different activities like the opening, minimize, maximize, closing windows, select all, drag and drop.
4. Pause the video if not focusing on the screen
5. Eye stress detection using blinking pattern

Other than these we can also add some more custom activities.
For games:
Most of us want to play video games virtually because it looks so fascinating, but can’t play because these types of devices are so expensive and even cannot be afforded. Now we can pay some basic games (Atari games, racing games) by using hand gestures like

Methodology / Approach

Getting refined frames:
The camera will record all the motion and send it to our application. Now OpenCV will preprocess all the frames, now refined frames will feed to trained model. The model will predict different gestures.

Facial expressions
Face recognition for getting status, the status will provide the correctness of gesture. If the face is not in the front position then it will not consider that gesture. We can also detect eye stress on the basis of the blinking pattern.

Getting gestures
Hand gesture recognition for getting different instructions. After getting gesture it will go to the controller that will interact with OS and perform the operation corresponding to the gesture.

Technologies Used

1.Python libraries( Numpy, Scipy )
2.Optimized OpenCV by Intel
3.Keras
4.Tensorflow
5.OpenVINO Model optimizer

OpenVINO toolkit:
The Modal optimizer will convert the pre-trained tensorflow model into an optimized model based on quantization and batch normalization and create an intermediate file then it will directly use by interference engine. We can now integrate the engine into our application.
(*attached Workflow for deploying trained deep learning model using the model optimizer)

Other links

• Complete documentation