34 lines
4.3 KiB
Markdown
34 lines
4.3 KiB
Markdown
|
## Web Cam Project
|
||
|
|
|
||
|
|
The sign detection can be achieved in real time using the webcam that is attached to every device that is prevalent today. By accessing the webcam through the browser in the form of permissions by the user.
|
||
|
|
|
||
|
|
The webcam sign detection consists of various components that facilitate the prediction of the letter that the user presents in the stipulated area that is shown by the green outline box.
|
||
|
|
|
||
|
|
Personal Note: _use an image of the webcam feed here in the document explaining the diagram filling the details below_
|
||
|
|
|
||
|
|
- **Letter Prediction** :
|
||
|
|
- **Mask Feed** :
|
||
|
|
- **Bounding Box** :
|
||
|
|
|
||
|
|
### Inner Workings
|
||
|
|
|
||
|
|
Two urls are used to display and extract the mask layer from the webcam.
|
||
|
|
|
||
|
|
- **/video**: is the URL that is used to capture and stream the image response in the form of bytes that is updated for every frame. Using the _VideoCamera_ class that initiates, getting the frame, masking and returning the _StreamingHttpResponse_ that is updated using the _gencamera_ that yields the byte form for the image.
|
||
|
|
|
||
|
|
- **/video_feed**: This represents the default page of the webcam prediction that is hyperlinked from the homepage of the web application. It consists of the image tag that is used to display the **multipart-x/data** that is returned by the _/video_. The image source is refreshed with new frame as the feed is processed.
|
||
|
|
|
||
|
|
### Data flow
|
||
|
|
|
||
|
|
The User frames are captured by the _Camera_ class that initiates and processes the images. The Video is captured by the _VideoCapture_ method from the OpenCV library. It takes the index of the camera to use defaulted to zero. The object is initialized with the camera feed. Once, the view is restored for the user the web camera is restarted to pre-process, store the image, predict the character and finally, display the reconstructed video frame.
|
||
|
|
|
||
|
|
- **get_frame()**: this method is used to retrieve the frame from the video instance from the above. The frame receives the mask that is added to the camera feed in the right corner. The frame is returned in the form of bytes from the JPEG image format.
|
||
|
|
|
||
|
|
- **preprocess()**: the frame from the above function is given as the argument. Firstly, the image undergoes a flip to correct the mirror effect given by webcam. The Bounding box is defined that will aid the user in displaying the gestures in the box. The image inside the box is cropped out from the frame and is henceforth used to apply various filters in order to get the desired masking for the cropped image. The filters include the HSV filter. The filter has to be applied in the range of arrays to filter all the three (R, G, B) channels in the image Numpy array. They are _upper_bound_ and _lower_bound_ for the upper and lower bounds respectively. It consists of constants such as u_s,u_h,u_v for the upper. Similarly, the lower bound contains the l_h,l_v but it is to note that l_s is a dynamic variable that is changed to meeting the lighting and contrast of the surroundings to better aid the user in getting the best prediction in any condition.
|
||
|
|
|
||
|
|
- **lighting adjuster**: is the slider that denotes the dynamic variable of l*s. The slider is set to 55 units as default. Once the slider is changed, the value is sent to the controller in the form of POST data from the input to the URL. If the method is POST the value is updated with the global value and pass on the updated value to the view again. Following the same flow of logic but with the updated variable thereby giving a better prediction which can be adjusted based on the \_mask feed* on the corner to better find the letter.
|
||
|
|
|
||
|
|
- **predictor**: is given the prediction from the trained model as the only argument. The image that is previously stored by from the _get_frame()_ as _1.png_ for each and every frame masking image is loaded and converted into the Numpy array. The image array is fed into the Tensorflow machine learning model of the _.h5_ format. This is performed with the help of Keras package and the function of _load_model_ defined by the package. Based on the output from the Model the letter can be predicted from a case-switch style conditionals where the indexing of the classifier_resultant array is used to specify the letter of the alphabet is predicted which is returned.
|
||
|
|
|
||
|
|
The letter is added to the frame using the _putText_ function into the frame and finally, the frame is complete to be presented to the user.
|