Artificialis Machine Learning Contest
Hey there! Recently, I took part in a friendly contest conducted by a discord community called Artificialis (great place for anyone interested in ML, by the way). The challenge was to turn in a complete machine learning project, deployed and all, by the end of one month (There were a few catches, which I will be telling later). My experience in this can be defined with only one word — Thrilling. In this blog, I will be writing about my experience in this contest and the problems I faced while completing this project.
Teams
The participants of this contest divided ourselves into groups at the beginning of the contest. I was made the leader of my team, and was responsible for the team’s progress.
Project Idea
After almost two days of discussion, we finalized the project idea — An app that would take input from a webcam and identify the ASL (American Sign Language) alphabet shown. This was chosen as our topic as it would help bridge the communication gap between those who know and don’t know the sign language.
Modelling
After deciding the project idea, we started to build a suitable model for it. But there was a catch — we couldn’t use transfer learning. This would considerably slow down the training process and also reduce the accuracy of the model.
In spite of this, we decided to implement the MobileNetV2 architecture without the ImageNet weights (weights are the patterns learnt by the model during its previous training). After a few experiments with different datasets and different hyperparameters, we got a model with an accuracy of above 90% (which wasn’t overfitting). We decided to go ahead with this model and get to the deployment side of things.
Deployment
As tough as modelling may seem, this part was the one which gave our team a tough time.
In our team, we all had done some TensorFlow before, but no one had done much of OpenCV(a library that is needed to interact with a webcam). We had only 2 weeks left (and my school tests were going on) before the project ended, so we decided not “waste” our time watching OpenCV tutorials and just google up code as we need it.
Our first task was to make a program that would read the input from the webcam, feed it to the model and then print out the predictions on a local device. This was pretty easy and we managed to do it in a couple of days.
We then started to try and deploy the model on Streamlit, but the program that was working just fine till now crashed as soo as we used Streamlit. We realised that Streamlit doesn’t support webcam integration directly, and tried to use a library built upon Streamlit called streamlit-webrtc that enabled webcam.
For some reason, we never managed to get the code right and spent almost the whole week trying to figure out how to make streamlit-webrtc work.
With one week at hand, I decided that we needed to try another alternative for this and found a simple script that allowed us to output video frames to streamlit. This had a drawback though — as the program was dependent on OpenCV, the project could not be deployed the way I wanted it to be (as a virtual machine where the app is hosted will have no webcams).
As we had no time, we had to move on and we integrated the code with our model. When we ran a few tests, we realised that our model wasn’t predicting normally — it was predicting only one alphabet for any sign you show to the camera!
We had two days of time and we had a huge flaw in our code.
The Last Day
Now this was the day where most of the action took place. At the beginning of the last day, we still had a problematic model (or that was what I thought it was).
In order to see what was being fed to the model, we displayed the webcam input. Then we realised, that the problem was not in the model, but the images it had been predicting on. While converting the frame from BRG format to RGB format, the colour scheme of the frame would get messed up, and the model would predict on data that is completely outlandish.
We fixed this up and uploaded our files to GitHub in the nick of time and submitted it to the server admin Alessandro Lamberti.
Here is the link for the project’s GitHub repo: https://github.com/Chiraagkv/ASL
Conclusion
Writing this, I felt almost like I was writing an action movie, but I have to say, this was no lesser. I learned a lot along the way and found out a practical use for the knowledge I had. This was a wonderful opportunity indeed.
This is it for now. Bye!