Contributed by: Shankara Varshith.S
I’m working as a Machine Learning DA in one of the Big 4 tech companies in the Data management team of a leading AI. We primarily deal with big data preprocessing, data cleansing, and wrangling operations to perform Text and Sentiment analysis, evaluate device performance and provide measures to improve model accuracy and model performance.
As a part of my daily work routine, I have to review the errors we made for the past day and copy-paste this content into an excel workbook to be used by SMEs’ to analyze the data and find trends; it also acts as a repository for Associates to look back for reference.
This process, however, was being manually done every day, and each associate was given 30 mins to do the same. But the problem with manual data entry is that it’s time-consuming, and there’s always room for data tampering. Since the team uses a common workbook, the possible risk of losing data and difficulty with maintaining such a repository is high.
So, I wanted to automate this process. I approached my manager with this problem statement and proposed a solution to automate the process after getting the required permissions.
I did my research on the frameworks to achieve this, and I started building a DB model kind of setup using Python, which extracts information from the Error repository and feeds it back to the Excel workbook in an organized fashion.
I used Pandas and NumPy libraries, some user-defined functions, and frameworks to build the model. The model extracts individual D.A.’s information alphabetically, fits the raw data in organized rows and columns, calculates error scores for each record, and finally exports it as an Excel file.
On average, the model was able to save 21.55 mins of production time for each D.A. on a daily basis which is 41.03% more time-efficient than the manual process before that.
The project earned me an award for the Extra Mile in Rewards and Recognition ceremony that took place in Dec 2020
Want to learn such concepts? Join Great Learning’s PGP Data Science and Business Analytics Course and upskill today!
