Applying AI and Machine Learning concepts at work can help you power ahead your career. Read to learn more about Kiran Kumar’s journey with Great Learning’s PGP Artificial Intelligence and Machine Learning Course in his own words!
In System On Chip (SOC) Verification, it is a usual practice to run 1000s of tests in a regression suite on a regular basis. This is to ascertain the stability of the design over many regression runs with Random varying seeds. When the design is in a premature state, the failing logs of such regression results are often cumbersome to analyze. There can be many failing tests, and in each test, there can be many failing points within the test. Engineers have to look at 100s of failing test logs and analyze each failure within each log, which again could be many. In reality, though, there would be only a handful of common design errors that are causing all the failures. I set out to find a way to use some ML algorithm to analyze all the failures across all the test cases and generate a unique error summary that will ease the engineer’s job to analyze the issue.
The error messages in the test logs are usually similar in the text (not the same, though). But, there will be some differences in context – For example, the address or data will be different for each message. So, I decided to use Cosine Similarity to find messages that are similar across all failing test logs. This way, I could derive the unique failures across the regression. Over this, I also used K-Means Clustering to group all test cases with similar errors so that designers can debug only one test log in a set of test cases with similar failure.
For Cosine Similarity:
- I collected all the error logs and formed a dictionary of unique words.
- Then for each error log in each test case, formed the word frequency vector.
- Once frequency vectors are formed, run a cosine similarity across all pairs of messages and form a Cosine similarity Score Matrix.
- On the above Cosine similarity score Matrix, did another Cosine Similarity to eliminate all the messages which correlate with another message to the same degree.
- Finally had a handful of messages with Cosine similarity less than say 99%
This handful of messages are the unique Error signatures to be debugged.
As a next step, for each test case, I created a ‘Test Case-Unique Error’ Message Matrix. For each Test Case, it will say which error message is present in it. I applied K-means Clustering on this to get a picture of which test cases have similar error signatures. Finally, recommend only the handful of test cases that cover all unique errors in that regression run.
With the above Solution, we could reduce the problem space from 1000s of test cases * 10s of errors to a handful test cases * handful of errors. As an example, from a 1200 test case log, each log having approximately ten errors, we found:
- 17 Unique errors
- 20 Testcase to debug
Problem space reduction is from ~12000 to 340 (17*20). This is a significant reduction in problem space. Engineers spent only a few hours debugging all the issues compared to a few days in the earlier method. Two key advantages:
- Time spent on debug analysis is significantly reduced (Nearly ten times less – 2-3 working days to 2-3hrs)
- We could run a regression every day since analysis completes the very next day. Previously we would run a regression once in two or three days since the analysis was taking time.
Many teams across our group benefited from this solution. I did this in my free time, inspired by the Recommendation system Module. The solution was quite simple, thanks to these algorithms. The impact was quite high.
