Clustering Malware via Measuring Similarity of Instruction Trace

• Main Authors: Chu, Ching-Feng, 朱慶峯
• Other Authors: Shieh, Shiuh-Pyng
• Format: Others
• Language: en_US
• Published: 2011
• Online Access: http://ndltd.ncl.edu.tw/handle/59376726252762726733

碩士 === 國立交通大學 === 資訊科學與工程研究所 === 100 === Although a large number of malicious programs are created every day, most of them mutate from existing ones. These mutant malware programs may seemingly appear differently, but actually act with similar behavior patterns. By clustering these malware programs into the same cluster, the malware analysis effort can be reduced significantly. In this paper, we propose a clustering approach to malware classification by comparing instruction trace similarity of binary programs being tested. We take advantage of dynamic analysis to trace malware instructions at runtime. Our method can discover malware in disguise by using techniques such as polymorphism or code injection. By tracing malware instructions, our scheme ensure that the detection mechanism cannot be circumvented or sabotaged by malicious API tampering. The taint technique we adopted can filter massive instructions created by normal system library as noise to the malware analysis. Collected instruction traces are then compared to measure their similarity so that the clustering can be performed. The results demonstrate that our system is able to cluster malware with similar codes, and can recognize new malware which is undetected by anti-virus tools.