On the (in)security of behavioral-based dynamic anti-malware techniques

• Main Author: Ersan, Erkan
• Other Authors: Malka, Lior
• Language: English; en
• Published: 2017
• Subjects: Malware; Malware Detection; Behavioral-based Dynamic Anti-Malware Techniques; Computer Security; Anti-malware; Antivirus; Cyber Attacks; Behavioral-based Detection; EMET; Hooking; Unhooking; Guard Pages; Bypassing Techniques; Evasion Techniques; Evasion; funneling; Control Flow Integrity; CFI; Web-based malware; ROP; Return-Oriented Programming; Shellcode; Payload; Windows; Anti-malware Evaluation; Antivirus Evaluation; Anti-malware Effectiveness; Antivirus Effectiveness
• Online Access: http://hdl.handle.net/1828/7935

The Internet has become the primary vector for the delivery of malicious code in cyber attacks, and malware has rapidly become a pervasive critical threat. Anti- malware products offer effective protection from malware threats for servers and endpoint devices using a variety of techniques. Advanced enterprise-level anti-malware products rely on state-of-art behavioral-based detection algorithms, in addition to traditional signature-based mechanisms. These dynamic detection techniques have been around for more than a decade and in response hackers have developed methods to evade them. However, currently known bypass methods require intensive manual labor. Moreover, this manual work has to be repeated whenever a parameter of the environment (such as the payload, operating system, Antivirus version, etc) changes, making these methods impractical. This may lead to the belief that dynamic techniques provide a good deterrence, and hence good protection. In this thesis we evaluate dynamic techniques. Specifically, we build tools to implement generic unhooking and funneling, and using these tools we show how dynamic techniques can be bypassed with considerably less effort than by fully manual methods. We also extend the repertoire of existing bypass methods and introduce a new malicious function call technique which exploits detection techniques that monitor a limited collection of critical system functions, as well as a method for bypassing guard-page protections. We demonstrate the effectiveness of all our techniques by conducting attacks against two enterprise antivirus products. Our results lead us to conclude that that dynamic techniques do not provide sufficient protection. === Graduate === 2018-02-07 === 0984 === [e-mail]