×

On the infeasibility of modeling polymorphic shellcode. Re-thinking the role of learning in intrusion detection systems. (English) Zbl 1470.68038

Summary: Current trends demonstrate an increasing use of polymorphism by attackers to disguise their exploits. The ability for malicious code to be easily, and automatically, transformed into semantically equivalent variants frustrates attempts to construct simple, easily verifiable representations for use in security sensors. In this paper, we present a quantitative analysis of the strengths and limitations of shellcode polymorphism, and describe the impact that these techniques have in the context of learning-based IDS systems. Our examination focuses on dual problems: shellcode encryption-based evasion methods and targeted “blending” attacks. Both techniques are currently being used in the wild, allowing real exploits to evade IDS sensors. This paper provides metrics to measure the effectiveness of modern polymorphic engines and provide insights into their designs. We describe methods to evade statistics-based IDS sensors and present suggestions on how to defend against them. Our experimental results illustrate that the challenge of modeling self-modifying shellcode by signature-based methods, and certain classes of statistical models, is likely an intractable problem.

MSC:

68M25 Computer security
68T05 Learning and adaptive systems in artificial intelligence
Full Text: DOI

References:

[1] Abadi, M., Budiu, M., Erlingsson, U., & Ligatti, J. (2005). Control-flow integrity: principles, implementations, and applications. In Proceedings of the ACM conference on computer and communications security (CCS).
[2] AlephOne (2001). Smashing the stack for fun and profit. Phrack, 7(49-14). · Zbl 1063.68045
[3] Anagnostakis, K. G., Sidiroglou, S., Akritidis, P., Xinidis, K., Markatos, E., & Keromytis, A. D. (2005). Detecting targeted attacks using shadow honeypots. In Proceedings of the 14th USENIX security symposium.
[4] Bania, P. (2009). Tapion polymorphic engine. http://pb.specialised.info/all/tapion/.
[5] Baratloo, A., Singh, N., & Tsai, T. (2000). Transparent run-time defense against stack smashing attacks. In Proceedings of the USENIX annual technical conference.
[6] Barrantes, E. G., Ackley, D. H., Forrest, S., Palmer, T. S., Stefanovic, D., & Zovi, D. D. (2003). Randomized instruction set emulation to distrupt binary code injection attacks. In Proceedings of the 10th ACM conference on computer and communications security (CCS).
[7] Bhatkar, S., DuVarney, D. C., & Sekar, R. (2003). Address obfuscation: an efficient approach to combat a broad range of memory error exploits. In Proceedings of the 12th USENIX security symposium (pp. 105-120).
[8] Biondi, P. (2006). Shellforge project. http://www.secdev.org/projects/shellforge/.
[9] Brumley, D., Newsome, J., Song, D., Wang, H., & Jha, S. (2006). Towards automatic generation of vulnerability-based signatures. In Proceedings of the IEEE symposium on security and privacy.
[10] CERT (2001). Code red I/II worm. http://www.cert.org/advisories/CA-2001-19.html.
[11] Chinchani, R., & Berg, E. V. D. (2005). A fast static analysis approach to detect exploit code inside network flows. In Proceedings of the 8th international symposium on recent advances in intrusion detection (RAID) (pp. 284-304).
[12] Costa, M., Crowcroft, J., Castro, M., & Rowstron, A. (2005). Vigilante: end-to-end containment of Internet worms. In Proceedings of the symposium on systems and operating systems principles (SOSP).
[13] Cowan, C., Pu, C., Maier, D., Hinton, H., Walpole, J., Bakke, P., Beattie, S., Grier, A., Wagle, P., & Zhang, Q. (1998). Stackguard: automatic adaptive detection and prevention of buffer-overflow attacks. In Proceedings of the USENIX security symposium.
[14] Crandall, J. R., Su, Z., Wu, S. F., & Chong, F. T. (2005a). On deriving unknown vulnerabilities from zero-day polymorphic and metamorphic worm exploits. In Proceedings of the 12th ACM conference on computer and communications security (CCS).
[15] Crandall, J. R., Wu, S. F., & Chong, F. T. (2005b). Experiences using minos as a tool for capturing and analyzing novel worms for unknown vulnerabilities. In Detection of intrusions and malware and vulnerability assessment (DIMVA).
[16] Cui, W., Peinado, M., Wang, H. J., & Locasto, M. E. (2007). ShieldGen: automated data patch generation for unknown vulnerabilities with informed probing. In Proceedings of the IEEE symposium on security and privacy.
[17] Detristan, T., Ulenspiegel, T., Malcom, Y., & von Underduk, M. S. (2003). Polymorphic shellcode engine using spectrum analysis. Phrack, 11(61-9).
[18] Etoh, J. (2000). GCC extension for protecting applications from stack-smashing attacks. http://www.trl.ibm.com/projects/security/ssp.
[19] Fogla, P., & Lee, W. (2006). Evading network anomaly detection systems: formal reasoning and practical techniques. In Proceedings of the 13th ACM conference on computer and communications security (CCS) (pp. 59-68). http://doi.acm.org/10.1145/1180405.1180414.
[20] Fogla, P., Sharif, M., Perdisci, R., Kolesnikov, O., & Lee, W. (2006). Polymorphic blending attacks. In Proceedings of the USENIX security conference.
[21] Foster, J. C., Osipov, V., Bhalla, N., & Heinen, N. (2005). Buffer overflow attacks: detect, exploit, prevent. Syngress.
[22] Joshi, A., King, S. T., Dunlap, G. W., & Chen, P. M. (2005). Detecting past and present intrusions through vulnerability-specific predicates. In Proceedings of the symposium on systems and operating systems principles (SOSP).
[23] K2 (2003). ADMmutate documentation. http://www.ktwo.ca/ADMmutate-0.8.4.tar.gz.
[24] Kc, G. S., Keromytis, A. D., & Prevelakis, V. (2003). Countering code-injection attacks with instruction-set randomization. In Proceedings of the 10th ACM conference on computer and communications security (CCS) (pp. 272-280).
[25] Kim, H. A., & Karp, B. (2004). Autograph: toward automated, distributed worm signature detection. In Proceedings of the USENIX security conference.
[26] Kiriansky, V., Bruening, D., & Amarasinghe, S. (2002). Secure execution via program shepherding. In Proceedings of the 11th USENIX security symposium. · Zbl 1063.68045
[27] Kolesnikov, A., & Lee, W. (2006). Advanced polymorphic worms: evading IDS by blending in with normal traffic. In Proceedings of the USENIX security conference.
[28] Kruegel, C., & Vigna, G. (2003). Anomaly detection of web-based attacks. In Proceedings of the 10th ACM conference on computer and communications security (CCS).
[29] Krugel, C., Kirda, E., Mutz, D., Robertson, W., & Vigna, G. (2005). Polymorphic worm detection using structural information of executables. In Proceedings of the 8th international symposium on recent advances in intrusion detection (RAID) (pp. 207-226).
[30] Liang, Z., & Sekar, R. (2005). Fast and automated generation of attack signatures: a basis for building self-protecting servers. In Proceedings of the 12th ACM conference on computer and communications security (CCS).
[31] Locasto, M. E., Wang, K., Keromytis, A. D., & Stolfo, S. J. (2005). FLIPS: hybrid adaptive intrusion prevention. In Proceedings of the 8th international symposium on recent advances in intrusion detection (RAID) (pp. 82-101).
[32] Metasploit Development Team (2006). Metasploit project. http://www.metasploit.com.
[33] Nethercote, N., & Seward, J. (2003). Valgrind: a program supervision framework. In Electronic notes in theoretical computer science (Vol. 89).
[34] Newsome, J., & Song, D. (2005). Dynamic taint analysis for automatic detection, analysis, and signature generation of exploits on commodity software. In Proceedings of the 12th symposium on network and distributed system security (NDSS).
[35] Newsome, J., Karp, B., & Song, D. (2005). Polygraph: automatically generating signatures for polymorphic worms. In Proceedings of the IEEE symposium on security and privacy.
[36] Obscou (2003). Building IA32 ‘Unicode-Proof’ shellcodes. Phrack, 11(61-11).
[37] Panda Labs (2007). MPack uncovered. http://pandalabs.pandasecurity.com/.
[38] Polychronakis, M., Anagnostakis, K. G., & Markatos, E. P. (2006). Network-level polymorhpic shellcode detection using emulation. In Detection of intrusions and malware and vulnerability assessment (DIMVA).
[39] Rix (2001). Writing IA-32 alphanumeric shellcodes. Phrack, 11(57-15).
[40] Russell, S., & Norvig, P. (2002). Artificial intelligence: a modern approach. New York: Prentice Hall. · Zbl 0835.68093
[41] SANS (2004a). IISMedia Exploit. http://www.sans.org/newsletters/cva/vol2_21.php.
[42] SANS (2004b). Santy worm. http://isc.sans.org/diary.html?date=2004-12-21.
[43] SANS (2004c). Webdav exploit. http://www.sans.org/resources/malwarefaq/webdav-exploit.php.
[44] Siddharth, S. (2005). Evading NIDS. http://www.securityfocus.com/infocus/1852.
[45] Sidiroglou, S., Giovanidis, G., & Keromytis, A. D. (2005). A dynamic mechanism for recovering from buffer overflow attacks. In Proceedings of the 8th information security conference (ISC) (pp. 1-15). · Zbl 1127.68397
[46] Singh, S., Estan, C., Varghese, G., & Savage, S. (2004). Automated worm fingerprinting. In Proceedings of symposium on operating systems design and implementation (OSDI).
[47] Snort Development Team (2009). Snort project. http://www.snort.org/.
[48] Song, Y., Locasto, M. E., Stavrou, A., Keromytis, A. D., & Stolfo, S. J. (2007). On the infeasibility of modeling polymorphic shellcode. In Proceedings of the ACM conference on computer and communications security (CCS). · Zbl 1470.68038
[49] Spinellis, D. (2003). Reliable identification of bounded-length viruses is NP-complete. IEEE Transactions on Information Theory, 49(1), 280-284. · Zbl 1063.68045 · doi:10.1109/TIT.2002.806137
[50] Tcpdump (2009). http://www.tcpdump.org.
[51] Toth, T., & Kruegel, C. (2002). Accurate buffer overflow detection via abstract payload execution. In Proceedings of the 5th international symposium on recent advances in intrusion detection (RAID) (pp. 274-291). · Zbl 1022.68554
[52] Wang, K., & Stolfo, S. J. (2004). Anomalous payload-based network intrusion detection. In Proceedings of the 7th international symposium on recent advances in intrusion detection (RAID) (pp. 203-222).
[53] Wang, H. J., Guo, C., Simon, D. R., & Zugenmaier, A. (2004). Shield: vulnerability-driven network filters for preventing known vulnerability exploits. In Proceedings of the ACM SIGCOMM conference (pp. 193-204).
[54] Wang, K., Cretu, G., & Stolfo, S. J. (2005). Anomalous payload-based worm detection and signature generation. In Proceedings of the 8th international symposium on recent advances in intrusion detection (RAID) (pp. 227-246).
[55] Wang, K., Parekh, J. J., & Stolfo, S. J. (2006a). Anagram: a content anomaly detector resistant to mimicry attack. In Proceedings of the 9th international symposium on recent advances in intrusion detection (RAID).
[56] Wang, X., Pan, C. C., Liu, P., & Zhu, S. (2006b). SigFree: a signature-free buffer overflow attack blocker. In Proceedings of the 15th USENIX security symposium (pp. 225-240).
[57] Yegneswaran, V., Giffin, J. T., Barford, P., & Jha, S. (2005). An architecture for generating semantics-aware signatures. In Proceedings of the 14th USENIX security symposium.
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.