一种抗语义攻击的虚拟化软件保护方法

DAS-VMP:A Virtual Machine-based Software Protection Method for Defending Against Semantic Attacks

随着计算机和网络的发展,软件核心算法面临着被逆向的威胁越来越大。虚拟机软件保护方法作为一种新型的软件保护方法,利用虚拟化技术保护软件的核心算法。因虚拟指令很难被理解,故其保护强度较高。但是,该方法仍无法抵御基于语义的攻击方法对虚拟机保护后的软件攻击,由此给软件安全带来了严重的威胁。针对现有的各类虚拟机软件保护方法无法应对目前恶意攻击者基于语义攻击的问题,提出了一种抗语义攻击的虚拟机软件保护方法即DAS-VMP。该方法分析了基于语义攻击的关键技术,依此研究出抵抗语义攻击的方法。从程序内部的数据流和执行流出发,通过设计数据流混淆引擎对虚拟机中虚拟解释器(Handlers)进行数据流混淆,使程序内部的数据流结构变得复杂多样,从而攻击者无法进行数据流的分析。隐藏虚拟机中的谓词信息,以抵抗攻击者的符号执行技术,同时将单一进程虚拟机设计为双进程虚拟机,控制软件运行过程中的执行流,使软件的执行过程更加难以被追踪,最终使经过保护后的软件呈现出一种复杂的数据流和执行流,从而阻止攻击者通过基于语义的攻击方法进行逆向分析。理论分析表明,DAS-VMP能够有效抵抗基于语义的攻击,与两款商业虚拟机保护系统的比较表明DAS-VMP对系统的性能开销较小。

With the development of computer and network technologies, the core codes of softwares are facing the risk of being reversed. To protect the core codes from being reserved, virtual machine-based （VM-based） software protection methods using virtualization technology were widely used because of its hard-understanding virtual instructions and high protection strength. However, traditional protection methods cannot resist semantic-based attacks, and lead to a serious threat to the software safety. Existing VM-based software protection methods cannot effectively cope with semantic-based attacks, which leverage the data flow and control flow information to obtain the core codes. To solve the problem, an effective protection system called DAS-VMP was proposed, with which semantics analysis based attacks can be defeated. In this paper, the key technologies of semantic-based attack were analyzed, and a method using complex obfuscation of the data flow and execution flow to defeat the sematic-based attacks was presented. In order to prevent attacker from analyzing the core algorithm, DAS-VMP obfuscated the data flow of handlers in virtual machine, and then more complex and diverse data flow structures were obtained. Moreover,to resist the attacker＇ s symbolic execution, the predicate information in the virtual machine was hidden by DAS-VMP. On the purpose of making the execution flow complex, a double process mechanism to control the execution flow during the execution process was introduced ,which made the software execution process more difficult to be tracked. Theoretical analysis and experimental results showed that DAS-VMP performs significantly well in resisting semantic-based attacks. Furthermore,it had less impact on performance overhead comparing to two existing commercial VM-based software protection methods.