Handler混淆增强的虚拟机保护方法

Virtual machine protection based on Handler obfuscation enhancement

按照一定顺序执行虚拟指令处理函数(Handler)可完成程序关键代码的保护,其为软件逆向分析者攻击的重点对象。针对"动态提取,静态分析"的Handler攻击方法,提出一种基于Handler混淆增强的虚拟机保护方法。运用等价指令替换规则生成多种等价Handler序列,对所有Handler进行变长切分和随机乱序,通过构建跳转表对乱序序列进行重组,构建随机地址数组对Handler调度地址表和执行跳转表进行隐藏。实验和分析表明:多样化Handler生成、切分和乱序增加了动态提取和分析的难度,Handler地址表和跳转表的隐藏增加了抵御静态逆向分析的难度,从而提升了虚拟机保护强度。

The combination of Handlers in virtual machine can protect key codes in the program, and these Handlers arethe main target for software reverse analysts to attack. Aiming at the reduction method for dynamic extraction and staticanalysis of Handlers, virtual machine protection method based on Handler obfuscation is proposed. Firstly, various equivalentinstruction rules are used to generate different equivalence Handlers, and then all Handlers are divided and disorderedby random scrambling algorithm, and they are restructured by constructing jump table, finally random address array isused to hide the data of Handler scheduling address table and execution jump table. Experiments and analysis show thatthe generation, segmentation and disorder of diverse Handlers increase the difficulty of dynamic extraction and analysis,the Handler address table and a jump table hidden enhances the difficulty of static reverse analysis.