基于API Hook的进程行为监控系统

Process behavior monitoring via API Hook

基于API Hook的进程行为监控系统,利用钩子技术和内存保护技术,实现了透明地对客户机进程API调用行为的安全监控.首先通过对客户虚拟机的API函数设置钩子,截获虚拟机中的进程API调用行为;接着利用内存保护技术,对客户机的钩子进行隐藏和保护,保证行为监控对客户虚拟机的透明性;然后利用虚拟机管理器的隔离性,将安全工具放在安全域中,一方面防止恶意进程检测并且攻击安全工具,另外一方面解决恶意租户利用虚拟机进行攻击的问题;最后在截获客户虚拟机API调用的基础上,利用语义重构技术,对客户虚拟机进程创建、文件操作、注册表操作等行为进行细粒度监控.测试结果表明:(1)监控系统可以有效的截获客户虚拟机进程API调用,结合语义重构技术,监控系统能够有效地对进程创建、文件操作、注册表操作等进程行为进行监控;(2)针对单个Hook点性能测试表明,监控系统截获API调用对系统性能的影响为4.8%;(3)在文件监控方面,基于API Hook的进程行为监控系统相对于现有截获系统调用的监控系统性能提高73%.

The process behavior monitoring system TAC based on API Hook realized the transparent monitoring of the API calling behavior of the client process,using hook and memory protection technology.First,TAC intercepted API calls of process in virtual machines by setting the hook of the API function of the client virtual machine.Then the memory protection technology was used to hide and protect the client＇s hook,so as to ensure the transparency of the behavior monitoring to the client virtual machine,and used the isolation of the virtual machine manager to put the security tools in the security domain.On the one hand,it prevented malicious process detection and attack security tools.On the other hand,it solved the problem that malicious tenants use virtual machines to attack.Finally,API called interception technology and semantic reconstruction technology providing fine-grained monitoring for process in guest VM,such as process creation,file operations and registry operations. The experimental results show that：（1） the monitoring system can effectively intercept the API call of the client virtual machine process.Combining the semantic reconfiguration technology,the monitoring system can effectively monitor process creation,file operation,registry operation and so on.（2）The performance of TAC is 4. 8% while it traces API call.（3） In file monitoring,the TAC based on API Hook improves the performance by 73% relative to monitoring system existing interception system calls.