一种分层强化学习的知识推理方法

Knowledge reasoning method based on hierarchical reinforcement learning

针对知识推理过程中,随着推理路径长度的增加,节点的动作空间急剧增长,使得推理难度不断提升的问题,提出一种分层强化学习的知识推理方法(knowledge reasoning method of hierarchical reinforcement learning,MutiAg-HRL),降低推理过程中的动作空间大小。MutiAg-HRL调用高级智能体对知识图谱中的关系进行粗略推理,通过计算下一步关系及给定查询关系之间的相似度,确定目标实体大致位置,依据高级智能体给出的关系,指导低级智能体进行细致推理,选择下一步动作;模型还构造交互奖励机制,对两个智能体的关系和动作选择及时给予奖励,防止模型出现奖励稀疏问题。为验证该方法的有效性,在FB15K-237和NELL-995数据集上进行实验,将实验结果与TransE、MINERVA、HRL等11种主流方法进行对比分析,MutiAg-HRL方法在链接预测任务上的hits@k平均提升了1.85%,MRR平均提升了2%。

In the process of knowledge inference,with the increase of the length of the inference path,the action space of the node increases sharply,which makes the inference difficulty continue to increase.This paper proposed a knowledge reasoning method of hierarchical reinforcement learning(MutiAg-HRL)to reduce the size of action space in the reasoning process.MutiAg-HRL invoked high-level agents to perform rough reasoning on the relationships in the knowledge graph,and determined the approximate location of the target entity by calculating the similarity between the next step relationship and the given query relationship.According to the relationship given by the high-level agent,the low-level agents were guided to conduct detailed reasoning and select the next action.The model also constructed an interactive reward mechanism to reward the relationship between the two agents and the choice of actions in time to prevent the problem of sparse reward in the model.To verify the effectiveness of the proposed method,it carried out experiments on FB15K-237 and NELL-995 datasets.The experimental results were compared with those of 11 mainstream methods such as TransE,MINERVA and HRL.The average value of the MutiAg-HRL method on the link prediction task hits@k is increased by 1.85%,MRR increases by an average of 2%.