Sample-Efficient Deep Reinforcement Learning with Directed Associative Graph

Reinforcement learning can be modeled as markov decision process mathematically.In consequence,the interaction samples as well as the connection relation between them are two main types of information for learning.However,most of recent works on deep reinforcement learning treat samples independently either in their own episode or between episodes.In this paper,in order to utilize more sample information,we propose another learning system based on directed associative graph(DAG).The DAG is built on all trajectories in real time,which includes the whole connection relation of all samples among all episodes.Through planning with directed edges on DAG,we offer another perspective to estimate stateaction pair,especially for the unknowns to deep neural network(DNN)as well as episodic memory(EM).Mixed loss function is generated by the three learning systems(DNN,EM and DAG)to improve the efficiency of the parameter update in the proposed algorithm.We show that our algorithm is significantly better than the state-of-the-art algorithm in performance and sample efficiency on testing environments.Furthermore,the convergence of our algorithm is proved in the appendix and its long-term performance as well as the effects of DAG are verified.

c-CSN:Single-cell RNA Sequencing Data Analysis by Conditional Cell-specific Network

t The rapid advancement of single-cell technologies has shed new light on the complex mechanisms of cellular heterogeneity.However,compared to bulk RNA sequencing(RNA-seq),single-cell RNA-seq(scRNA-seq)suffers from higher noise and lower coverage,which brings new computational difficulties.Based on statistical independence,cell-specific network(CSN)is able to quantify the overall associations between genes for each cell,yet suffering from a problem of overestimation related to indirect effects.To overcome this problem,we propose the c-CSN method,which can construct the conditional cell-specific network(CCSN)for each cell.c-CSN method can measure the direct associations between genes by eliminating the indirect associations.c-CSN can be used for cell clustering and dimension reduction on a network basis of single cells.Intuitively,each CCSN can be viewed as the transformation from less“reliable”gene expression to more“reliable”gene–gene associations in a cell.Based on CCSN,we further design network flow entropy(NFE)to estimate the differentiation potency of a single cell.A number of scRNA-seq datasets were used to demonstrate the advantages of our approach.1)One direct association network is generated for one cell.2)Most existing scRNA-seq methods designed for gene expression matrices are also applicable to c-CSN-transformed degree matrices.3)CCSN-based NFE helps resolving the direction of differentiation trajectories by quantifying the potency of each cell.c-CSN is publicly available at https://github.com/LinLi-0909/c-CSN.