Adaptive Cyber Defense Technique Based on Multiagent Reinforcement Learning Strategies

The static nature of cyber defense systems gives attackers a sufficient amount of time to explore and further exploit the vulnerabilities of information technology systems.In this paper,we investigate a problem where multiagent sys-tems sensing and acting in an environment contribute to adaptive cyber defense.We present a learning strategy that enables multiple agents to learn optimal poli-cies using multiagent reinforcement learning(MARL).Our proposed approach is inspired by the multiarmed bandits(MAB)learning technique for multiple agents to cooperate in decision making or to work independently.We study a MAB approach in which defenders visit a system multiple times in an alternating fash-ion to maximize their rewards and protect their system.We find that this game can be modeled from an individual player’s perspective as a restless MAB problem.We discover further results when the MAB takes the form of a pure birth process,such as a myopic optimal policy,as well as providing environments that offer the necessary incentives required for cooperation in multiplayer projects.

Millimeter-Wave Concurrent Beamforming:A Multi-Player Multi-Armed Bandit Approach

The communication in the Millimeter-wave(mmWave)band,i.e.,30~300 GHz,is characterized by short-range transmissions and the use of antenna beamforming(BF).Thus,multiple mmWave access points(APs)should be installed to fully cover a target environment with gigabits per second(Gbps)connectivity.However,inter-beam interference prevents maximizing the sum rates of the established concurrent links.In this paper,a reinforcement learning(RL)approach is proposed for enabling mmWave concurrent transmissions by finding out beam directions that maximize the long-term average sum rates of the concurrent links.Specifically,the problem is formulated as a multiplayer multiarmed bandit(MAB),where mmWave APs act as the players aiming to maximize their achievable rewards,i.e.,data rates,and the arms to play are the available beam directions.In this setup,a selfish concurrent multiplayer MAB strategy is advocated.Four different MAB algorithms,namely,ϵ-greedy,upper confidence bound(UCB),Thompson sampling(TS),and exponential weight algorithm for exploration and exploitation(EXP3)are examined by employing them in each AP to selfishly enhance its beam selection based only on its previous observations.After a few rounds of interactions,mmWave APs learn how to select concurrent beams that enhance the overall system performance.The proposed MAB based mmWave concurrent BF shows comparable performance to the optimal solution.

Design of Self‑Reconfigurable Multiarm Robot Mechanism Based on Deployable Kinematic Chains

As the structures of multiarm robots are serially arranged,the packaging and transportation of these robots are often inconvenient.The ability of these robots to operate objects must also be improved.Addressing this issue,this paper presents a type of multiarm robot that can be adequately folded into a designed area.The robot can achieve different operation modes by combining different arms and objects.First,deployable kinematic chains(DKCs)are designed,which can be folded into a designated area and be used as an arm structure in the multiarm robot mechanism.The strategy of a platform for storing DKCs is proposed.Based on the restrictions in the storage area and the characteristics of parallel mechanisms,a class of DKCs,called base assembly library,is obtained.Subsequently,an assembly method for the synthesis of the multiarm robot mechanism is proposed,which can be formed by the connection of a multiarm robot mechanism with an operation object based on a parallel mechanism structure.The formed parallel mechanism can achieve a reconfigurable characteristic when different DKCs connect to the operation object.Using this method,two types of multiarm robot mechanisms with four DKCs that can switch operation modes to perform different tasks through autonomous combination and release operation is proposed.The obtained mechanisms have observable advantages when compared with the traditional mechanisms,including optimizing the occupied volume during transportation and using parallel mechanism theory to analyze the switching of operation modes.

THE AMPHIPHILIC MULTIARM COPOLYMERS BASED ON HYPERBRANCHED POLYESTER AND LYSINE:SYNTHESIS AND SELF-ASSEMBLY

The amphiphilic multiarm copolymers were synthesized through the modification of commercially available hyperbranched polyesters（Boltorn H40） with N-ε-carbobenzoxy-L-Lysine N-carboxyanhydride（ZLys-NCA）.After being condensed with N-Boc-phenylalanine（Boc-^NPhe） and deprotected the Boc-groups in trifluoroacetic acid（TFA）,the original terminal hydroxyl groups were transformed into the amino groups and then initiated the ring-opening polymerization of ZLys-NCA.The hydrophilic poly（L-lysine） was grafted to the surface of Boltorn H40 successfully after the protecting benzyl groups were removed by the HBr solution in glacial acetic acid（33 wt%）.The resulting multiarm copolymers were characterized by the ^1H-NMR,GPC and FTIR.The arm length calculated by NMR and GPC analysis was about 3 and 13 lysine-units for H40-Phe-PLysl and H40-Phe-PLys2 respectively.Due to the amphiphilic molecular structure,they displayed ability to self-assemble into spherical micelles in aqueous solution with the average diameter in the range from 70 nm to 250 nm.The CMC of H40-Phe-PLysl and H40-Phe-PLys2 was 0.013 mg/mL and 0.028 mg/mL,respectively, indicating that H40-Phe-PLysl with shorter arm length is easier to self-assemble than H40-Phe-PLys2 with longer arm length.