Human-computer interactions for virtual reality

Human-computer interactions constitute an important subject for the development and popularization of information technologies,as they are not only an important frontier technology in computer science but also an important auxiliary technology in virtual reality(VR).In recent years,Chinese researchers have made significant advances in human-computer interactions.To systematically display China's latest advances in human-computer interactions and thus provide an impetus for the development of VR and other related fields,we have solicited articles for this special issue from experts in this area to participate in the review process.The following articles have been selected for publication in this special issue.

Defect detection of gear parts in virtual manufacturing

Gears play an important role in virtual manufacturing systems for digital twins;however,the image of gear tooth defects is difficult to acquire owing to its non-convex shape.In this study,a deep learning network is proposed to detect gear defects based on their point cloud representation.This approach mainly consists of three steps:(1)Various types of gear defects are classified into four cases(fracture,pitting,glue,and wear);A 3D gear dataset was constructed with 10000 instances following the aforementioned classification.(2)Gear-PCNet++introduces a novel Combinational Convolution Block,proposed based on the gear dataset for gear defect detection to effectively extract the local gear information and identify its complex topology;(3)Compared with other methods,experiments show that this method can achieve better recognition results for gear defects with higher efficiency and practicability.

Bottleneck Prediction Method Based on Improved Adaptive Network-based Fuzzy Inference System (ANFIS) in Semiconductor Manufacturing System

Semiconductor manufacturing (SM) system is one of the most complicated hybrid processes involved continuously variable dynamical systems and discrete event dynamical systems. The optimization and scheduling of semiconductor fabrication has long been a hot research direction in automation. Bottleneck is the key factor to a SM system, which seriously influences the throughput rate, cycle time, time-delivery rate, etc. Efficient prediction for the bottleneck of a SM system provides the best support for the consequent scheduling. Because categorical data (product types, releasing strategies) and numerical data (work in process, processing time, utilization rate, buffer length, etc.) have significant effect on bottleneck, an improved adaptive network-based fuzzy inference system (ANFIS) was adopted in this study to predict bottleneck since conventional neural network-based methods accommodate only numerical inputs. In this improved ANFIS, the contribution of categorical inputs to firing strength is reflected through a transformation matrix. In order to tackle high-dimensional inputs, reduce the number of fuzzy rules and obtain high prediction accuracy, a fuzzy c-means method combining binary tree linear division method was applied to identify the initial structure of fuzzy inference system. According to the experimental results, the main-bottleneck and sub-bottleneck of SM system can be predicted accurately with the proposed method.

On Self-adjustment of Social Conventions to Small Perturbations

We present a model for self-adjustment of social conventions to small perturbations, and investigate how perturbations can influence the convergence of social convention in different situations. The experimental results show that the sensitivity of social conventions is determined by not only the perturbations themselves but also the agent adjustment functions for the perturbations; and social conventions are more sensitive to the outlier agent number than to the strategy fluctuation magnitudes and localities of perturbations.

Gesture interaction in virtual reality

With the development of virtual reality(VR)and human-computer interaction technology,how to use natural and efficient interaction methods in the virtual environment has become a hot topic of research.Gesture is one of the most important communication methods of human beings,which can effectively express users'demands.In the past few decades,gesture-based interaction has made significant progress.This article focuses on the gesture interaction technology and discusses the definition and classification of gestures,input devices for gesture interaction,and gesture interaction recognition technology.The application of gesture interaction technology in virtual reality is studied,the existing problems in the current gesture interaction are summarized,and the future development is prospected.

Jacobi-Sobolev Orthogonal Polynomialsand Spectral Methods for Elliptic Boundary Value Problems

Generalized Jacobi polynomials with indexes α,β∈ R are introduced and some basic properties are established. As examples of applications,the second- and fourth-order elliptic boundary value problems with Dirichlet or Robin boundary conditions are considered,and the generalized Jacobi spectral schemes are proposed. For the diagonalization of discrete systems,the Jacobi-Sobolev orthogonal basis functions are constructed,which allow the exact solutions and the approximate solutions to be represented in the forms of infinite and truncated Jacobi series. Error estimates are obtained and numerical results are provided to illustrate the effectiveness and the spectral accuracy.

Typical curve with G1 constraints for curve completion

This paper presents a novel algorithm for planar G1 interpolation using typical curves with monotonic curvature.The G1 interpolation problem is converted into a system of nonlinear equations and sufficient conditions are provided to check whether there is a solution.The proposed algorithm was applied to a curve completion task.The main advantages of the proposed method are its simple construction,compatibility with NURBS,and monotonic curvature.

Influence of multi-modality on moving target selection in virtual reality

Background Owing to recent advances in virtual reality(VR)technologies,effective user interaction with dynamic content in 3D scenes has become a research hotspot.Moving target selection is a basic interactive task in which the user performance research in tasks is significant to user interface design in VR.Different from the existing static target selection studies,the moving target selection in VR is affected by the change in target speed,angle and size,and lack of research on some key factors.Methods This study designs an experimental scenario in which the users play badminton under the condition of VR.By adding seven kinds of modal clues such as vision,audio,haptics,and their combinations,five kinds of moving speed and four kinds of serving angles,and the effect of these factors on the performance and subjective feelings in moving target selection in VR,is studied.Results The results show that the moving speed of the shuttlecock has a significant impact on the user performance.The angle of service has a significant impact on hitting rate,but has no significant impact on the hitting distance.The acquisition of the user performance by the moving target is mainly influenced by vision under the combined modalities;adding additional modalities can improve user performance.Although the hitting distance of the target is increased in the trimodal condition,the hitting rate decreases.Conclusion This study analyses the results of user performance and subjective perception,and then provides suggestions on the combination of modality clues in different scenarios.

C1-Conforming Quadrilateral Spectral Element Method for Fourth-Order Equations

This paper is devoted to Professor Benyu Guo's open question on the C1-conforming quadrilateral spectral element method for fourth-order equations which has been endeavored for years. Starting with generalized Jacobi polynomials on the reference square, we construct the C1-conforming basis functions using the bilinear mapping from the reference square onto each quadrilateral element which fall into three categories-interior modes, edge modes, and vertex modes. In contrast to the triangular element, compulsively compensatory requirements on the global C1-continuity should be imposed for edge and vertex mode basis functions such that their normal derivatives on each common edge are reduced from rational functions to polynomials, which depend on only parameters of the common edge. It is amazing that the C1-conforming basis functions on each quadrilateral element contain polynomials in primitive variables, the completeness is then guaranteed and further confirmed by the numerical results on the Petrov-Galerkin spectral method for the non-homogeneous boundary value problem of fourth-order equations on an arbitrary quadrilateral. Finally, a C1-conforming quadrilateral spectral element method is proposed for the biharmonic eigenvalue problem, and numerical experiments demonstrate the effectiveness and efficiency of our spectral element method.

Automatic GUI Test by Using SIFT Matching

In software development process, the last step is usually the Graphic User In- terface（GUI） test, which is part of the final user experience （UE） test. Traditionally, there exist some GUI test tools in the market, such as Abbot Java GUI Test Framework and Pounder, in which testers pre-configure in the script all desired actions and instructions for the computer, nonetheless requiring too much of invariance of GUI environment; and they require reconfiguration in case of GUI changes, therefore still to be done mostly manually and hard for non-programmer testers to. Consequently, we proposed GUI tests by image recognition to automate the last process; we managed to innovate upon current algorithms such as SIFT and Random Fern, from which we develop the new algorithm scheme retrieving most efficient feature and dispelling inefficient part of each algorithm. Computers then apply the algorithm, to search for target patterns themselves and take subsequent actions such as manual mouse, keyboard and screen I/O automatically to test the GUI without any manual instructions. Test results showed that the proposed approach can accelerate GU! test largely compared to current benchmarks.

Pseudo complementary measurement for traditional single-pixel cameras

A traditional single-pixel camera needs a large number of measurements to reconstruct the object with compressive sensing computation.Compared with the 1/0 matrices in classical measurement,the 1/-1 matrices in the complementary measurement has better property for reconstruction computation and returns better reconstruction results.However,each row of the 1/-1 matrices needs two measurements with the traditional single-pixel camera which results into double measurements compared with the 1/0 matrices.In this paper,we consider the pseudo complementary measurement which only takes the same amount of measurements with the row number of some properly designed 1/0 matrix to compute the total luminous flux of the objective and derives the measurement data of the corresponding 1/-1 matrix in a mathematical way.The numerical simulation and experimental result show that the pseudo complementary measurement is an efficient tool for the traditional single-pixel camera imaging under low measurement rate,which can combine the advantages of the classical and complementary measurements and significantly improve the peak signal-to-noise ratio.

RADepthNet:Reflectance-aware monocular depth estimation

Background Monocular depth estimation aims to predict a dense depth map from a single RGB image,and has important applications in 3D reconstruction,automatic driving,and augmented reality.However,existing methods directly feed the original RGB image into the model to extract depth features without avoiding the interference of depth-irrelevant information on depth-estimation accuracy,which leads to inferior performance.Methods To remove the influence of depth-irrelevant information and improve the depth-prediction accuracy,we propose RADepthNet,a novel reflectance-guided network that fuses boundary features.Specifically,our method predicts depth maps using the following three steps:(1)Intrinsic Image Decomposition.We propose a reflectance extraction module consisting of an encoder-decoder structure to extract the depth-related reflectance.Through an ablation study,we demonstrate that the module can reduce the influence of illumination on depth estimation.(2)Boundary Detection.A boundary extraction module,consisting of an encoder,refinement block,and upsample block,was proposed to better predict the depth at object boundaries utilizing gradient constraints.(3)Depth Prediction Module.We use an encoder different from(2)to obtain depth features from the reflectance map and fuse boundary features to predict depth.In addition,we proposed FIFADataset,a depth-estimation dataset applied in soccer scenarios.Results Extensive experiments on a public dataset and our proposed FIFADataset show that our method achieves state-of-the-art performance.

DSD-MatchingNet:Deformable sparse-to-dense feature matching for learning accurate correspondences

Background Exploring correspondences across multiview images is the basis of various computer vision tasks.However,most existing methods have limited accuracy under challenging conditions.Method To learn more robust and accurate correspondences,we propose DSD-MatchingNet for local feature matching in this study.First,we develop a deformable feature extraction module to obtain multilevel feature maps,which harvest contextual information from dynamic receptive fields.The dynamic receptive fields provided by the deformable convolution network ensure that our method obtains dense and robust correspondence.Second,we utilize sparse-to-dense matching with symmetry of correspondence to implement accurate pixel-level matching,which enables our method to produce more accurate correspondences.Result Experiments show that our proposed DSD-MatchingNet achieves a better performance on the image matching benchmark,as well as on the visual localization benchmark.Specifically,our method achieved 91.3%mean matching accuracy on the HPatches dataset and 99.3%visual localization recalls on the Aachen Day-Night dataset.

Affine particle-in-cell method for two-phase liquid simulation

Background The interaction of gas and liquid can produce many interesting phenomena,such as bubbles rising from the bottom of the liquid.The simulation of two-phase fluids is a challenging topic in computer graphics.To animate the interaction of a gas and liquid,MultiFLIP samples the two types of particles,and a Euler grid is used to track the interface of the liquid and gas.However,MultiFLIP uses the fluid implicit particle(FLIP)method to interpolate the velocities of particles into the Euler grid,which suffer from additional noise and instability.Methods To solve the problem caused by fluid implicit particles(FLIP),we present a novel velocity transport technique for two individual particles based on the affine particle-in-cell(APIC)method.First,we design a weighed coupling method for interpolating the velocities of liquid and gas particles to the Euler grid such that we can apply the APIC method to the simulation of a two-phase fluid.Second,we introduce a narrowband method to our system because MultiFLIP is a time-consuming approach owing to the large number of particles.Results Experiments show that our method is well integrated with the APIC method and provides a visually credible two-phase fluid animation.Conclusions The proposed method can successfully handle the simulation of a two phase fluid.

The Life Cycle of Knowledge in Big Language Models:A Survey

Knowledge plays a critical role in artificial intelligence.Recently,the extensive success of pre-trained language models(PLMs)has raised significant attention about how knowledge can be acquired,maintained,updated and used by language models.Despite the enormous amount of related studies,there is still a lack of a unified view of how knowledge circulates within language models throughout the learning,tuning,and application processes,which may prevent us from further understanding the connections between current progress or realizing existing limitations.In this survey,we revisit PLMs as knowledge-based systems by dividing the life circle of knowledge in PLMs into five critical periods,and investigating how knowledge circulates when it is built,maintained and used.To this end,we systematically review existing studies of each period of the knowledge life cycle,summarize the main challenges and current limitations,and discuss future directions.

Hierarchical vectorization for facial images

The explosive growth of social media means portrait editing and retouching are in high demand.While portraits are commonly captured and stored as raster images,editing raster images is non-trivial and requires the user to be highly skilled.Aiming at developing intuitive and easy-to-use portrait editing tools,we propose a novel vectorization method that can automatically convert raster images into a 3-tier hierarchical representation.The base layer consists of a set of sparse diffusion curves(DCs)which characterize salient geometric features and low-frequency colors,providing a means for semantic color transfer and facial expression editing.The middle level encodes specular highlights and shadows as large,editable Poisson regions(PRs)and allows the user to directly adjust illumination by tuning the strength and changing the shapes of PRs.The top level contains two types of pixel-sized PRs for high-frequency residuals and fine details such as pimples and pigmentation.We train a deep generative model that can produce high-frequency residuals automatically.Thanks to the inherent meaning in vector primitives,editing portraits becomes easy and intuitive.In particular,our method supports color transfer,facial expression editing,highlight and shadow editing,and automatic retouching.To quantitatively evaluate the results,we extend the commonly used FLIP metric(which measures color and feature differences between two images)to consider illumination.The new metric,illumination-sensitive FLIP,can effectively capture salient changes in color transfer results,and is more consistent with human perception than FLIP and other quality measures for portrait images.We evaluate our method on the FFHQR dataset and show it to be effective for common portrait editing tasks,such as retouching,light editing,color transfer,and expression editing.

A buffer overflow detection and defense method based on RiSC-V instruction set extension

Buffer overflow poses a serious threat to the memory security of modern operating systems.It overwrites the con-tents of other memory areas by breaking through the buffer capacity limit,destroys the system execution environ-ment,and provides implementation space for various system attacks such as program control flow hijacking.That makes it a wide range of harms.A variety of security technologies have been proposed to deal with system security problems including buffer overflow.For example,No eXecute(NX for short)is a memory management technology commonly used in Harvard architecture.It can refuse the execution of code which residing in a specific memory,and can effectively suppress the abnormal impact of buffer overflow on control flow.Therefore,in recent years,it has also been used in the field of system security,deriving a series of solutions based on NX technology,such as ExecShield,DEP,StackGuard,etc.However,these security solutions often rely too much on the processor archi-tecture so that the protection coverage is insufficient and the accuracy is limited.Especially in the emerging system architecture field represented by RiSC-V,there is still a lack of effective solutions for buffer overflow vulnerabilities.With the continuous rapid development of the system architecture,it is urgent to develop defense methods that are applicable to different system application environments and oriented to all executable memory spaces to meet the needs of system security development.Therefore,we propose BOP,A new system memory security design method based on RISC-V extended instructions,to build a RISC-V buffer overflow detection and defense system and deal with the buffer overflow threat in RIsC-V.According to this method,NX technology can be combined with program control flow analysis,and Nx bit mechanism can be used to manage the executability of memory space,so as to achieve a more granular detection and defense of buffer overflow attacks that may occur in RISC-V system environment.In addition,The memory management and control function of BOP is not only very suitable for solving the security problems in the existing single architecture system,but also widely applicable to the combina-tion of multiple heterogeneous systems.

EFFICIENT SPECTRAL METHODS FOR EIGENVALUE PROBLEMS OF THE INTEGRAL FRACTIONAL LAPLACIAN ONABALLOFANYDIMENSION

An efficient spectral-Galerkin method for eigenvalue problems of the integral fractional Laplacian on a unit ball of any dimension is proposed in this paper.The symmetric positive definite linear system is retained explicitly which plays an important role in the numerical analysis.And a sharp estimate on the algebraic system's condition number is established which behaves as N4s with respect to the polynomial degree N,where 2s is the fractional derivative order.The regularity estimate of solutions to source problems of the fractional Laplacian in arbitrary dimensions is firstly investigated in weighted Sobolev spaces.Then the regularity of eigenfunctions of the fractional Laplacian eigenvalue problem is readily derived.Meanwhile,rigorous error estimates of the eigenvalues and eigenvectors are ob-tained.Numerical experiments are presented to demonstrate the accuracy and efficiency and to validate the theoretical results.

Modelling and Verification of Real-Time Publish and Subscribe Protocol Using UPPAAL and Simulink/Stateflow

Real-Time Publish and Subscribe (RTPS) protocol is a protocol for implementing message exchange over an unreliable transport in data distribution service (DDS). Formal modelling and verification of the protocol provide stronger guarantees of its correctness and efficiency than testing alone. In this paper, we build formal models for the RTPS protocol using UPPAAL and Simulink/Stateflow. Modelling using Simulink/Stateflow allows analyzing the protocol through simula-tion, as well as generate executable code. Modelling using UPPAAL allows us to verify properties of the model stated in TCTL (Timed Computation Tree Logic), as well as estimate its performance using statistical model checking. We further describe a procedure for translation from Stateflow to timed automata, where a subset of major features in Stateflow is supported, and prove the soundness statement that the Stateflow model is a refinement of the translated timed automata model. As a consequence, any property in a certain fragment of TCTL that we have verified for the timed automata model in UPPAAL is preserved for the original Stateflow model.

MemSC： A Scan-Resistant and Compact Cache Replacement Framework for Memory-Based Key-Value Cache Systems

Memory-based key-value cache systems, such as Memcached and Redis, have become indispensable components of data center infrastructures and have been used to cache performance-critical data to avoid expensive back-end database accesses. As the memory is usually not large enough to hold all the items, cache replacement must be performed to evict some cached items to make room for the newly coming items when there is no free space. Many real-world workloads target small items and have frequent bursts of scans （a scan is a sequence of one-time access requests）. The commonly used LRU policy does not work well under such workloads since LRU needs a large amount of metadata and tends to discard hot items with scans. Small decreases in hit ratio can result in large end-to-end losses in these systems. This paper presents MemSC, which is a scan-resistant and compact cache replacement framework for Memcached. MemSC assigns a multi-granularity reference flag for each item, which requires only a few bits （two bits are enough for general use） per item to support scanresistant cache replacement policies. To evaluate MemSC, we implement three representative cache replacement policies （MemSC-HM, MemSC-LH, and MemSC-LF） on MemSC and test them using various workloads. The experimental results show that MemSC outperforms prior techniques. Compared with the optimized LRU policy in Memcached, MemSC-LH reduces the cache miss ratio and the memory usage of the resulting system by up to 23% and 14% respectively.