Human Interaction Recognition in Surveillance Videos Using Hybrid Deep Learning and Machine Learning Models

Human Interaction Recognition(HIR)was one of the challenging issues in computer vision research due to the involvement of multiple individuals and their mutual interactions within video frames generated from their movements.HIR requires more sophisticated analysis than Human Action Recognition(HAR)since HAR focuses solely on individual activities like walking or running,while HIR involves the interactions between people.This research aims to develop a robust system for recognizing five common human interactions,such as hugging,kicking,pushing,pointing,and no interaction,from video sequences using multiple cameras.In this study,a hybrid Deep Learning(DL)and Machine Learning(ML)model was employed to improve classification accuracy and generalizability.The dataset was collected in an indoor environment with four-channel cameras capturing the five types of interactions among 13 participants.The data was processed using a DL model with a fine-tuned ResNet(Residual Networks)architecture based on 2D Convolutional Neural Network(CNN)layers for feature extraction.Subsequently,machine learning models were trained and utilized for interaction classification using six commonly used ML algorithms,including SVM,KNN,RF,DT,NB,and XGBoost.The results demonstrate a high accuracy of 95.45%in classifying human interactions.The hybrid approach enabled effective learning,resulting in highly accurate performance across different interaction types.Future work will explore more complex scenarios involving multiple individuals based on the application of this architecture.

A Novel Human Interaction Framework Using Quadratic Discriminant Analysis with HMM

Human-human interaction recognition is crucial in computer vision fields like surveillance,human-computer interaction,and social robotics.It enhances systems’ability to interpret and respond to human behavior precisely.This research focuses on recognizing human interaction behaviors using a static image,which is challenging due to the complexity of diverse actions.The overall purpose of this study is to develop a robust and accurate system for human interaction recognition.This research presents a novel image-based human interaction recognition method using a Hidden Markov Model(HMM).The technique employs hue,saturation,and intensity(HSI)color transformation to enhance colors in video frames,making them more vibrant and visually appealing,especially in low-contrast or washed-out scenes.Gaussian filters reduce noise and smooth imperfections followed by silhouette extraction using a statistical method.Feature extraction uses the features from Accelerated Segment Test(FAST),Oriented FAST,and Rotated BRIEF(ORB)techniques.The application of Quadratic Discriminant Analysis(QDA)for feature fusion and discrimination enables high-dimensional data to be effectively analyzed,thus further enhancing the classification process.It ensures that the final features loaded into the HMM classifier accurately represent the relevant human activities.The impressive accuracy rates of 93%and 94.6%achieved in the BIT-Interaction and UT-Interaction datasets respectively,highlight the success and reliability of the proposed technique.The proposed approach addresses challenges in various domains by focusing on frame improvement,silhouette and feature extraction,feature fusion,and HMM classification.This enhances data quality,accuracy,adaptability,reliability,and reduction of errors.

Exploiting Human Pose and Scene Information for Interaction Detection

Identifying human actions and interactions finds its use in manyareas, such as security, surveillance, assisted living, patient monitoring, rehabilitation,sports, and e-learning. This wide range of applications has attractedmany researchers to this field. Inspired by the existing recognition systems,this paper proposes a new and efficient human-object interaction recognition(HOIR) model which is based on modeling human pose and scene featureinformation. There are different aspects involved in an interaction, includingthe humans, the objects, the various body parts of the human, and the backgroundscene. Themain objectives of this research include critically examiningthe importance of all these elements in determining the interaction, estimatinghuman pose through image foresting transform (IFT), and detecting the performedinteractions based on an optimizedmulti-feature vector. The proposedmethodology has six main phases. The first phase involves preprocessing theimages. During preprocessing stages, the videos are converted into imageframes. Then their contrast is adjusted, and noise is removed. In the secondphase, the human-object pair is detected and extracted from each image frame.The third phase involves the identification of key body parts of the detectedhumans using IFT. The fourth phase relates to three different kinds of featureextraction techniques. Then these features are combined and optimized duringthe fifth phase. The optimized vector is used to classify the interactions in thelast phase. TheMSRDaily Activity 3D dataset has been used to test this modeland to prove its efficiency. The proposed system obtains an average accuracyof 91.7% on this dataset.

Inferring object properties from human interaction and transferring them to new motions

Humans regularly interact with their surrounding objects.Such interactions often result in strongly correlated motions between humans and the interacting objects.We thus ask:"Is it possible to infer object properties from skeletal motion alone,even without seeing the interacting object itself?"In this paper,we present a fine-grained action recognition method that learns to infer such latent object properties from human interaction motion alone.This inference allows us to disentangle the motion from the object property and transfer object properties to a given motion.We collected a large number of videos and 3 D skeletal motions of performing actors using an inertial motion capture device.We analyzed similar actions and learned subtle differences between them to reveal latent properties of the interacting objects.In particular,we learned to identify the interacting object,by estimating its weight,or its spillability.Our results clearly demonstrate that motions and interacting objects are highly correlated and that related object latent properties can be inferred from 3 D skeleton sequences alone,leading to new synthesis possibilities for motions involving human interaction.Our dataset is available at http://vcc.szu.edu.cn/research/2020/IT.html.

Human Pose Estimation and Object Interaction for Sports Behaviour

In the new era of technology,daily human activities are becoming more challenging in terms of monitoring complex scenes and backgrounds.To understand the scenes and activities from human life logs,human-object interaction(HOI)is important in terms of visual relationship detection and human pose estimation.Activities understanding and interaction recognition between human and object along with the pose estimation and interaction modeling have been explained.Some existing algorithms and feature extraction procedures are complicated including accurate detection of rare human postures,occluded regions,and unsatisfactory detection of objects,especially small-sized objects.The existing HOI detection techniques are instancecentric(object-based)where interaction is predicted between all the pairs.Such estimation depends on appearance features and spatial information.Therefore,we propose a novel approach to demonstrate that the appearance features alone are not sufficient to predict the HOI.Furthermore,we detect the human body parts by using the Gaussian Matric Model(GMM)followed by object detection using YOLO.We predict the interaction points which directly classify the interaction and pair them with densely predicted HOI vectors by using the interaction algorithm.The interactions are linked with the human and object to predict the actions.The experiments have been performed on two benchmark HOI datasets demonstrating the proposed approach.

RFHUI:an RFID based human-unmanned aerial vehicle interaction system in an indoor environment

In this paper,we present an RFID based human and Unmanned Aerial Vehicle(UAV)Interaction system,termed RFHUI,to provide an intuitive and easy-to-operate method to navigate a UAV in an indoor environment.It relies on the passive Radio-Frequency IDentification(RFID)technology to precisely track the pose of a handheld controller,and then transfer the pose information to navigate the UAV.A prototype of the handheld controller is created by attaching three or more Ultra High Frequency(UHF)RFID tags to a board.A Commercial Off-The-Shelf(COTS)RFID reader with multiple antennas is deployed to collect the observations of the tags.First,the precise positions of all the tags can be obtained by our proposed method,which leverages a Bayesian filter and Channel State Information(CSI)phase measurements collected from the RFID reader.Second,we introduce a Singular Value Decomposition(SVD)based approach to obtain a 6-DoF(Degrees of Freedom)pose of the controller from estimated positions of the tags.Furthermore,the pose of the controller can be precisely tracked in a real-time manner,while the user moves the controller.Finally,control commands will be generated from the controller's pose and sent to the UAV for navigation.The performance of the RFHUI is evaluated by several experiments.The results show that it provides precise poses with 0.045m mean error in position and 2.5∘mean error in orientation for the controller,and enables the controller to precisely and intuitively navigate the UAV in an indoor environment.

An Architecture Supporting Intelligent Mobile Healthcare Using Human-Computer Interaction HCI Principles

Recent advancements in the Internet of Things IoT and cloud computing have paved the way for mobile Healthcare(mHealthcare)services.A patient within the hospital is monitored by several devices.Moreover,upon leaving the hospital,the patient can be remotely monitored whether directly using body wearable sensors or using a smartphone equipped with sensors to monitor different user-health parameters.This raises potential challenges for intelligent monitoring of patient's health.In this paper,an improved architecture for smart mHealthcare is proposed that is supported by HCI design principles.The HCI also provides the support for the User-Centric Design(UCD)for smart mHealthcare models.Furthermore,the HCI along with IoT's(Internet of Things)5-layered architecture has the potential of improving User Experience(UX)in mHealthcare design and help saving lives.The intelligent mHealthcare system is supported by the IoT sensing and communication layers and health care providers are supported by the application layer for the medical,behavioral,and health-related information.Health care providers and users are further supported by an intelligent layer performing critical situation assessment and performing a multi-modal communication using an intelligent assistant.The HCI design focuses on the ease-of-use,including user experience and safety,alarms,and error-resistant displays of the end-user,and improves user's experience and user satisfaction.

Fire Hawk Optimizer with Deep Learning Enabled Human Activity Recognition

Human-Computer Interaction(HCI)is a sub-area within computer science focused on the study of the communication between people(users)and computers and the evaluation,implementation,and design of user interfaces for computer systems.HCI has accomplished effective incorporation of the human factors and software engineering of computing systems through the methods and concepts of cognitive science.Usability is an aspect of HCI dedicated to guar-anteeing that human–computer communication is,amongst other things,efficient,effective,and sustaining for the user.Simultaneously,Human activity recognition(HAR)aim is to identify actions from a sequence of observations on the activities of subjects and the environmental conditions.The vision-based HAR study is the basis of several applications involving health care,HCI,and video surveillance.This article develops a Fire Hawk Optimizer with Deep Learning Enabled Activ-ity Recognition(FHODL-AR)on HCI driven usability.In the presented FHODL-AR technique,the input images are investigated for the identification of different human activities.For feature extraction,a modified SqueezeNet model is intro-duced by the inclusion of few bypass connections to the SqueezeNet among Fire modules.Besides,the FHO algorithm is utilized as a hyperparameter optimization algorithm,which in turn boosts the classification performance.To detect and cate-gorize different kinds of activities,probabilistic neural network(PNN)classifier is applied.The experimental validation of the FHODL-AR technique is tested using benchmark datasets,and the outcomes reported the improvements of the FHODL-AR technique over other recent approaches.

Implementation of Level-3 Autonomous Patient-Specific Quality Assurance with Automated Human Interactive Devices

Purpose: Patient-specific quality assurance (PSQA) requires manual operation of different workstations, which is time-consuming and error-prone. Therefore, developing automated solutions to improve efficiency and accuracy is a priority. The purpose of this study was to develop a general software interface with scripting on a human interactive device (HID) for improving the efficiency and accuracy of manual quality assurance (QA) procedures. Methods: As an initial application, we aimed to automate our PSQA workflow that involves Varian Eclipse treatment planning system, Elekta MOSAIQ oncology information system and PTW Verisoft application. A general platform, the AutoFrame interface with two imbedded subsystems—the AutoFlow and the PyFlow, was developed with a scripting language for automating human operations of aforementioned systems. The interface included three functional modules: GUI module, UDF script interpreter and TCP/IP communication module. All workstations in the PSQA process were connected, and most manual operations were automated by AutoFrame sequentially or in parallel. Results: More than 20 PSQA tasks were performed both manually and using the developed AutoFrame interface. On average, 175 (±12) manual operations of the PSQA procedure were eliminated and performed by the automated process. The time to complete a PSQA task was 8.23 (±0.78) minutes for the automated workflow, in comparison to 13.91 (±3.01) minutes needed for manual operations. Conclusion: We have developed the AutoFrame interface framework that successfully automated our PSQA procedure, and significantly reduced the time, human (control/clicking/typing) errors, and operators’ stress. Future work will focus on improving the system’s flexibility and stability and extending its operations to other QA procedures.

Context-Aware Augmented Reality Using Human-Computer Interaction Models

Augmented Reality is a technique that allows users to overlap digital information with their physical world.The Augmented Reality(AR)displays have an exceptional characteristic from the Human–Computer Interaction(HCI)perspective.Due to its increasing popularity and application in diverse domains,increasing user-friendliness and AR usage are critical.Context-aware is one approach since an AR application can adapt to the user,environment,needs and enhance ergonomic principles and functionality.This paper proposes the Intelligent Contextaware Augmented Reality Model(ICAARM)for Human–Computer Interaction systems.This study explores and reduces interaction uncertainty by semantically modeling user-specific interaction with context,allowing personalised interaction.Sensory information is captured from an AR device to understand user interactions and context.These depictions carry semantics to Augmented Reality applications about the user’s intention to interact with a specific device affordance.Thus,this study describes personalised gesture interaction in VR/AR applications for immersive/intelligent environments.

Home-based Detection and Prediction of Diabetic Foot Ulcers at Early Stage Using Sensor Technology and Supervised Learning

For years,foot ulcers linked with diabetes mellitus and neuropathy have significantly impacted diabetic patients’ health-related quality of life(HRQoL). Diabetes foot ulcers impact15% of all diabetic patients at some point in their lives. The facilities and resources used for DFU detection and treatment are only available at hospitals and clinics,which results in the unavailability of feasible and timely detection at an early stage. This necessitates the development of an at-home DFU detection system that enables timely predictions and seamless communication with users,thereby preventing amputations due to neglect and severity. This paper proposes a feasible system consisting of three major modules:an IoT device that works to sense foot nodes to send vibrations onto a foot sole,a machine learning model based on supervised learning which predicts the level of severity of the DFU using four different classification techniques including XGBoost,K-SVM,Random Forest,and Decision tree,and a mobile application that acts as an interface between the sensors and the patient. Based on the severity levels,necessary steps for prevention,treatment,and medications are recommended via the application.

APPLICATION OF HUMAN COGNITIVE MODEL FOR TIME DEPENDENT OPERATOR BEHAVIOR IN CHINESE NPP

This paper discusses some issues on human reliability model of time dependent human behavior. Some results of the crew reliability experiment on Tsinghua training simulator in China are given, Meanwhile, a case of calculation for human error probability during anticipated transient without scram (ATWS) based on the data drew from the recent experiment is offered.

Systematic Review on Social Engineering: Hacking by Manipulating Humans

Despite the availability of advanced security software and hardware mechanisms available, still, there has been a breach in the defence system of an organization or individual. Social engineering mostly targets the weakest link in the security system i.e. “Humans” for gaining access to sensitive information by manipulating human psychology. Social engineering attacks are arduous to defend as such attacks are not easily detected by available security software or hardware. This article surveys recent studies on social engineering attacks with discussion on the social engineering phases and categorizing the various attacks into two groups. The main aim of this survey is to examine the various social engineering attacks on individuals and countermeasures against social engineering attacks are also discussed.

A holistic framework for facilitating environmental and human health

In the Anthropocene,health is necessary to achieve global sustainable development.This is a challenge because health issues are complex and span from humans to ecosystems and the environment through dynamic interac-tions across scales.We find that the health issues have been mainly addressed by disciplinary endeavors which unfortunately will not result in panoramic theories or effective solutions.We recommend focusing on the intri-cate interactions between humans,ecosystems and the environment for developing common theoretical under-standings and practical solutions for safeguarding planetary health,with human health as the key indicator and endpoint.To facilitate this paradigm shift,a holistic framework is formulated that incorporates disturbances from inner Earth and our solar system,and accommodates interactions between humans,ecosystems and the environ-ment in a nested hierarchy.An integrative and transdisciplinary health science is advocated along with holistic thinking to resolve our current health challenges and to achieve the health-related sustainable development goals.

VR and AR in human performance research―An NUS experience

With the mindset of constant improvement in efficiency and safety in the workspace and training in Singapore,there is a need to explore varying technologies and their capabilities to fulfil this need.The ability of Virtual Reality(VR)and Augmented Reality(AR)to create an immersive experience of tying the virtual and physical environments coupled with information filtering capabilities brings a possibility of introducing this technology into the training process and workspace.This paper surveys current research trends,findings and limitation of VR and AR in its effect on human performance,specifically in Singapore,and our experience in the National University of Singapore(NUS).

COMTIS:Customizable touchless interaction system for large screen visualization

Background Large screen visualization sys tems have been widely utilized in many industries.Such systems can help illustrate the working states of different production systems.However,efficient interaction with such systems is still a focus of related research.Methods In this paper,we propose a touchless interaction system based on RGB-D camera using a novel bone-length constraining method.The proposed method optimizes the joint data collected from RGB-D cameras with more accurate and more stable results on very noisy data.The user can customize the system by modifying the finite-state machine in the system and reuse the gestures in multiple scenarios,reducing the number of gestures that need to be designed and memorized.Results/Conclusions The authors tested the system in two cases.In the first case,we illustrated a process in which we improved the gesture designs on our system and tested the system through user study.In the second case,we utilized the system in the mining industry and conducted a user study,where users say that they think the system is easy to use.

Mobile Interaction Capturing System in Real Environments

Mobile applications are being used in a great range of fields and application areas. As a result, many research fields have focused on the study and improvement of such devices. The current Smartphones are the best example of the research and the evolution of these technologies. Moreover, the software design and development is progressively more focused on the user; finding and developing new mobile interaction models. In order to do so, knowing what kind of problems the users could have is vital to enhance a bad interaction design. Unfortunately, a good software quality evaluation takes more time than the companies can invest. The contribution revealed in this work is a new approach to quality testing methodology focused on mobile interactions and their context in use where external capturing tools, such as cameras, are suppressed and the evaluation environments are the same as the user will use the application. By this approach, the interactions can be captured without changing the context and consequently, the data will be more accurate, enabling the evaluation of the quality-in-use in real environments.

A Robust Approach for Detection and Classification of KOA Based on BILSTM Network

A considerable portion of the population now experiences osteoarthritis of the knee,spine,and hip due to lifestyle changes.Therefore,early treatment,recognition and prevention are essential to reduce damage;nevertheless,this time-consuming activity necessitates a variety of tests and in-depth analysis by physicians.To overcome the existing challenges in the early detection of Knee Osteoarthritis(KOA),an effective automated technique,prompt recognition,and correct categorization are required.This work suggests a method based on an improved deep learning algorithm that makes use of data from the knee images after segmentation to detect KOA and its severity using the Kellgren-Lawrence(KL) classification schemes,such as Class-I,Class-II,Class-III,and Class-IV.Utilizing ResNet to segregate knee pictures,we first collected features from these images before using the Bidirectional Long Short-Term Memory(BiLSTM)architecture to classify them.Given that the technique is a pre-trained network and doesn’t require a large training set,the Mendeley VI dataset has been utilized for the training of the proposed model.To evaluate the effectiveness of the suggested model,cross-validation has also been employed using the Osteoarthritis Initiative(OAI)dataset.Furthermore,our suggested technique is more resilient,which overcomes the challenge of imbalanced training data due to the hybrid architecture of our proposed model.The suggested algorithm is a cuttingedge and successful method for documenting the successful application of the timely identification and severity categorization of KOA.The algorithm showed a cross-validation accuracy of 78.57%and a testing accuracy of 84.09%.Numerous tests have been conducted to show that our suggested algorithm is more reliable and capable than the state-of-the-art at identifying and categorizing KOA disease.

Eye-shaped keyboard for dual-hand text entry in virtual reality

We propose an eye-shaped keyboard for high-speed text entry in virtual reality (VR), having the shape of dual eyes with characters arranged along the curved eyelids, which ensures low density and short spacing of the keys. The eye-shaped keyboard references the QWERTY key sequence, allowing the users to benefit from their experience using the QWERTY keyboard. The user interacts with an eye-shaped keyboard using rays controlled with both the hands. A character can be entered in one step by moving the rays from the inner eye regions to regions of the characters. A high-speed auto-complete system was designed for the eye-shaped keyboard. We conducted a pilot study to determine the optimal parameters, and a user study to compare our eye-shaped keyboard with the QWERTY and circular keyboards. For beginners, the eye-shaped keyboard performed significantly more efficiently and accurately with less task load and hand movement than the circular keyboard. Compared with the QWERTY keyboard, the eye-shaped keyboard is more accurate and significantly reduces hand translation while maintaining similar efficiency. Finally, to evaluate the potential of eye-shaped keyboards, we conducted another user study. In this study, the participants were asked to type continuously for three days using the proposed eye-shaped keyboard, with two sessions per day. In each session, participants were asked to type for 20min, and then their typing performance was tested. The eye-shaped keyboard was proven to be efficient and promising, with an average speed of 19.89 words per minute (WPM) and mean uncorrected error rate of 1.939%. The maximum speed reached 24.97 WPM after six sessions and continued to increase.

Towards Human‑Vehicle Interaction: Driving Risk Analysis Under Different Driver Vigilance States and Driving Risk Detection Method

The driver's behavior plays a crucial role in transportation safety.It is widely acknowledged that driver vigilance is a major contributor to traffic accidents.However,the quantitative impact of driver vigilance on driving risk has yet to be fully explored.This study aims to investigate the relationship between driver vigilance and driving risk,using data recorded from 28 drivers who maintain a speed of 80 km/h on a monotonous highway for 2 hours.The k-means and linear fitting methods are used to analyze the driving risk distribution under different driver vigilance states.Additionally,this study proposes a research framework for analyzing driving risk and develops three classification models(KNN,SVM,and DNN)to recognize the driving risk status.The results show that the frequency of low-risk incidents is negatively correlated with the driver's vigilance level,whereas the frequency of moderate-risk and high-risk incidents is positively correlated with the driver's vigilance level.The DNN model performs the best,achieving an accuracy of 0.972,recall of 0.972,precision of 0.973,and f1-score of 0.972,compared to KNN and SVM.This research could serve as a valuable reference for the design of warning systems and intelligent vehicles.