DTBVis:An interactive visual comparison system for digital twin brain and human brain

The digital twin brain(DTB)computing model from brain-inspired computing research is an emerging artificial intelligence technique,which is realized by a computational modeling approach of hardware and software.It can achieve various cognitive abilities and their synergistic mechanisms in a manner similar to the human brain.Given that the task of the DTB is to simulate the functions of the human brain,comparing the similarities and differences between the two is crucial.However,the visualization study of the DTB is still under-researched.Moreover,the complexity of the datasets(multilevel spatiotemporal granularity and different types of comparison tasks)presents new challenges to the analysis and exploration of visualization.Therefore,in this study,we proposed DTBVis,a visual analytics system that supports comparison tasks for the DTB.DTBVis supports iterative explorations from different levels and at different granularities.Combined with automatic similarity recommendation,and high-dimensional exploration,DTBVis can assist experts in understanding the similarities and differences between the DTB and the human brain,thus helping them adjust their model and enhance its functionality.The highest level of DTBVis shows an overview of the datasets from the brain,which is used for comparison and exploration of the function and structure of the DTB and the human brain.The medium level is used for the comparison and exploration of a designated brain region.The low level can analyze a designated brain voxel.We worked closely with experts of brain science and held regular seminars with them.Feedback from the experts indicates that our approach helps them conduct comparative studies of the DTB and human brain and make modeling adjustments of the DTB through intuitive visual comparisons and interactive explorations.

Identifying,exploring,and interpreting time series shapes in multivariate time intervals

We introduce a concept of episode referring to a time interval in the development of a dynamic phenomenon that is characterized by multiple time-variant attributes.A data structure representing a single episode is a multivariate time series.To analyse collections of episodes,we propose an approach that is based on recognition of particular patterns in the temporal variation of the variables within episodes.Each episode is thus represented by a combination of patterns.Using this representation,we apply visual analytics techniques to fulfil a set of analysis tasks,such as investigation of the temporal distribution of the patterns,frequencies of transitions between the patterns in episode sequences,and co-occurrences of patterns of different variables within same episodes.We demonstrate our approach on two examples using real-world data,namely,dynamics of human mobility indicators during the COVID-19 pandemic and characteristics of football team movements during episodes of ball turnover.

A survey of immersive visualization:Focus on perception and interaction

Immersive visualization utilizes virtual reality,mixed reality devices,and other interactive devices to create a novel visual environment that integrates multimodal perception and interaction.This technology has been maturing in recent years and has found broad applications in various fields.Based on the latest research advancements in visualization,this paper summarizes the state-of-theart work in immersive visualization from the perspectives of multimodal perception and interaction in immersive environments,additionally discusses the current hardware foundations of immersive setups.By examining the design patterns and research approaches of previous immersive methods,the paper reveals the design factors for multimodal perception and interaction in current immersive environments.Furthermore,the challenges and development trends of immersive multimodal perception and interaction techniques are discussed,and potential areas of growth in immersive visualization design directions are explored.

Multi-scale visual analysis of cycle characteristics in spatially-embedded graphs

We present a visual analysis environment based on a multi-scale partitioning of a 2d domain intoregions bounded by cycles in weighted planar embedded graphs.The work has been inspired by anapplication in granular materials research,where the question of scale plays a fundamental role inthe analysis of material properties.We propose an efficient algorithm to extract the hierarchical cyclestructure using persistent homology.The core of the algorithm is a filtration on a dual graph exploitingAlexander’s duality.The resulting partitioning is the basis for the derivation of statistical properties thatcan be explored in a visual environment.We demonstrate the proposed pipeline on a few syntheticand one real-world dataset.

IVMS:An immersive virtual meteorological sandbox based on WYSIWYG

A novel approach to visually represent meteorological data has emerged with the maturation of Immersive Analytics(IA).We have proposed an immersive meteorological virtual sandbox as a solution to the limitations of 2D analysis in expressing and perceiving data.This innovative visual method enables users to interact directly with data through non-contact aerial gestures(NCAG).Referring to the“What you see is what you get”concept in scientific visualization,we proposed a novel approach for the visual exploration of meteorological data that aims to immerse users in the analysis process.We hope this approach can inspire immersive visualization techniques for other types of geographic data as well.Finally,we conducted a user questionnaire to evaluate our system and work.The evaluation results demonstrate that our system effectively reduces cognitive burden,alleviates mental workload,and enhances users’retention of analysis findings.

X-Space:Interaction design of extending mixed reality space from Web2D visualization

Mixed reality offers a larger visualization space and more intuitive means of interaction for data exploration,and many works have been dedicated to combining 2D visualizations on screen with mixe reality.However,for each combination,we need to customize the implementation of the corresponding mixed reality 3D visualization.It is a challenge to simplify this development process and enable agile building of mixed reality 3D visualizations for 2D visualizations.In addition,many existing 2D visualizations do not provide interfaces oriented to immersive analytics,so how to extend the mixed reality 3D space from existing 2D visualizations is another challenge.This work presents an agile and flexible approach to interactively transfer visualizations from 2D screens to mixed reality 3D spaces.We designed an interactive process for spatial generation of mixed-reality 3D visualizations,defined a unified data transfer framework,integrated data deconstruction techniques for 2D visualizations,implemented interfaces to immersive visualization building tool-kits,and encapsulated these techniques into a tool named X-Space.We validated that the approach is feasible and effective through 2D visualization cases including scatter plots,stacked bar charts,and adjacency matrix.Finally,we conducted expert interviews to discuss the usability and value of the method.

Augmented reality for supporting geo-spatial planning:An open access review

Augmented reality is gaining traction across many domains.One of these is participation within geo-spatial planning projects.The interactive and three-dimensional nature of augmented reality is suitably placed to cater for a higher quality of communication and information exchange in planning processes.Thus,this research provides an overview of the use of AR in planning processes,specifically regarding the participation aspect,through an open-access systematic literature review,for which the investigation identifies 35 articles concerning the current state-of-the-art of augmented reality in planning.Findings indicate the rather limited use of augmented reality in the overall planning process due to technical limitations.Nonetheless,it shows to be a useful technology where it allows for higher user engagement and a clearer understanding among users in planning projects.Additionally,in participation,the technology offers a motivational solution and creates an overall higher acceptance and awareness of the plan,making the participants more engaged and represented in the planning process.

Simplifying social networks via triangle-based cohesive subgraphs

One main challenge for simplifying node-link diagrams of large-scale social networks lies in that simplified graphs generally contain dense subgroups or cohesive subgraphs.Graph triangles quantify the solid and stable relationships that maintain cohesive subgraphs.Understanding the mechanism of triangles within cohesive subgraphs contributes to illuminating patterns of connections within social networks.However,prior works can hardly handle and visualize triangles in cohesive subgraphs.In this paper,we propose a triangle-based graph simplification approach that can filter and visualize cohesive subgraphs by leveraging a triangle-connectivity called k-truss and a force-directed algorithm.We design and implement TriGraph,a web-based visual interface that provides detailed information for exploring and analyzing social networks.Quantitative comparisons with existing methods,two case studies on real-world datasets,and feedback from domain experts demonstrate the effectiveness of TriGraph.

On generated artistic styles:Image generation experiments with GAN algorithms

As computer graphics technology supports pursuing a photorealistic style,replicated artworks with a photorealistic style overwhelmingly predominate in the computer-generated art circle.Along with the progression of generative technology,this trend may make generative art a virtual world of photorealistic fake,in which the single criterion of expressive style imperils art into the context of a single boring stereotype.This article focuses on the issue of style diversity and its technical feasibility by artistic experiments of generating flower images in StyleGAN.The author insisted that photo both technology and artistic style should not be confined merely for realistic purposes.This proposition was validated in the GAN generation experiment by changing the training materials.

Visualizing ordered bivariate data on node-link diagrams

Node-link visual representation is a widely used tool that allows decision-makers to see details about a network through the appropriate choice of visual metaphor.However,existing visualization methods are not always effective and efficient in representing bivariate graph-based data.This study proposes a novel node-link visual model–visual entropy(Vizent)graph–to effectively represent both primary and secondary values,such as uncertainty,on the edges simultaneously.We performed two user studies to demonstrate the efficiency and effectiveness of our approach in the context of static nodelink diagrams.In the first experiment,we evaluated the performance of the Vizent design to determine if it performed equally well or better than existing alternatives in terms of response time and accuracy.Three static visual encodings that use two visual cues were selected from the literature for comparison:Width-Lightness,Saturation-Transparency,and Numerical values.We compared the Vizent design to the selected visual encodings on various graphs ranging in complexity from 5 to 25 edges for three different tasks.The participants achieved higher accuracy of their responses using Vizent and Numerical values;however,both Width-Lightness and Saturation-Transparency did not show equal performance for all tasks.Our results suggest that increasing graph size has no impact on Vizent in terms of response time and accuracy.The performance of the Vizent graph was then compared to the Numerical values visualization.The Wilcoxon signed-rank test revealed that mean response time in seconds was significantly less when the Vizent graphs were presented,while no significant difference in accuracy was found.The results from the experiments are encouraging and we believe justify using the Vizent graph as a good alternative to traditional methods for representing bivariate data in the context of node-link diagrams.

CARVING-DETC: A network scaling and NMS ensemble for Balinese carving motif detection method

Balinese carvings are cultural objects that adorn sacred buildings. The carvings consist of several motifs,each representing the values adopted by the Balinese people. Detection of Balinese carving motifs ischallenging due to the unavailability of a Balinese carving dataset for detection tasks, high variance,and tiny-size carving motifs. This research aims to improve carving motif detection performance onchallenging Balinese carving motifs detection task through a modification of YOLOv5 to support adigital carving conservation system. We proposed CARVING-DETC, a deep learning-based Balinesecarving detection method consisting of three steps. First, the data generation step performs dataaugmentation and annotation on Balinese carving images. Second, we proposed a network scalingstrategy on the YOLOv5 model and performed non-maximum suppression (NMS) on the modelensemble to generate the most optimal predictions. The ensemble model utilizes NMS to producehigher performance by optimizing the detection results based on the highest confidence score andsuppressing other overlap predictions with a lower confidence score. Third, performance evaluation onscaled-YOLOv5 versions and NMS ensemble models. The research findings are beneficial in conservingthe cultural heritage and as a reference for other researchers. In addition, this study proposed a novelBalinese carving dataset through data collection, augmentation, and annotation. To our knowledge,it is the first Balinese carving dataset for the object detection task. Based on experimental results,CARVING-DETC achieved a detection performance of 98%, which outperforms the baseline model.

The use of facial expressions in measuring students’interaction with distance learning environments during the COVID-19 crisis

Digital learning is becoming increasingly important in the crisis COVID-19 and is widespread in most countries.The proliferation of smart devices and 5G telecommunications systems are contributing to the development of digital learning systems as an alternative to traditional learning systems.Digital learning includes blended learning,online learning,and personalized learning which mainly depends on the use of new technologies and strategies,so digital learning is widely developed to improve education and combat emerging disasters such as COVID-19 diseases.Despite the tremendous benefits of digital learning,there are many obstacles related to the lack of digitized curriculum and collaboration between teachers and students.Therefore,many attempts have been made to improve the learning outcomes through the following strategies:collaboration,teacher convenience,personalized learning,cost and time savings through professional development,and modeling.In this study,facial expressions and heart rates are used to measure the effectiveness of digital learning systems and the level of learners’engagement in learning environments.The results showed that the proposed approach outperformed the known related works in terms of learning effectiveness.The results of this research can be used to develop a digital learning environment.

Towards the automation of book typesetting

This paper proposes a generative approach for the automatic typesetting of books in desktop publishing.The presented system consists in a computer script that operates inside a widely used design software tool and implements a generative process based on several typographic rules,styles and principles which have been identified in the literature.The performance of the proposed system is tested through an experiment which included the evaluation of its outputs with people.The results reveal the ability of the system to consistently create varied book designs from the same input content as well as visually coherent book designs with different contents while complying with fundamental typographic principles.

PubExplorer:An interactive analytical system for visualizing publication data

With the intersection and convergence of multiple disciplines and technologies,more and more researchers are actively exploring interdisciplinary cooperation outside their main research fields.Facing a new research field,researchers often hope to quickly learn what is being studied in this field,which research points are receiving high attention,which researchers are studying these research points,and then consider the possibility of collaborating with core researchers on these research points.In addition,students who are preparing for academic further education usually conduct research on mentors and mentors’research platforms,including academic connections,employment opportunities,etc.In order to satisfy these requirements,we(1)design a research point state map based on a science map to help researchers and students understand the development state of a new research field;(2)design a bar-link author-affiliation information graph to help researchers and students clarify academic networks of scholars and find suitable collaborators or mentors;(3)designs citation pattern histogram to quickly discover research achievements with high research value,such as the Sleeping Beauty papers,recently hot papers,classic papers and so on.Finally,an interactive analytical system named PubExplorer was implemented with IEEE VIS publication data,and its effectiveness is verified through case studies.

Perspectives on point cloud-based 3D scene modeling and XR presentation within the cloud-edge-client architecture

With the support of edge computing,the synergy and collaboration among central cloud,edge cloud,and terminal devices form an integrated computing ecosystem known as the cloud-edge-client architecture.This integration unlocks the value of data and computational power,presenting significant opportunities for large-scale 3D scene modeling and XR presentation.In this paper,we explore the perspectives and highlight new challenges in 3D scene modeling and XR presentation based on point cloud within the cloud-edge-client integrated architecture.We also propose a novel cloud-edge-client integrated technology framework and a demonstration of municipal governance application to address these challenges.

Visualization of isomorphism-synesthesia of colour and music

Music and colour,as human hearing and visual art,are closely related to human psychological feelings and symbolic associations.There is an isomorphic relationship between music and colour.The article uses the concept of“synesthesia”in psychology and the“co-construction”relationship in mathematics as a bridge,based on Kandinsky’s“inner sound”theory and Mallion’s“tone-colour system”,an interdisciplinary theoretical model of“timbre isomorphism synesthesia”(ISCM)is constructed.At the practical level,based on the ISCM theory,a set of timbre synesthesia visualization tools ASAH and visualization processes are designed,through music data input,graphics mapping visualization,colour mapping visualization,real-time interactive visualization,and finally output timbre synesthesia visualization works.In order to avoid the visual homogenization caused by algorithm design,ASHA has set up a custom editor,which emphasizes the individual differences and multi-sensory experience of tonal synesthesia visualization.

VISHIEN-MAAT:Scrollytelling visualization design for explaining Siamese Neural Network concept to non-technical users

The past decade has witnessed rapid progress in AI research since the breakthrough in deep learning.AI technology has been applied in almost every field;therefore,technical and non-technical endusers must understand these technologies to exploit them.However existing materials are designed for experts,but non-technical users need appealing materials that deliver complex ideas in easy-tofollow steps.One notable tool that fits such a profile is scrollytelling,an approach to storytelling that provides readers with a natural and rich experience at the reader’s pace,along with in-depth interactive explanations of complex concepts.Hence,this work proposes a novel visualization design for creating a scrollytelling that can effectively explain an AI concept to non-technical users.As a demonstration of our design,we created a scrollytelling to explain the Siamese Neural Network for the visual similarity matching problem.Our approach helps create a visualization valuable for a shorttimeline situation like a sales pitch.The results show that the visualization based on our novel design helps improve non-technical users’perception and machine learning concept knowledge acquisition compared to traditional materials like online articles.

KnowU social teleprompter:Interaction design applied to intervention therapy for language decline in early AD patients

Art therapy as an intervention has been shown to alleviate social impairment in people with AD.Meanwhile,digital technology(DTS)has been shown to perform well in different degenerative dementias through mobile devices and apps.However,it is unclear whether digital art creation therapy has an impact on the speech function of people with early AD.Therefore,the aim of this study was to confirm whether digital art creation therapy has an ameliorating effect on language decline in AD patients through the KnowU social teleprompter.This study was a controlled trial in which 16 patients with early AD worked with us and were divided into a paper-based art creation therapy group(control group)and a KnowU social teleprompter therapy group for a 6-week intervention.In the digital art creation intervention group we introduced the KnowU digital kit,consisting of a creation plug-in for the Procreate app on a tablet and a wearable device and its app.The entire treatment process is recorded and combined with a quantitative analysis of the McNemarχ^(2)test to analyze the differences in outcomes of verbal communication function in early AD patients after different therapies.Ultimately,it is shown that early AD patients utilizing the KnowU social teleprompter are more effective in the intervention treatment of language decline in the real social domain compared to the paper-based art creation therapy group.The discussion further demonstrates that DTs and art therapy can provide a better social experience,creative approach and emotional recall of language loss in early AD patients,as well as increase the collaborative relationship between early AD patients and their caregivers.

TopicBubbler:An interactive visual analytics system for cross-level fine-grained exploration of social media data

How to explore fine-grained but meaningful information from the massive amount of social media data is critical but challenging.To address this challenge,we propose the TopicBubbler,a visual analytics system that supports the cross-level fine-grained exploration of social media data.To achieve the goal of cross-level fine-grained exploration,we propose a new workflow.Under the procedure of the workflow,we construct the fine-grained exploration view through the design of bubble-based word clouds.Each bubble contains two rings that can display information through different levels,and recommends six keywords computed by different algorithms.The view supports users collecting information at different levels and to perform fine-grained selection and exploration across different levels based on keyword recommendations.To enable the users to explore the temporal information and the hierarchical structure,we also construct the Temporal View and Hierarchical View,which satisfy users to view the cross-level dynamic trends and the overview hierarchical structure.In addition,we use the storyline metaphor to enable users to consolidate the fragmented information extracted across levels and topics and ultimately present it as a complete story.Case studies from real-world data confirm the capability of the TopicBubbler from different perspectives,including event mining across levels and topics,and fine-grained mining of specific topics to capture events hidden beneath the surface.

Exploring and visualizing temporal relations in multivariate time series

This paper introduces an approach to analyzing multivariate time series(MVTS)data through progressive temporal abstraction of the data into patterns characterizing the behavior of the studied dynamic phenomenon.The paper focuses on two core challenges:identifying basic behavior patterns of individual attributes and examining the temporal relations between these patterns across the range of attributes to derive higher-level abstractions of multi-attribute behavior.The proposed approach combines existing methods for univariate pattern extraction,computation of temporal relations according to the Allen’s time interval algebra,visual displays of the temporal relations,and interactive query operations into a cohesive visual analytics workflow.The paper describes the application of the approach to real-world examples of population mobility data during the COVID-19 pandemic and characteristics of episodes in a football match,illustrating its versatility and effectiveness in understanding composite patterns of interrelated attribute behaviors in MVTS data.