Filter and Embedded Feature Selection Methods to Meet Big Data Visualization Challenges

This study focuses on meeting the challenges of big data visualization by using of data reduction methods based the feature selection methods.To reduce the volume of big data and minimize model training time(Tt)while maintaining data quality.We contributed to meeting the challenges of big data visualization using the embedded method based“Select from model(SFM)”method by using“Random forest Importance algorithm(RFI)”and comparing it with the filter method by using“Select percentile(SP)”method based chi square“Chi2”tool for selecting the most important features,which are then fed into a classification process using the logistic regression(LR)algorithm and the k-nearest neighbor(KNN)algorithm.Thus,the classification accuracy(AC)performance of LRis also compared to theKNN approach in python on eight data sets to see which method produces the best rating when feature selection methods are applied.Consequently,the study concluded that the feature selection methods have a significant impact on the analysis and visualization of the data after removing the repetitive data and the data that do not affect the goal.After making several comparisons,the study suggests(SFMLR)using SFM based on RFI algorithm for feature selection,with LR algorithm for data classify.The proposal proved its efficacy by comparing its results with recent literature.

Exploring the growth value equity valuation model with data visualization

The Growth Value Model(GVM)proposed theoretical closed form formulas consist-ing of Return on Equity(ROE)and the Price-to-Book value ratio(P/B)for fair stock prices and expected rates of return.Although regression analysis can be employed to verify these theoretical closed form formulas,they cannot be explored by classical quintile or decile sorting approaches with intuition due to the essence of multi-factors and dynamical processes.This article uses visualization techniques to help intuitively explore GVM.The discerning findings and contributions of this paper is that we put forward the concept of the smart frontier,which can be regarded as the reasonable lower limit of P/B at a specific ROE by exploring fair P/B with ROE-P/B 2D dynamical process visualization.The coefficients in the formula can be determined by the quantile regression analysis with market data.The moving paths of the ROE and P/B in the cur-rent quarter and the subsequent quarters show that the portfolios at the lower right of the curve approaches this curve and stagnates here after the portfolios are formed.Furthermore,exploring expected rates of return with ROE-P/B-Return 3D dynamical process visualization,the results show that the data outside of the lower right edge of the“smart frontier”has positive quarterly return rates not only in the t+1 quarter but also in the t+2 quarter.The farther away the data in the t quarter is from the“smart frontier”,the larger the return rates in the t+1 and t+2 quarter.

Examining data visualization pitfalls in scientific publications

Data visualization blends art and science to convey stories from data via graphical representations.Considering different problems,applications,requirements,and design goals,it is challenging to combine these two components at their full force.While the art component involves creating visually appealing and easily interpreted graphics for users,the science component requires accurate representations of a large amount of input data.With a lack of the science component,visualization cannot serve its role of creating correct representations of the actual data,thus leading to wrong perception,interpretation,and decision.It might be even worse if incorrect visual representations were intentionally produced to deceive the viewers.To address common pitfalls in graphical representations,this paper focuses on identifying and understanding the root causes of misinformation in graphical representations.We reviewed the misleading data visualization examples in the scientific publications collected from indexing databases and then projected them onto the fundamental units of visual communication such as color,shape,size,and spatial orientation.Moreover,a text mining technique was applied to extract practical insights from common visualization pitfalls.Cochran’s Q test and McNemar’s test were conducted to examine if there is any difference in the proportions of common errors among color,shape,size,and spatial orientation.The findings showed that the pie chart is the most misused graphical representation,and size is the most critical issue.It was also observed that there were statistically significant differences in the proportion of errors among color,shape,size,and spatial orientation.

Painting image browser applying an associate-rule-aware multidimensional data visualization technique

Exploration of artworks is enjoyable but often time consuming.For example,it is not always easy to discover the favorite types of unknown painting works.It is not also always easy to explore unpopular painting works which looks similar to painting works created by famous artists.This paper presents a painting image browser which assists the explorative discovery of user-interested painting works.The presented browser applies a new multidimensional data visualization technique that highlights particular ranges of particular numeric values based on association rules to suggest cues to find favorite painting images.This study assumes a large number of painting images are provided where categorical information(e.g.,names of artists,created year)is assigned to the images.The presented system firstly calculates the feature values of the images as a preprocessing step.Then the browser visualizes the multidimensional feature values as a heatmap and highlights association rules discovered from the relationships between the feature values and categorical information.This mechanism enables users to explore favorite painting images or painting images that look similar to famous painting works.Our case study and user evaluation demonstrates the effectiveness of the presented image browser.

Design of a Web Crawler for Water Quality Monitoring Data and Data Visualization

Many countries are paying more and more attention to the protection of water resources at present,and how to protect water resources has received extensive attention from society.Water quality monitoring is the key work to water resources protection.How to efficiently collect and analyze water quality monitoring data is an important aspect of water resources protection.In this paper,python programming tools and regular expressions were used to design a web crawler for the acquisition of water quality monitoring data from Global Freshwater Quality Database(GEMStat)sites,and the multi-thread parallelism was added to improve the efficiency in the process of downloading and parsing.In order to analyze and process the crawled water quality data,Pandas and Pyecharts are used to visualize the water quality data to show the intrinsic correlation and spatiotemporal relationship of the data.

A Review of the Status and Development Strategies of Computer Science and Technology Under the Background of Big Data

This article discusses the current status and development strategies of computer science and technology in the context of big data.Firstly,it explains the relationship between big data and computer science and technology,focusing on analyzing the current application status of computer science and technology in big data,including data storage,data processing,and data analysis.Then,it proposes development strategies for big data processing.Computer science and technology play a vital role in big data processing by providing strong technical support.

A Survey of Multi-Space Techniques in Spatio-Temporal Simulation Data Visualization

The widespread use of numerical simulations in different scientific domains provides a variety of research opportunities.They often output a great deal of spatio-temporal simulation data,which are traditionally characterized as single-run,multi-run,multi-variate,multi-modal and multi-dimensional.From the perspective of data exploration and analysis,we noticed that many works focusing on spatiotemporal simulation data often share similar exploration techniques,for example,the exploration schemes designed in simulation space,parameter space,feature space and combinations of them.However,it lacks a survey to have a systematic overview of the essential commonalities shared by those works.In this survey,we take a novel multi-space perspective to categorize the state-ofthe-art works into three major categories.Specifically,the works are characterized as using similar techniques such as visual designs in simulation space(e.g,visual mapping,boxplot-based visual summarization,etc.),parameter space analysis(e.g,visual steering,parameter space projection,etc.)and data processing in feature space(e.g,feature definition and extraction,sampling,reduction and clustering of simulation data,etc.).

WebScope: A New Tool for Fusion Data Analysis and Visualization

A visualization tool was developed through a web browser based on Java applets embedded into HTML pages, in order to provide a world access to the EAST experimental data. It can display data from various trees in different servers in a single panel. With WebScope, it is easier to make a comparison between different data sources and perform a simple calculation over different data sources.

Image-guided color mapping for categorical data visualization

Appropriate color mapping for categorical data visualization can significantly facilitate the discovery of underlying data patterns and effectively bring out visual aesthetics.Some systems suggest pre-defined palettes for this task.However,a predefined color mapping is not always optimal,failing to consider users’needs for customization.Given an input cate-gorical data visualization and a reference image,we present an effective method to automatically generate a coloring that resembles the reference while allowing classes to be easily distinguished.We extract a color palette with high perceptual distance between the colors by sampling dominant and discriminable colors from the image’s color space.These colors are assigned to given classes by solving an integer quadratic program to optimize point distinctness of the given chart while preserving the color spatial relations in the source image.We show results on various coloring tasks,with a diverse set of new coloring appearances for the input data.We also compare our approach to state-of-the-art palettes in a controlled user study,which shows that our method achieves comparable performance in class discrimination,while being more similar to the source image.User feedback after using our system verifies its efficiency in automatically generating desirable colorings that meet the user’s expectations when choosing a reference.

Visualization Research and Application of Water Quality Monitoring Data Based on ECharts

Water resources are one of the basic resources for human survival,and water protection has been becoming a major problem for countries around the world.However,most of the traditional water quality monitoring research work is still concerned with the collection of water quality indicators,and ignored the analysis of water quality monitoring data and its value.In this paper,by adopting Laravel and AdminTE framework,we introduced how to design and implement a water quality data visualization platform based on Baidu ECharts.Through the deployed water quality sensor,the collected water quality indicator data is transmitted to the big data processing platform that deployed on Tencent Cloud in real time through the 4G network.The collected monitoring data is analyzed,and the processing result is visualized by Baidu ECharts.The test results showed that the designed system could run well and will provide decision support for water resource protection.

IoT Based Greenhouse Real-Time Data Acquisition and Visualization through Message Queuing Telemetry Transfer (MQTT) Protocol

One of the most indispensable needs of life is food and its worldwide availability endorsement has made agriculture an essential sector in recent years. As the technology evolved, the need to maintain a good and suitable climate in the greenhouse became imperative to ensure that the indoor plants are more productive hence the agriculture sector was not left behind. That notwithstanding, the introduction and deployment of IoT technology in agriculture solves many problems and increases crop production. This paper focuses mainly on the deployment of the Internet of Things (IoT) in acquiring real- time data of environmental parameters in the greenhouse. Various IoT technologies that can be applicable in greenhouse monitoring system was presented and in the proposed model, a method is developed to send the air temperature and humidity data obtained by the DHT11 sensor to the cloud using an ESP8266-based NodeMCU and firstly to the cloud platform Thing- Speak, and then to Adafruit.IO in which MQTT protocol was used for the reception of sensor data to the application layer referred as Human-Machine Interface. The system has been completely implemented in an actual prototype, allowing the acquiring of data and the publisher/subscriber concept used for communication. The data is published with a broker’s aid, which is responsible for transferring messages to the intended clients based on topic choice. Lastly, the functionality testing of MQTT was carried out and the results showed that the messages are successfully published.

Improving symbolic data visualization for pattern recognition and knowledge discovery

This paper examines the visualization of symbolic data and considers the challenges rising from its complex structure.Symbolic data is usually aggregated from large data sets and used to hide entry specific details and to transform huge amounts of data(like big data)into analyzable quantities.It is also used to offer an overview in places where general trends are more important than individual details.Symbolic data comes in many forms like intervals,histograms,categories and modal multi-valued objects.Symbolic data can also be considered as a distribution.Currently,the de facto visualization approach for symbolic data is zoomstars which has many limitations.The biggest limitation is that the default distributions(histograms)are not supported in 2D as additional dimension is required.This paper proposes several new improvements for zoomstars which would enable it to visualize histograms in 2D by using a quantile or an equivalent interval approach.In addition,several improvements for categorical and modal variables are proposed for a clearer indication of presented categories.Recommendations for different approaches to zoomstars are offered depending on the data type and the desired goal.Furthermore,an alternative approach that allows visualizing the whole data set in comprehensive table-like graph,called shape encoding,is proposed.These visualizations and their usefulness are verified with three symbolic data sets in exploratory data mining phase to identify trends,similar objects and important features,detecting outliers and discrepancies in the data.

DAViS:a unified solution for data collection, analyzation,and visualization in real‑time stock market prediction

The explosion of online information with the recent advent of digital technology in information processing,information storing,information sharing,natural language processing,and text mining techniques has enabled stock investors to uncover market movement and volatility from heterogeneous content.For example,a typical stock market investor reads the news,explores market sentiment,and analyzes technical details in order to make a sound decision prior to purchasing or selling a particular company’s stock.However,capturing a dynamic stock market trend is challenging owing to high fluctuation and the non-stationary nature of the stock market.Although existing studies have attempted to enhance stock prediction,few have provided a complete decision-support system for investors to retrieve real-time data from multiple sources and extract insightful information for sound decision-making.To address the above challenge,we propose a unified solution for data collection,analysis,and visualization in real-time stock market prediction to retrieve and process relevant financial data from news articles,social media,and company technical information.We aim to provide not only useful information for stock investors but also meaningful visualization that enables investors to effectively interpret storyline events affecting stock prices.Specifically,we utilize an ensemble stacking of diversified machine-learning-based estimators and innovative contextual feature engineering to predict the next day’s stock prices.Experiment results show that our proposed stock forecasting method outperforms a traditional baseline with an average mean absolute percentage error of 0.93.Our findings confirm that leveraging an ensemble scheme of machine learning methods with contextual information improves stock prediction performance.Finally,our study could be further extended to a wide variety of innovative financial applications that seek to incorporate external insight from contextual information such as large-scale online news articles and social media data.

On Visualization Analysis of Stock Data

Big data technology is changing with each passing day,generating massive amounts of data every day.These data have large capacity,many types,fast growth,and valuable features.The same is true for the stock investment market.The growth of the amount of stock data generated every day is difficult to predict.The price trend in the stock market is uncertain,and the valuable information hidden in the stock data is difficult to detect.For example,the price trend of stocks,profit trends,how to make a reasonable speculation on the price trend of stocks and profit trends is a major problem that needs to be solved at this stage.This article uses the Python language to visually analyze,calculate,and predict each stock.Realize the integration and calculation of stock data to help people find out the valuable information hidden in stocks.The method proposed in this paper has been tested and proved to be feasible.It can reasonably extract,analyze and calculate the stock data,and predict the stock price trend to a certain extent.

VISUALIZATION OF THREE-DIMENSIONAL DATA FIELD AND ITS APPLICATION IN MACHINE TESTING

In order to realize visualization of three-dimensional data field （TDDF） in instrument, two methods of visualization of TDDF and the usual manner of quick graphic and image processing are analyzed. And how to use OpenGL technique and the characteristic of analyzed data to construct a TDDF, the ways of reality processing and interactive processing are described. Then the medium geometric element and a related realistic model are constructed by means of the first algorithm. Models obtained for attaching the third dimension in three-dimensional data field are presented. An example for TDDF realization of machine measuring is provided. The analysis of resultant graphic indicates that the three-dimensional graphics built by the method developed is featured by good reality, fast processing and strong interaction

Visualizing risk factors of dementia from scholarly literature using knowledge maps and next-generation data models

Scholarly communication of knowledge is predominantly document-based in digital repositories,and researchers find it tedious to automatically capture and process the semantics among related articles.Despite the present digital era of big data,there is a lack of visual representations of the knowledge present in scholarly articles,and a time-saving approach for a literature search and visual navigation is warranted.The majority of knowledge display tools cannot cope with current big data trends and pose limitations in meeting the requirements of automatic knowledge representation,storage,and dynamic visualization.To address this limitation,the main aim of this paper is to model the visualization of unstructured data and explore the feasibility of achieving visual navigation for researchers to gain insight into the knowledge hidden in scientific articles of digital repositories.Contemporary topics of research and practice,including modifiable risk factors leading to a dramatic increase in Alzheimer’s disease and other forms of dementia,warrant deeper insight into the evidence-based knowledge available in the literature.The goal is to provide researchers with a visual-based easy traversal through a digital repository of research articles.This paper takes the first step in proposing a novel integrated model using knowledge maps and next-generation graph datastores to achieve a semantic visualization with domain-specific knowledge,such as dementia risk factors.The model facilitates a deep conceptual understanding of the literature by automatically establishing visual relationships among the extracted knowledge from the big data resources of research articles.It also serves as an automated tool for a visual navigation through the knowledge repository for faster identification of dementia risk factors reported in scholarly articles.Further,it facilitates a semantic visualization and domain-specific knowledge discovery from a large digital repository and their associations.In this study,the implementation of the proposed model in the Neo4j graph data repository,along with the results achieved,is presented as a proof of concept.Using scholarly research articles on dementia risk factors as a case study,automatic knowledge extraction,storage,intelligent search,and visual navigation are illustrated.The implementation of contextual knowledge and its relationship for a visual exploration by researchers show promising results in the knowledge discovery of dementia risk factors.Overall,this study demonstrates the significance of a semantic visualization with the effective use of knowledge maps and paves the way for extending visual modeling capabilities in the future.

Database system for managing 20,00020-inch PMTs at JUNO

A database system,known as the large PMT characterization and instrumentation database system(LPMT-CIDS),was designed and implemented for the Jiangmen Underground Neutrino Observatory(JUNO).The system is based on a Linux+Apache+MySQL+PHP(LAMP)server and focuses on modularization and architecture separation.It covers all the testing stages for the 20-inch photomultiplier tubes(PMTs)at JUNO and provides its users with data storage,analysis,and visualization services.Based on the successful use of the system in the 20-inch PMT testing program,its design approach and construction elements can be extended to other projects.

Image Post-Processing Method for Visual Data Mining

Visual data mining is one of important approach of data mining techniques. Most of them are based on computer graphic techniques but few of them exploit image-processing techniques. This paper proposes an image processing method, named RNAM （resemble neighborhood averaging method）, to facilitate visual data mining, which is used to post-process the data mining result-image and help users to discover significant features and useful patterns effectively. The experiments show that the method is intuitive, easily-understanding and effectiveness. It provides a new approach for visual data mining.

Search Processes in the Exploration of Complex Data under Different Display Conditions

The study investigated user experience, display complexity, display type (tables versus graphs), and task difficulty as variables affecting the user’s ability to navigate through complex visual data. A total of 64 participants, 39 undergraduate students (novice users) and 25 graduate students (intermediate-level users) participated in the study. The experimental design was 2 × 2 × 2 × 3 mixed design using two between-subject variables (display complexity, user experience) and two within-subject variables (display format, question difficulty). The results indicated that response time was superior for graphs (relative to tables), especially when the questions were difficult. The intermediate users seemed to adopt more extensive search strategies than novices, as revealed by an analysis of the number of changes they made to the display prior to answering questions. It was concluded that designers of data displays should consider the (a) type of display, (b) difficulty of the task, and (c) expertise level of the user to obtain optimal levels of performance.

Design and Implementation of Log Data Analysis Management System Based on Hadoop

With the rapid development of the Internet,many enterprises have launched their network platforms.When users browse,search,and click the products of these platforms,most platforms will keep records of these network behaviors,these records are often heterogeneous,and it is called log data.To effectively to analyze and manage these heterogeneous log data,so that enterprises can grasp the behavior characteristics of their platform users in time,to realize targeted recommendation of users,increase the sales volume of enterprises’products,and accelerate the development of enterprises.Firstly,we follow the process of big data collection,storage,analysis,and visualization to design the system,then,we adopt HDFS storage technology,Yarn resource management technology,and gink load balancing technology to build a Hadoop cluster to process the log data,and adopt MapReduce processing technology and data warehouse hive technology analyze the log data to obtain the results.Finally,the obtained results are displayed visually,and a log data analysis system is successfully constructed.It has been proved by practice that the system effectively realizes the collection,analysis and visualization of log data,and can accurately realize the recommendation of products by enterprises.The system is stable and effective.