On Multi-Thread Crawler Optimization for Scalable Text Searching

Web crawlers are an important part of modern search engines.With the development of the times,data has exploded and humans have entered a“big data era”.For example,Wikipedia carries the knowledge from all over the world,records the realtime news that occurs every day,and provides users with a good database of data,but because of the large amount of data,it puts a lot of pressure on users to search.At present,single-threaded crawling data can no longer meet the requirements of text crawling.In order to improve the performance and program versatility of single-threaded crawlers,a high-speed multi-threaded web crawler is designed to crawl the network hyper-scale text database.Multi-threaded crawling uses multiple threads to process web pages in parallel,combining breadth-first and depth-first algorithms to control web crawling.The practice project is based on the Python language to achieve multi-threaded optimization network hyper-large-scale text database-Wikipedia book crawling method,the project is inspired by the article on the Wikipedia article in the Big Data Digest public number.

Using multi-threads to hide deduplication I/O latency with low synchronization overhead

Data deduplication, as a compression method, has been widely used in most backup systems to improve bandwidth and space efficiency. As data exploded to be backed up, two main challenges in data deduplication are the CPU-intensive chunking and hashing works and the I/0 intensive disk-index access latency. However, CPU-intensive works have been vastly parallelized and speeded up by multi-core and many-core processors; the I/0 latency is likely becoming the bottleneck in data deduplication. To alleviate the challenge of I/0 latency in multi-core systems, multi-threaded deduplication （Multi-Dedup） architecture was proposed. The main idea of Multi-Dedup was using parallel deduplication threads to hide the I/0 latency. A prefix based concurrent index was designed to maintain the internal consistency of the deduplication index with low synchronization overhead. On the other hand, a collisionless cache array was also designed to preserve locality and similarity within the parallel threads. In various real-world datasets experiments, Multi-Dedup achieves 3-5 times performance improvements incorporating with locality-based ChunkStash and local-similarity based SiLo methods. In addition, Multi-Dedup has dramatically decreased the synchronization overhead and achieves 1.5-2 times performance improvements comparing to traditional lock-based synchronization methods.

An HLA/RTI Architecture Based on Multi-thread Processing

In order to improve the real-time performance of the real-time HLA(high level architecture) in the application of massive data communication volume,multi-thread processing was adopted,thread pool structure was introduced into the system,different threads to handle corresponding message queues was utilized to respond different message requests.Furthermore,an allocation strategy of semi-complete deprivation of priority was adopted,which reduces thread switching cost and processing burden in the system,provided that the message requests with high priority can be responded in time,thus improves the system's overall performance.The design and experiment results indicate that the method proposed in this paper can improve the real-time performance of HLA in distributed system applications greatly.

Dynamically Translating Binary Code for Multi-Threaded Programs Using Shared Code Cache

mc211vm is a process-level ARM-to-x86 binary translator developed in our lab in the past several years. Currently, it is able to emulate singlethreaded programs. We extend mc211vm to emulate multi-threaded programs. Our main task is to reconstruct its architecture for multi-threaded programs. Register mapping, code cache management, and address mapping in mc2llvm have all been modified. In addition, to further speed up the emulation, we collect hot paths, aggressively optimize and generate code for them at run time. Additional threads are used to alleviate the overhead. Thus, when the same hot path is walked through again, the corresponding optimized native code will be executed instead. In our experiments, our system is 8.8X faster than QEMU （quick emulator） on average when emulating the specified benchmarks with 8 guest threads.

Research and Review on the Java Multi-thread Programming and Its Further Development Tendency

In this paper, we conduct research on the Java multi-thread programming and its further development tendency. Multithreading mechanisms can run several programs at the same time, make the program run effi ciency becomes higher that also can overcome the problem of basic traditional programming language design while its design is the key to the realization of the synchronous thread. Multithreading is a mechanism that allows concurrent execution of multiple instruction stream in the program, each instruction stream is called a thread, independent from each other between each other. Thread is also known as a lightweight process, it have independent execution and process control. Our research starts from the analysis of the corresponding mechanism to enhance the performance that is innovative and meaningful.

Reduction of distortion and improvement of efficiency for gridding of scattered gravity and magnetic data

This paper presents a reasonable gridding-parameters extraction method for setting the optimal interpolation nodes in the gridding of scattered observed data. The method can extract optimized gridding parameters based on the distribution of features in raw data. Modeling analysis proves that distortion caused by gridding can be greatly reduced when using such parameters. We also present some improved technical measures that use human- machine interaction and multi-thread parallel technology to solve inadequacies in traditional gridding software. On the basis of these methods, we have developed software that can be used to grid scattered data using a graphic interface. Finally, a comparison of different gridding parameters on field magnetic data from Ji Lin Province, North China demonstrates the superiority of the proposed method in eliminating the distortions and enhancing gridding efficiency.

Improved Software Implementation for Montgomery Elliptic Curve Cryptosystem

The last decade witnessed rapid increase in multimedia and other applications that require transmitting and protecting huge amount of data streams simultaneously.For such applications,a high-performance cryptosystem is compulsory to provide necessary security services.Elliptic curve cryptosystem(ECC)has been introduced as a considerable option.However,the usual sequential implementation of ECC and the standard elliptic curve(EC)form cannot achieve required performance level.Moreover,the widely used Hardware implementation of ECC is costly option and may be not affordable.This research aims to develop a high-performance parallel software implementation for ECC.To achieve this,many experiments were performed to examine several factors affecting ECC performance including the projective coordinates,the scalar multiplication algorithm,the elliptic curve(EC)form,and the parallel implementation.The ECC performance was analyzed using the different factors to tune-up them and select the best choices to increase the speed of the cryptosystem.Experimental results illustrated that parallel Montgomery ECC implementation using homogenous projection achieves the highest performance level,since it scored the shortest time delay for ECC computations.In addition,results showed thatNAF algorithm consumes less time to perform encryption and scalar multiplication operations in comparison withMontgomery ladder and binarymethods.Java multi-threading technique was adopted to implement ECC computations in parallel.The proposed multithreaded Montgomery ECC implementation significantly improves the performance level compared to previously presented parallel and sequential implementations.

The Double Edge Sword Based Distributed Executor Service

Scalability is one of the most important quality attribute of softwareintensive systems,because it maintains an effective performance parallel to the large fluctuating and sometimes unpredictable workload.In order to achieve scalability,thread pool system(TPS)(which is also known as executor service)has been used extensively as a middleware service in software-intensive systems.TPS optimization is a challenging problem that determines the optimal size of thread pool dynamically on runtime.In case of distributed-TPS(DTPS),another issue is the load balancing b/w available set of TPSs running at backend servers.Existing DTPSs are overloaded either due to an inappropriate TPS optimization strategy at backend servers or improper load balancing scheme that cannot quickly recover an overload.Consequently,the performance of software-intensive system is suffered.Thus,in this paper,we propose a new DTPS that follows the collaborative round robin load balancing that has the effect of a double-edge sword.On the one hand,it effectively performs the load balancing(in case of overload situation)among available TPSs by a fast overload recovery procedure that decelerates the load on the overloaded TPSs up to their capacities and shifts the remaining load towards other gracefully running TPSs.And on the other hand,its robust load deceleration technique which is applied to an overloaded TPS sets an appropriate upper bound of thread pool size,because the pool size in each TPS is kept equal to the request rate on it,hence dynamically optimizes TPS.We evaluated the results of the proposed system against state of the art DTPSs by a clientserver based simulator and found that our system outperformed by sustaining smaller response times.

Development of Data Processing Software for NBI Spectroscopic Analysis System

A set of data processing software is presented in this paper for processing NBI spectroscopic data.For better and more scientific managment and querying these data,they are managed uniformly by the NBI data server.The data processing software offers the functions of uploading beam spectral original and analytic data to the data server manually and automatically,querying and downloading all the NBI data,as well as dealing with local LZO data.The set software is composed of a server program and a client program.The server software is programmed in C/C＋＋under a CentOS development environment.The client software is developed under a VC6.0 platform,which offers convenient operational human interfaces.The network communications between the server and the client are based on TCP.With the help of this set software,the NBI spectroscopic analysis system realizes the unattended automatic operation,and the clear interface also makes it much more convenient to offer beam intensity distribution data and beam power data to operators for operation decision-making.

Panoramic Imaging System Inspired by Insect Compound Eyes

Inspired by the unique structure of insect compound eyes,a multi-channel image acquisition system is designed to photograph a cylindrical panorama of its surroundings with one shot. The hardware structure consists of an embedded ARM system and one array of 16 micro-image sensors. The system achieves the synchronization of captured photos in 10 ms,as well as 10 f /s video capture. The software architecture includes the TCP /IP protocol,video capture procedures in"Poll/Read"or"video streaming"modes,thread pool monitoring in multi-threading mutex,synchronization control with the"event""mutex signal"and"critical region"functions,and a synthetic image algorithm characterized by its portability,modularity,and remote transmission. The panoramic imaging system is expected to be a vision sensor for mobile robotics.

Fault Detection and Isolation for Low Hardware Redundancy Flight Control System

The problems of current highly redundant flight control system are analyzed in this paper. Our study gives methods of utilizing other information to reduce physical components on the condition of meeting the reliability requirements for flight control system. The strategies presented in this paper mainly include information redundancy, multi-thread, time redundancy, geometry space redundancy, etc.. Analysis and simulation show these non-hardware based methods can reduce the requirement of system hardware level and thus reduce the system complexity, weight, space, costs and R＆D （research and development） time.

Error Robust H.264 Video Transmission Schemes Based on Multi-frame

Multi-frame coding is supported by the emerging H.264. It is important for the enhancement of both coding efficiency and error robustness. In this paper, error resilient schemes for H.264 based on multi-frame were investigated. Error robust H.264 video transmission schemes were introduced for the applications with and without a feedback channel. The experimental results demonstrate the effectiveness of the proposed schemes.

Implement of the Production Monitoring and Control System for Coning,Drawing and Twisting's Workshop

To achieve the production information management of the coning,drawing and twisting's workshop,and promote a good and fast development of the textile enterprise,firstly,we analyze the information technology development status of the textile enterprise at home and abroad,study the weaknesses of the existing computer monitoring system,introduce the system design project,and propose a system network model based on client/server(C/S)mode.Secondly,a multi-agent production monitoring and control system is developed,and the communication of the single-port-multi-protocol and a multi-thread data collection technology are studied via the multi-thread and standard template library(STL)technologies.Thirdly,the system management functions,the structure design of the database,and the technical difficulties in the process of software design are also introduced.As proved in practice,the system satisfies the production management requirements of the workshop,and implements the network management of the production information.

A Generic Graph Model for WCET Analysis of Multi-Core Concurrent Applications

Worst-case execution time (WCET) analysis of multi-threaded software is still a challenge. This comes mainly from the fact that synchronization has to be taken into account. In this paper, we focus on this issue and on automatically calculating and incorporating stalling times (e.g. caused by lock contention) in a generic graph model. The idea that thread interleavings can be studied with a matrix calculus is novel in this research area. Our sparse matrix representations of the program are manipulated using an extended Kronecker algebra. The resulting graph represents multi-threaded programs similar as CFGs do for sequential programs. With this graph model, we are able to calculate the WCET of multi-threaded concurrent programs including stalling times which are due to synchronization. We employ a generating function-based approach for setting up data flow equations which are solved by well-known elimination-based dataflow analysis methods or an off-the-shelf equation solver. The WCET of multi-threaded programs can finally be calculated with a non-linear function solver.

An Approach to Parallel Simulation of Ordinary Differential Equations

Cyber-physical systems (CPS) represent a class of complex engineered systems where functionality and behavior emerge through the interaction between the computational and physical domains. Simulation provides design engineers with quick and accurate feedback on the behaviors generated by their designs. However, as systems become more complex, simulating their behaviors becomes computation all complex. But, most modern simulation environments still execute on a single thread, which does not take advantage of the processing power available on modern multi-core CPUs. This paper investigates methods to partition and simulate differential equation-based models of cyber-physical systems using multiple threads on multi-core CPUs that can share data across threads. We describe model partitioning methods using fixed step and variable step numerical in-tegration methods that consider the multi-layer cache structure of these CPUs to avoid simulation performance degradation due to cache conflicts. We study the effectiveness of each parallel simu-lation algorithm by calculating the relative speedup compared to a serial simulation applied to a series of large electric circuit models. We also develop a series of guidelines for maximizing performance when developing parallel simulation software intended for use on multi-core CPUs.

Minimal Context-Switching Data Race Detection with Dataflow Tracking

Data race is one of the most important concurrent anomalies in multi-threaded programs.Emerging con-straint-based techniques are leveraged into race detection,which is able to find all the races that can be found by any oth-er sound race detector.However,this constraint-based approach has serious limitations on helping programmers analyze and understand data races.First,it may report a large number of false positives due to the unrecognized dataflow propa-gation of the program.Second,it recommends a wide range of thread context switches to schedule the reported race(in-cluding the false one)whenever this race is exposed during the constraint-solving process.This ad hoc recommendation imposes too many context switches,which complicates the data race analysis.To address these two limitations in the state-of-the-art constraint-based race detection,this paper proposes DFTracker,an improved constraint-based race detec-tor to recommend each data race with minimal thread context switches.Specifically,we reduce the false positives by ana-lyzing and tracking the dataflow in the program.By this means,DFTracker thus reduces the unnecessary analysis of false race schedules.We further propose a novel algorithm to recommend an effective race schedule with minimal thread con-text switches for each data race.Our experimental results on the real applications demonstrate that 1)without removing any true data race,DFTracker effectively prunes false positives by 68%in comparison with the state-of-the-art constraint-based race detector;2)DFTracker recommends as low as 2.6-8.3(4.7 on average)thread context switches per data race in the real world,which is 81.6%fewer context switches per data race than the state-of-the-art constraint based race detec-tor.Therefore,DFTracker can be used as an effective tool to understand the data race for programmers.

Multi-threaded code generation from Signal program to OpenMP

The use of multi-core processors will become a trend in safety critical systems. For safe execution of multi- threaded code, automatic code generation from formal spec- ification is a desirable method. Signal, a synchronous lan- guage dedicated for the functional description of safety crit- ical systems, provides soundness semantics for determinis- tic concurrency. Although sequential code generation of Sig- nal has been implemented in Polychrony compiler, deter- ministic multi-threaded code generation strategy is still far from mature. Moreover, existing code generation methods use certain multi-thread library, which limits the cross plat- form executions. OpenMP is an application program inter- face （API） standard for parallel programming, supported by several mainstream compilers from different platforms. This paper presents a methodology translating Signal program to OpenMP-based multi-threaded C code. First, the intermedi- ate representation of the core syntax of Signal using syn- chronous guarded actions is defined. Then, according to the compositional semantics of Signal equations, the Signal pro- gram is synthesized to dependency graph （DG）. After par- allel tasks are extracted from dependency graph, the Signal program can be finally translated into OpenMP-based C code which can be executed on multiple platforms.

GPU Accelerated Discontinuous Galerkin Methods for Shallow Water Equations

We discuss the development,verification,and performance of a GPU accelerated discontinuous Galerkin method for the solutions of two dimensional nonlinear shallow water equations.The shallow water equations are hyperbolic partial differential equations and are widely used in the simulation of tsunami wave propagations.Our algorithms are tailored to take advantage of the single instruction multiple data(SIMD)architecture of graphic processing units.The time integration is accelerated by local time stepping based on a multi-rate Adams-Bashforth scheme.A total variational bounded limiter is adopted for nonlinear stability of the numerical scheme.This limiter is coupled with a mass and momentum conserving positivity preserving limiter for the special treatment of a dry or partially wet element in the triangulation.Accuracy,robustness and performance are demonstrated with the aid of test cases.Furthermore,we developed a unified multi-threading model OCCA.The kernels expressed in OCCA model can be cross-compiled with multi-threading models OpenCL,CUDA,and OpenMP.We compare the performance of the OCCA kernels when cross-compiled with these models.

Multiple Welding Seam Video Transmission and Monitoring of Welding Robot in Local Area Network

A method of transmission and monitoring in local area network(LAN) for the welding seam video information of binocular vision welding robot is proposed.This paper uses MPEG-4 video CODEC functions of DivX,video for Windows(VFW) video capture technology,multi-threaded network communication idea and WinSock Socket in Visual C++ 6.0.In the end,client/server(C/S) communication model based on TCP/IP protocol has been developed.Experimental results show that the method can achieve the multiple welding seam video's real-time transmission in LAN.Moreover,the Canny operator is used to realize the edge feature extraction of the seam image which is captured by the client,so that the edge of the seam extracted is continuous and clear.

An embedded lightweight GUI component library and ergonomics optimization method for industry process monitoring

Developing an efficient and robust lightweight graphic user interface （GUI） for industry process monitoring is always a challenging task. Current implementation methods for embedded GUI are with the matters of real-time processing and ergonomics performance. To address the issue, an embedded lightweight GUI component library design method based on quasar technology embedded （Qt/E） is proposed. First, an entity-relationship （E-R） model for the GUI library is developed to define the functional framework and data coupling relations. Second, a cross-compilation environment is constructed, and the QI/E shared library files are tailored to satisfy the requirements of embedded target systems. Third, by using the signal-slot communication interfaces, a message mapping mechanism that does not require a call-back pointer is developed, and the context switching performance is improved. According to the multi-thread method, the parallel task processing capabilities fbr data collection, calculation, and display are enhanced, and the real-time performance and robustness are guaranteed. Finally, the human-computer interaction process is optimized by a scrolling page method, and the ergonomics pertbrmance is verified by the industrial psychology methods Two numerical cases and five industrial experiments show that the proposed method can increase real-time read-write correction ratios by more than 26% and 29%, compared with Windows-CE-GUl and Android-GUl, respectively. The component library can be tailored to 900 KB and supports 12 hardware platforms. The average session switch time can be controlled within 0.6 s and six key indexes for ergonomics are verified by different industrial applications.