Neural Machine Translation Models with Attention-Based Dropout Layer

In bilingual translation,attention-based Neural Machine Translation(NMT)models are used to achieve synchrony between input and output sequences and the notion of alignment.NMT model has obtained state-of-the-art performance for several language pairs.However,there has been little work exploring useful architectures for Urdu-to-English machine translation.We conducted extensive Urdu-to-English translation experiments using Long short-term memory(LSTM)/Bidirectional recurrent neural networks(Bi-RNN)/Statistical recurrent unit(SRU)/Gated recurrent unit(GRU)/Convolutional neural network(CNN)and Transformer.Experimental results show that Bi-RNN and LSTM with attention mechanism trained iteratively,with a scalable data set,make precise predictions on unseen data.The trained models yielded competitive results by achieving 62.6%and 61%accuracy and 49.67 and 47.14 BLEU scores,respectively.From a qualitative perspective,the translation of the test sets was examined manually,and it was observed that trained models tend to produce repetitive output more frequently.The attention score produced by Bi-RNN and LSTM produced clear alignment,while GRU showed incorrect translation for words,poor alignment and lack of a clear structure.Therefore,we considered refining the attention-based models by defining an additional attention-based dropout layer.Attention dropout fixes alignment errors and minimizes translation errors at the word level.After empirical demonstration and comparison with their counterparts,we found improvement in the quality of the resulting translation system and a decrease in the perplexity and over-translation score.The ability of the proposed model was evaluated using Arabic-English and Persian-English datasets as well.We empirically concluded that adding an attention-based dropout layer helps improve GRU,SRU,and Transformer translation and is considerably more efficient in translation quality and speed.

Neural Machine Translation by Fusing Key Information of Text

When the Transformer proposed by Google in 2017,it was first used for machine translation tasks and achieved the state of the art at that time.Although the current neural machine translation model can generate high quality translation results,there are still mistranslations and omissions in the translation of key information of long sentences.On the other hand,the most important part in traditional translation tasks is the translation of key information.In the translation results,as long as the key information is translated accurately and completely,even if other parts of the results are translated incorrect,the final translation results’quality can still be guaranteed.In order to solve the problem of mistranslation and missed translation effectively,and improve the accuracy and completeness of long sentence translation in machine translation,this paper proposes a key information fused neural machine translation model based on Transformer.The model proposed in this paper extracts the keywords of the source language text separately as the input of the encoder.After the same encoding as the source language text,it is fused with the output of the source language text encoded by the encoder,then the key information is processed and input into the decoder.With incorporating keyword information from the source language sentence,the model’s performance in the task of translating long sentences is very reliable.In order to verify the effectiveness of the method of fusion of key information proposed in this paper,a series of experiments were carried out on the verification set.The experimental results show that the Bilingual Evaluation Understudy(BLEU)score of the model proposed in this paper on theWorkshop on Machine Translation(WMT)2017 test dataset is higher than the BLEU score of Transformer proposed by Google on the WMT2017 test dataset.The experimental results show the advantages of the model proposed in this paper.

Research on system combination of machine translation based on Transformer

Influenced by its training corpus,the performance of different machine translation systems varies greatly.Aiming at achieving higher quality translations,system combination methods combine the translation results of multiple systems through statistical combination or neural network combination.This paper proposes a new multi-system translation combination method based on the Transformer architecture,which uses a multi-encoder to encode source sentences and the translation results of each system in order to realize encoder combination and decoder combination.The experimental verification on the Chinese-English translation task shows that this method has 1.2-2.35 more bilingual evaluation understudy(BLEU)points compared with the best single system results,0.71-3.12 more BLEU points compared with the statistical combination method,and 0.14-0.62 more BLEU points compared with the state-of-the-art neural network combination method.The experimental results demonstrate the effectiveness of the proposed system combination method based on Transformer.

Improvements of Google Neural Machine Translation

Machine Translation has been playing an important role in modern society due to its effectiveness and efficiency,but the great demand for corpus makes it difficult for users to use traditional Machine Translation systems.To solve this problem and improve translation quality,in November 2016,Google introduces Google Neural Machine Translation system,which implements the latest techniques to achieve better outcomes.The conspicuous achievement has been proved by experiments using BLEU score to measure performance of different systems.With GNMT,the gap between human and machine translation is narrowing.

Progress in Machine Translation

After more than 70 years of evolution,great achievements have been made in machine translation.Especially in recent years,translation quality has been greatly improved with the emergence of neural machine translation(NMT).In this article,we first review the history of machine translation from rule-based machine translation to example-based machine translation and statistical machine translation.We then introduce NMT in more detail,including the basic framework and the current dominant framework,Transformer,as well as multilingual translation models to deal with the data sparseness problem.In addition,we introduce cutting-edge simultaneous translation methods that achieve a balance between translation quality and latency.We then describe various products and applications of machine translation.At the end of this article,we briefly discuss challenges and future research directions in this field.

Corpus Augmentation for Improving Neural Machine Translation

The translation quality of neural machine translation(NMT)systems depends largely on the quality of large-scale bilingual parallel corpora available.Research shows that under the condition of limited resources,the performance of NMT is greatly reduced,and a large amount of high-quality bilingual parallel data is needed to train a competitive translation model.However,not all languages have large-scale and high-quality bilingual corpus resources available.In these cases,improving the quality of the corpora has become the main focus to increase the accuracy of the NMT results.This paper proposes a new method to improve the quality of data by using data cleaning,data expansion,and other measures to expand the data at the word and sentence-level,thus improving the richness of the bilingual data.The long short-term memory(LSTM)language model is also used to ensure the smoothness of sentence construction in the process of sentence construction.At the same time,it uses a variety of processing methods to improve the quality of the bilingual data.Experiments using three standard test sets are conducted to validate the proposed method;the most advanced fairseq-transformer NMT system is used in the training.The results show that the proposed method has worked well on improving the translation results.Compared with the state-of-the-art methods,the BLEU value of our method is increased by 2.34 compared with that of the baseline.

Dependency-Based Local Attention Approach to Neural Machine Translation

Recently dependency information has been used in different ways to improve neural machine translation.For example,add dependency labels to the hidden states of source words.Or the contiguous information of a source word would be found according to the dependency tree and then be learned independently and be added into Neural Machine Translation(NMT)model as a unit in various ways.However,these works are all limited to the use of dependency information to enrich the hidden states of source words.Since many works in Statistical Machine Translation(SMT)and NMT have proven the validity and potential of using dependency information.We believe that there are still many ways to apply dependency information in the NMT structure.In this paper,we explore a new way to use dependency information to improve NMT.Based on the theory of local attention mechanism,we present Dependency-based Local Attention Approach(DLAA),a new attention mechanism that allowed the NMT model to trace the dependency words related to the current translating words.Our work also indicates that dependency information could help to supervise attention mechanism.Experiment results on WMT 17 Chineseto-English translation task shared training datasets show that our model is effective and perform distinctively on long sentence translation.

Improve Neural Machine Translation by Building Word Vector with Part of Speech

Neural Machine Translation(NMT)based system is an important technology for translation applications.However,there is plenty of rooms for the improvement of NMT.In the process of NMT,traditional word vector cannot distinguish the same words under different parts of speech(POS).Aiming to alleviate this problem,this paper proposed a new word vector training method based on POS feature.It can efficiently improve the quality of translation by adding POS feature to the training process of word vectors.In the experiments,we conducted extensive experiments to evaluate our methods.The experimental result shows that the proposed method is beneficial to improve the quality of translation from English into Chinese.

Evaluating Classification Research for Machine Translation Course Teaching

Teaching evaluation can be divided into different types,additionally their functions and applicable conditions are different.According to different standards,teaching evaluation can be divided into different types:(1)according to different evaluation functions,it can be divided into pre-evaluation,intermediate evaluation,and post-evaluation;(2)according to different evaluation reference standards,it can be divided into relative evaluation,absolute evaluation,and individual difference evaluation;(3)according to different evaluation and analysis methods,it can be divided into qualitative and quantitative evaluation;(4)according to the different evaluation subjects,it can be divided into self-evaluation and others’evaluation.This paper introduced research work using different types of teaching evaluation in the machine translation course according to different situations.The research results showed that the rational selection of different types of teaching evaluation methods and the combination of these methods can greatly promote teaching.

Integrating Deep Learning and Machine Translation for Understanding Unrefined Languages

In the field of natural language processing(NLP),the advancement of neural machine translation has paved the way for cross-lingual research.Yet,most studies in NLP have evaluated the proposed language models on well-refined datasets.We investigatewhether amachine translation approach is suitable for multilingual analysis of unrefined datasets,particularly,chat messages in Twitch.In order to address it,we collected the dataset,which included 7,066,854 and 3,365,569 chat messages from English and Korean streams,respectively.We employed several machine learning classifiers and neural networks with two different types of embedding:word-sequence embedding and the final layer of a pre-trained language model.The results of the employed models indicate that the accuracy difference between English,and English to Korean was relatively high,ranging from 3%to 12%.For Korean data(Korean,and Korean to English),it ranged from 0%to 2%.Therefore,the results imply that translation from a low-resource language(e.g.,Korean)into a high-resource language(e.g.,English)shows higher performance,in contrast to vice versa.Several implications and limitations of the presented results are also discussed.For instance,we suggest the feasibility of translation from resource-poor languages for using the tools of resource-rich languages in further analysis.

Korean Morphological Analysis for Korean-Vietnamese Statistical Machine Translation

This paper describes the experiments with Korean-to-Vietnamese statistical machine translation(SMT). The fact that Korean is a morphologically complex language that does not have clear optimal word boundaries causes a major problem of translating into or from Korean. To solve this problem, we present a method to conduct a Korean morphological analysis by using a pre-analyzed partial word-phrase dictionary(PWD).Besides, we build a Korean-Vietnamese parallel corpus for training SMT models by collecting text from multilingual magazines. Then, we apply such a morphology analysis to Korean sentences that are included in the collected parallel corpus as a preprocessing step. The experiment results demonstrate a remarkable improvement of Korean-to-Vietnamese translation quality in term of bi-lingual evaluation understudy(BLEU).

A Novel Beam Search to Improve Neural Machine Translation for English-Chinese

Neural Machine Translation(NMT)is an end-to-end learning approach for automated translation,overcoming the weaknesses of conventional phrase-based translation systems.Although NMT based systems have gained their popularity in commercial translation applications,there is still plenty of room for improvement.Being the most popular search algorithm in NMT,beam search is vital to the translation result.However,traditional beam search can produce duplicate or missing translation due to its target sequence selection strategy.Aiming to alleviate this problem,this paper proposed neural machine translation improvements based on a novel beam search evaluation function.And we use reinforcement learning to train a translation evaluation system to select better candidate words for generating translations.In the experiments,we conducted extensive experiments to evaluate our methods.CASIA corpus and the 1,000,000 pairs of bilingual corpora of NiuTrans are used in our experiments.The experiment results prove that the proposed methods can effectively improve the English to Chinese translation quality.

Towards consistent machine translation of abbreviated terms in scientific literature

Scientific literature often contains abbreviated terms in English for brief.Machine translation(MT)systems can help to share knowledge in different languages among researchers.Current MT systems may translate the same abbreviated term in different sentences into different target terms.MT systems translate the abbreviated term in two ways:one is to use translation of the full name,the other is to use the abbreviated term directly.Abbreviated terms may be ambiguous and polysemous,and MT systems do not have an explicit strategy to decide which way to use without context information.To get the consistent translation for abbreviated terms in scientific literature,this paper proposes a translation model for abbreviated terms that integrates context information to get consistent translation of abbreviated terms.The context information includes the positions of abbreviated term and domain attributes of scientific literature.The first abbreviated term is translated in full name while the latter ones of the same abbreviated term will show the abbreviated form in the translation text.Experiments of translation from Chinese to English show the effectiveness of the proposed translation model.

Improving Parallel Corpus Quality for Chinese-Vietnamese Statistical Machine Translation

The performance of a machine translation system heavily depends on the quantity and quality of the bilingual language resource. However,getting a parallel corpus,which has a large scale and is of high quality,is a very difficult task especially for low resource languages such as Chinese-Vietnamese. Fortunately,multilingual user generated contents（ UGC）,such as bilingual movie subtitles,provide us access to automatic construction of the parallel corpus. Although the amount of UGC parallel corpora can be considerable,the original corpus is not suitable for statistical machine translation（ SMT） systems. The corpus may contain translation errors,sentence mismatching,free translations,etc. To improve the quality of the bilingual corpus for SMT systems,three filtering methods are proposed： sentence length difference,the semantic of sentence pairs,and machine learning. Experiments are conducted on the Chinese to Vietnamese translation corpus.Experimental results demonstrate that all the three methods effectively improve the corpus quality,and the machine translation performance（ BLEU score） can be improved by 1. 32.

Japanese-Chinese Machine Translation of Japanese Determiners Based on Templates

The machine translation of Japanese sentences with determiners,like“shi­ka...nai”,“tyoutto...dakedeha”,“tada...dake”and so on,are more special and regular on sentences structure.The research collects and classifies the Japanese sentences which contain the determiners.The classification is carried out by according to the characteristics of Japanese sentences and translation habit of Chinese sentences.Through further abstraction and sim­plification,translation templates are extracted by gathering grammar rules information,studying syntax and analysis the collocation mode of sentenc­es.Those determiners express confirmed meaning,and the corresponding translation Chinese sentences have the same characteristic.By analyzing the sentence characteristics with determiners and formalizing the sentences struc­ture,the translation templates are abstracted.By investigating the structure characteristic of original sentences with translation templates,the similarity algorithm was defined.The threshold value of the similarity calculation was obtained by preliminary experiments,and the experiments of Japanese-Chi­nese translation are carried out by a small corpus.The experimental results for several kinds of Japanese sentences with determiners show the translation accuracy rate is 68.6%,template coverage rate reach 83.3%.At last,through the analysis for the translation errors,following conclusion is drawn:the results of morphological analysis are erroneous,because the error of word segmentation the part of speech tagging also are erroneous,result in the grammar structure cannot match with templates;the original sentences are long and especially complex sentences;the templates are too complicated;the similarity calculation method needs to discuss further,and so on.

A Research on the Function of Teaching Evaluation for Machine Translation Course

Teaching evaluation refers to the process of measuring and giving value judgment to the process and results of teaching activities by using effective technical means.Through teaching evaluation,teachers can direct the teaching process to develop toward the predetermined goal and effectively complete the teaching task.This paper investigates the function of teaching evaluation in the Machine Translation course in Northeastern University.The function of teaching evaluation for teachers includes adjustment,diagnosis,teaching stimulation,and orientation.The function of teaching evaluation for students includes feedback,guidance,intensification,and purpose orientation.This paper also discusses the influence of teaching evaluation on the psychology of both,teachers and students.Positive evaluation can improve the enthusiasm of teachers and students,but other times,it may reduce their enthusiasm,whereas when negative evaluation and guidance are appropriate,the enthusiasm of teachers and students may improve.This research reveals that a reasonable teaching evaluation plays a huge role in promoting the psychology of both,teachers and students.

Transformer: A General Framework from Machine Translation to Others

Machine translation is an important and challenging task that aims at automatically translating natural language sentences from one language into another.Recently,Transformer-based neural machine translation(NMT)has achieved great break-throughs and has become a new mainstream method in both methodology and applications.In this article,we conduct an overview of Transformer-based NMT and its extension to other tasks.Specifically,we first introduce the framework of Transformer,discuss the main challenges in NMT and list the representative methods for each challenge.Then,the public resources and toolkits in NMT are listed.Meanwhile,the extensions of Transformer in other tasks,including the other natural language processing tasks,computer vision tasks,audio tasks and multi-modal tasks,are briefly presented.Finally,possible future research directions are suggested.

Visual Topic Semantic Enhanced Machine Translation for Multi-Modal Data Efficiency

The scarcity of bilingual parallel corpus imposes limitations on exploiting the state-of-the-art supervised translation technology.One of the research directions is employing relations among multi-modal data to enhance perfor-mance.However,the reliance on manually annotated multi-modal datasets results in a high cost of data labeling.In this paper,the topic semantics of images is proposed to alleviate the above problem.First,topic-related images can be auto-matically collected from the Internet by search engines.Second,topic semantics is sufficient to encode the relations be-tween multi-modal data such as texts and images.Specifically,we propose a visual topic semantic enhanced translation(VTSE)model that utilizes topic-related images to construct a cross-lingual and cross-modal semantic space,allowing the VTSE model to simultaneously integrate the syntactic structure and semantic features.In the above process,topic similar texts and images are wrapped into groups so that the model can extract more robust topic semantics from a set of similar images and then further optimize the feature integration.The results show that our model outperforms competitive base-lines by a large margin on the Multi30k and the Ambiguous COCO datasets.Our model can use external images to bring gains to translation,improving data efficiency.

Research on high-performance English translation based on topic model

Retelling extraction is an important branch of Natural Language Processing(NLP),and high-quality retelling resources are very helpful to improve the performance of machine translation.However,traditional methods based on the bilingual parallel corpus often ignore the document background in the process of retelling acquisition and application.In order to solve this problem,we introduce topic model information into the translation mode and propose a topic-based statistical machine translation method to improve the translation performance.In this method,Probabilistic Latent Semantic Analysis(PLSA)is used to obtains the co-occurrence relationship between words and documents by the hybrid matrix decomposition.Then we design a decoder to simplify the decoding process.Experiments show that the proposed method can effectively improve the accuracy of translation.

Neural machine translation:Challenges,progress and future

Machine translation(MT)is a technique that leverages computers to translate human languages automatically.Nowadays,neural machine translation(NMT)which models direct mapping between source and target languages with deep neural networks has achieved a big breakthrough in translation performance and become the de facto paradigm of MT.This article makes a review of NMT framework,discusses the challenges in NMT,introduces some exciting recent progresses and finally looks forward to some potential future research trends.