大语言模型及其在政务领域的应用

Large language models and their application in government affairs

大语言模型是自然语言处理的核心研究内容之一,已广泛应用于包括政务在内的诸多领域。首先,介绍了统计语言模型、神经网络语言模型等早期语言模型的研究进展;其次,重点综述了大语言模型研究进展;最后,介绍了大语言模型在政务领域的应用情况,包括政务文本分类、政务问答、政务命名实体识别、舆情风险识别和政务关系抽取,并提出政务大语言模型研究需要解决的问题,即数据多模态化、正确面对“模型即服务”趋势、注重数据高安全性、明确责任边界。此外,还提出了政务大语言模型研究的技术路径。大语言模型正处于蓬勃发展的阶段,随着中国推动人工智能技术研究及其在政务领域的应用,大语言模型将在政务领域发挥更大作用。

[Significance]Since the turn of the 21st century,artificial intelligence(AI)has advanced considerably in many domains,including government affairs.Furthermore,the emergence of deep learning has taken the development of many AI fields,including natural language processing(NLP),to a new level.Language models(LMs)are key research directions of NLP.Referred to as statistical models,LMs were initially used to calculate the probability of a sentence;however,in recent years,there have been substantial developments in large language models(LLMs).Notably,LLM products,such as the generative pretrained transformer(GPT)series,have driven the rapid revolution of large language research.Domestic enterprises have also researched LLMs,for example,Huawei's Pangu and Baidu's enhanced language representation with informative entities(ERNIE)bot.These models have been widely used in language translation,abstract construction,named-entity recognition,text classification,and relationship extraction,among other applications,and in government affairs,finance,biomedicine,and other domains.[Progress]In this study,we observe that improving the efficiency of governance has become one of the core tasks of the government in the era of big data.With the continuous accumulation of government data,traditional statistical models relying on expert experience and local features gradually suffer limitations during application.However,LLMs,which offer the advantages of high flexibility,strong representation ability,and effective results,can rapidly enhance the intelligence level of government services.First,we review the research progress on early LMs,such as statistical LMs and neural network LMs.Subsequently,we focus on the research progress on LLMs,namely the Transformers series,GPT series,and bidirectional encoder representations from transformers(BERT)series.Finally,we introduce the application of LLMs in government affairs,including government text classification,relationship extraction,public opinion risk identification,named-entity recognition,and government question answering.Moreover,we propose that research on LLMs for government affairs must focus on multimodality,correctly benefit from the trend of“model as a service,”focus on high data security,and clarify government responsibility boundaries.Additionally,a technical path for studying LLMs for government affairs has been proposed.[Conclusions and Prospects]The application of LLMs in government affairs mainly focuses on small-scale models,lacking examples of application in large-scale models.Compared with smaller models,large models offer many advantages,including high efficiency,broader application scenarios,and more convenience.These advantages can be understood as follows.In terms of efficiency,large models are usually trained on a large amount of heterogeneous data,thus delivering better performance.In terms of application scenarios,large models gradually support multimodal data,resulting in more diverse application scenarios.In terms of convenience,we emphasize the“pretraining+fine-tuning”mode and the invocation method of interfaces,making LLMs more convenient for research and practical applications.This study also analyzes the issues suffered by LLMs,specifically from the technological and ethical perspectives,which have resulted in a panic to a certain extent.For example,ChatGPT has generated many controversies,including whether the generated files are novel,whether using ChatGPT will lead to plagiarism and ambiguity as to who are property rights owners for the generated files.Overall,it can be said that LLMs are in the stage of vigorous development.As the country promotes research on AI and its application in government affairs,LLMs will play an increasingly crucial role in the field.