符号学视角下的远程视觉示意图:来自两个多模态语料库的洞见

Semiotically-grounded Distant Viewing of Diagrams:Insights from Two Multimodal Corpora

多模态研究探索人类如何依赖各种有机结合的表达资源进行交流,已建构为一门新兴学科。本文认为,当下数字人文的研究结合多模态研究的洞察力对可视材料进行大规模分析是有益的举措。多模态研究开发的概念为大型语料库和数据集中的各种表意模态提供了合适的元数据方案。本文使用常见的表意模态和示意图来阐释文中提出的方法,并运用统计和计算的方法分析两个新建的多模态语料库。研究结果表明,由多模态驱动的元数据方案可以为大型语料库和数据集的计算分析提供坚实基础。即使是语料库或数据集的设计不支持全面分析多模态交际,多模态理论依然可以用于对各种视觉材料进行比较严格的分析。

In this article,we argue for the benefits of combining large-scale analyses of visual materials currently pursued within digital humanities with insights from multimodality research,which is an emerging discipline that studies how human communication relies on appropriate combinations of expressive resources.We show that concepts developed within the field of multimodality research provide appropriate metadata schemes for various modes of expression in large corpora and datasets.We illustrate the proposed approach using a common mode of expression,diagrams,and analyse two recent multimodal diagram corpora using statistical and computational methods.Our results suggest that multimodally-motivated metadata schemes can provide a robust foundation for computational analyses of large corpora and datasets.Even if a corpus or dataset is not designed to support fullblown analyses of multimodal communication,our results imply that multimodality theory can still be used to impose tighter analytical control over a variety of visual materials.