Integrating Audio-Visual Features and Text Information for Story Segmentation of News Video

Video data are composed of multimodal information streams including visual, auditory and textual streams, so an approach of story segmentation for news video using multimodal analysis is described in this paper. The proposed approach detects the topic-caption frames, and integrates them with silence clips detection results, as well as shot segmentation results to locate the news story boundaries. The integration of audio-visual features and text information overcomes the weakness of the approach using only image analysis techniques. On test data with 135 400 frames, when the boundaries between news stories are detected, the accuracy rate 85.8% and the recall rate 97.5% are obtained. The experimental results show the approach is valid and robust.

Audio-visual emotion recognition with multilayer boosted HMM

Emotion recognition has become an important task of modern human-computer interac- tion. A multilayer boosted HMM （ MBHMM ） classifier for automatic audio-visual emotion recognition is presented in this paper. A modified Baum-Welch algorithm is proposed for component HMM learn- ing and adaptive boosting （AdaBoost） is used to train ensemble classifiers for different layers （cues）. Except for the first layer, the initial weights of training samples in current layer are decided by recognition results of the ensemble classifier in the upper layer. Thus the training procedure using current cue can focus more on the difficult samples according to the previous cue. Our MBHMM clas- sifier is combined by these ensemble classifiers and takes advantage of the complementary informa- tion from multiple cues and modalities. Experimental results on audio-visual emotion data collected in Wizard of Oz scenarios and labeled under two types of emotion category sets demonstrate that our approach is effective and promising.

On‐device audio‐visual multi‐person wake word spotting

Audio‐visual wake word spotting is a challenging multi‐modal task that exploits visual information of lip motion patterns to supplement acoustic speech to improve overall detection performance.However,most audio‐visual wake word spotting models are only suitable for simple single‐speaker scenarios and require high computational complexity.Further development is hindered by complex multi‐person scenarios and computational limitations in mobile environments.In this paper,a novel audio‐visual model is proposed for on‐device multi‐person wake word spotting.Firstly,an attention‐based audio‐visual voice activity detection module is presented,which generates an attention score matrix of audio and visual representations to derive active speaker representation.Secondly,the knowledge distillation method is introduced to transfer knowledge from the large model to the on‐device model to control the size of our model.Moreover,a new audio‐visual dataset,PKU‐KWS,is collected for sentence‐level multi‐person wake word spotting.Experimental results on the PKU‐KWS dataset show that this approach outperforms the previous state‐of‐the‐art methods.

The Effect of Audio Visual Entrainment on Pre-Attentive Dysfunctional Processing to Stressful Events in Anxious Individuals

Experimental single case studies on automatic processing of emotion were carried on a sample of people with an anxiety disorder. Participants were required to take three Audio Visual Entrainment (AVE) sessions to test for anxiety reduction as proclaimed by some academic research. Explicit reports were measured as well as pre-attentive bias to stressing information by using affective priming studies before and after AVE intervention. Group analysis shows that indeed AVEs program applications do reduce anxiety producing significant changes over explicit reports on anxiety levels and automatic processing bias of emotion. However, case by case analysis of six anxious participants shows that even when all of the participants report emotional improvement after intervention, not all of them reduce or eliminate dysfunctional bias to stressing information. Rather, they show a variety of processing styles due to intervention and some of them show no change at all. Implications of this differential effect to clinical sets are discussed.

Prioritized MPEG-4 Audio-Visual Objects Streaming over the DiffServ

The object-based scalable coding in MPEG-4 is investigated, and a prioritized transmission scheme of MPEG-4 audio-visual objects （AVOs） over the DiffServ network with the QoS guarantee is proposed. MPEG-4 AVOs are extracted and classified into different groups according to their priority values and scalable layers （visual importance）. These priority values are mapped to the 1P DiffServ per hop behaviors （PHB）. This scheme can selectively discard packets with low importance, in order to avoid the network congestion. Simulation results show that the quality of received video can gracefully adapt to network state, as compared with the ‘best-effort＇ manner. Also, by allowing the content provider to define prioritization of each audio-visual object, the adaptive transmission of object-based scalable video can be customized based on the content.

Microsoft Agent语音技术在Visual Basic中的处理

充分调用Microsoft Agent技术,增添了多媒体应用软件的智能性、趣味性和人性化特征。通过对微软公司Agent角色开发文档及国内外相关文献的研究,分析了Agent的LanguageID、TTSModeID和SRModeID等的语音关键特征。借助于Add()方法和Command事件添加语音命令实现了在VB中自由进行语音识别与输出。另外,必须安装SAPI4.0运行期支持文件才能在Windows XP坏境下进行输出语音。

基于Visual C++6.0的声音文件处理技术

对WAVE声音文件的格式进行了分析 ,对在VisualC ++6.0下进行声音文件处理的技术进行了深入的探讨 ,并给出了部分示例代码。

基于Visual Basic的局域网实时语音系统的设计

详细描述使用Visual Basic编程工具,通过调用低级音频函数,实现局域网内计算机间的实时语音通信的方法。通过引入发送令牌和问候机制实现简单的流量控制,经分析和实践,达到比较理想的效果,就语音通信模式的扩展提出对策。

二重情境:数字视听文化中的身份构建与认同疏离

数字视听媒介及其文化的发展创造了新的情境,推动用户产生新的媒介实践,但也带来了新的问题和现象,即情境生成逻辑向流媒体用户让渡的转变、情境功能作为交往工具的偏移,以及情境秩序在去公共性过程中权力再结构化的取向。其中隐含着情境之于用户身份构建的二重性:一方面,流媒体用户依据情境构建身份,身份的构建过程进一步激发了情境的创造;另一方面,这些被构建的身份呈现出去情境化的流动趋向,造成了身份与认同的疏离。对此,文章在厘清数字视听文化新情境特征的基础上,阐释流媒体用户基于身份的情境互动过程,以及身份认同疏离的成因及危机。更进一步,文章尝试将新媒介—新情境—新行为的线性模式延展为更具解释力的用户主导循环模式,并提出通过共识的凝聚、公共性的重拾和共同体的重建,推动数字视听文化的良序发展。

Visual Design and Searching of Multimedia Percussion Theatre ＂The Call from Sigangli A Dialogue of Natural Character and Avant-garde＂

Multimedia Percussion Theatre ＂The Call from Sigangli--A Dialogue of Natural Character and Avant-garde＂ tried comprehensive practice and searching from visual and audio design. Visual and audio, this two kinds of vocabulary brought out the best in each other with the support of multimedia and digital audio technology, and also formed a new audio-visual language. The original ecological of percussion, multimedia image and interactive technologies impacted the natural and avant-garde. It is the possibility that this practice provides new form in the spread of Chinese culture.

基于ⅢF A/V规范和Avalon系统的大学图书馆视听数据库建设研究

随着中国网络基础设施的不断改善,视听媒体在年轻一代中非常流行,给以文本资源为主的图书馆带来了挑战。本研究旨在探究国内外大学图书馆视听资源数据库建设的现状,借鉴ⅢF规范在图像资源管理方面的成功经验和各种视听保存社区的实践,提出基于ⅢF A/V规范与开源软件的中国大学图书馆视听资源管理方法。通过分析华东师范大学图书馆在视听资源保存、流媒体发布、时间轴气泡注释、转录、视听结构化和开放共享方面的实践,进行实证研究。

网络视听视域下中国式现代化图景的现实表征与构建逻辑

随着信息技术的发展,网络视听已成为文化传播与日常生活的重要媒介,也日益成为文化生产的重要领域。网络视听在创新主体、核心根脉和重要目标等方面,与中国式现代化有着深刻而自洽的内涵关系,包括以人民为中心的影像赋权、彰显中国特色的文化“双创”以及打造共同富裕的产业驱动等先天优势。围绕中国式现代化的五大特征,网络视听空间以丰富的案例多维度地反映和塑造了中国式现代化的生动图景。究其背后的生成逻辑,实为网络视听承担了五种身份,即作为“引领者”弘扬主流价值,作为“记录者”打造多模态中国文化符号,作为“讲述者”激发情感共鸣的时代语态,作为“参与者”构建多元互动的社会场景,作为“驱动者”正确处理技术赋能与艺术实践的关系。网络视听媒体以其突出的视听艺术特性和共创共享的互联网品质,正在成为讲好中国式现代化故事的重要力量。

基于视音互补语义清晰化的隐私视频动作识别方法

视频隐私保护是当前社会面临的重要挑战之一,对视频进行模糊处理是保护人们隐私权益的重要手段.由于模糊视频天然缺失视觉模态的信息,主流的视频动作识别算法无法取得令人满意的效果.模糊视频作为多模态介质不仅仅只有视觉模态信息,同时,也含有丰富的音频模态信息,从人类的认知角度而言,音频也是获取信息的重要来源.本文提出一种基于多模态融合的隐私视频动作识别方法,在保证不侵犯使用者隐私的前提下进行人类动作行为识别.具体来说,使用音频-视觉特征融合模块将音频模态特征图融入到视觉模态中,充分融合音视频模态的深层语义信息.除此之外,模型还引入清晰视频帧图像作为标签,在模型训练阶段监督动作识别网络的参数更新,为隐私视频动作识别网络提供清晰的语义信息.在多组隐私行为数据集上,通过大量消融和对比实验验证了所提方法的有效性.

基于扩张卷积和Transformer的视听融合语音分离方法

为了提高语音分离的效果,除了利用混合的语音信号,还可以借助视觉信号作为辅助信息。这种融合了视觉与音频信号的多模态建模方式,已被证实可以有效地提高语音分离的性能,为语音分离任务提供了新的可能性。为了更好地捕捉视觉与音频特征中的长期依赖关系,并强化网络对输入上下文信息的理解,本文提出了一种基于一维扩张卷积与Transformer的时域视听融合语音分离模型。将基于频域的传统视听融合语音分离方法应用到时域中,避免了时频变换带来的信息损失和相位重构问题。所提网络架构包含四个模块:一个视觉特征提取网络,用于从视频帧中提取唇部嵌入特征;一个音频编码器,用于将混合语音转换为特征表示;一个多模态分离网络,主要由音频子网络、视频子网络,以及Transformer网络组成,用于利用视觉和音频特征进行语音分离;以及一个音频解码器,用于将分离后的特征还原为干净的语音。本文使用LRS2数据集生成的包含两个说话者混合语音的数据集。实验结果表明,所提出的网络在尺度不变信噪比改进(Scale-Invariant Signal-to-Noise Ratio Improvement,SISNRi)与信号失真比改进(Signal-to-Distortion Ratio Improvement,SDRi)这两种指标上分别达到14.0 dB与14.3 dB,较纯音频分离模型和普适的视听融合分离模型有明显的性能提升。

视听语言智能康复技术对语言发育迟缓儿童语言功能的影响

目的 探究视听语言智能康复技术对语言发育迟缓儿童语言功能的影响,为语言发育迟缓儿童的语言功能训练提供临床参考依据。方法 选取2022年1月~2023年3月兰州大学第二医院收治的80例语言发育迟缓儿童,随机分为研究组和对照组各40例。对照组进行常规语言训练,研究组在常规语言训练基础上联合使用视听语言智能康复技术,治疗3个月。治疗前后采用Gesell发育量表评估两组的适应性能力、语言能力、个人社交能力,并采用ICF(international classification of functioning disability and health)早期语言障碍评估与干预治疗仪对儿童词语、句子理解能力进行评估。结果 治疗前两组患儿适应性能力、语言能力、个人社交能力3大能区发育商以及词语理解、句子理解能力得分无显著差异(P>0.05)。治疗后研究组适应性能力、语言能力、个人社交能力3大能区发育商显著提高(P<0.05)。两组患儿词语理解、句子理解能力得分均显著提高,且研究组显著高于对照组(P<0.05)。结论 针对语言发育迟缓儿童语言能力的改善,常规语言训练结合视听语言智能康复技术比单纯常规语言训练效果更好。

基于多元读写能力发展的英语视听说多模态教学模式构建与应用

现代信息技术的发展对人们的交际形式提出了更高的要求。作为语言技能课程,“大学英语视听说”不仅应关注学生的听力理解和口语交际能力,还应注重培养学生在交际过程中的个人素质、信息处理、技术操作、批判创新等多元读写能力。英语视听说多模态教学依托多元读写教学法提出的情景实践、明确指导、批评框定、转化实践教学流程,构建多模态语篇体验、知识技能内化、多模态语篇评析、多模态语篇构建四个教学环节。实践证明,该教学模式能够有效提升学生的多元读写能力。

智能计算时代深化电影科技创新和推进电影强国建设的思考与启示

人类社会正加快步入智能计算时代,现代智能科技对电影产业的影响持续深化和不断泛化,机器学习、深度学习、人工智能生成内容(AIGC)、大语言模型(LLM)、多模态模型等技术正在深刻影响和重构优化电影全产业链全价值链。伴随电影科技创新进入攻坚期和深水区,电影产业发展面临新形势新要求,我们要深化电影科技自主创新,积极推进电影行业的智能化升级,推进生成式人工智能、人工智能大模型、沉浸式视听等技术在我国电影行业的定制设计、自主研制与科学应用,有力支撑和服务社会主义电影强国建设。

视听文本机器翻译质量测评和译后编辑研究——以《畅游赣鄱——水文化英文之旅》慕课字幕翻译为例

机器翻译给人们带来了便利,同时促成了译后编辑的诞生。本文以《畅游赣鄱——水文化英文之旅》慕课字幕文本为研究样本,在MQM(Multidimensional Quality Metrics)质量评估框架基础上提出视听文本机器翻译产品质量评估标准,量化其错误值,测试机器翻译质量,并由此提出译后编辑的三级模式:轻度译后编辑(Light Post-editing, LPE)、中度译后编辑(Moderate Post-editing, MPE)和重度译后编辑(Full Post-editing, FPE)。