基于普通器件实现快1000倍的相机与机器视觉

在数码相机中,我们发现了一个重大缺陷,即从胶片相机继承的图像和视频模型阻碍了相机捕捉快速变化的光子世界。我们提出了一种新的视觉形式,称为视象(vform),这是一个比特序列阵列,其中每个比特表示光子的累积是否达到了一个阈值,从而可以记录和重建任何时刻场景的光强。仅使用消费级CMOS传感器和集成电路,开发了一种比传统相机快1000倍的脉冲相机。将视象看作生物视觉中的脉冲序列,进一步开发了基于脉冲神经网络的机器视觉系统,它可以将机器的速度和生物视觉的机理结合起来,从而实现了比人类视觉快1000倍的高速目标检测和跟踪,并通过辅助裁判和目标瞄准系统证明了脉冲相机和超级视觉系统的效用。视象模型和芯片有望从根本上改变图像和视频的概念以及摄影、电影和视觉媒体等相关行业,并开启一个全新的基于脉冲神经网络的速度自由的机器视觉时代。

Toward the Next Generation of Retinal Neuroprosthesis: Visual Computation with Spikes

A neuroprosthesis is a type of precision medical device that is intended to manipulate the neuronal signals of the brain in a closed-loop fashion,while simultaneously receiving stimuli from the environment and controlling some part of a human brain or body.Incoming visual information can be processed by the brain in millisecond intervals.The retina computes visual scenes and sends its output to the cortex in the form of neuronal spikes for further computation.Thus,the neuronal signal of interest for a retinal neuroprosthesis is the neuronal spike.Closed-loop computation in a neuroprosthesis includes two stages:encoding a stimulus as a neuronal signal,and decoding it back into a stimulus.In this paper,we review some of the recent progress that has been achieved in visual computation models that use spikes to analyze natural scenes that include static images and dynamic videos.We hypothesize that in order to obtain a better understanding of the computational principles in the retina,a hypercircuit view of the retina is necessary,in which the different functional network motifs that have been revealed in the cortex neuronal network are taken into consideration when interacting with the retina.The different building blocks of the retina,which include a diversity of cell types and synaptic connections-both chemical synapses and electrical synapses(gap junctions)-make the retina an ideal neuronal network for adapting the computational techniques that have been developed in artificial intelligence to model the encoding and decoding of visual scenes.An overall systems approach to visual computation with neuronal spikes is necessary in order to advance the next generation of retinal neuroprosthesis as an artificial visual system.

Cluster structure prediction via CALYPSO method

Cluster science as a bridge linking atomic molecular physics and condensed matter inspired the nanomaterials development in the past decades, ranging from the single-atom catalysis to ligand-protected noble metal clusters. The corresponding studies not only have been restricted to the search for the geometrical structures of clusters, but also have promoted the development of cluster-assembled materials as the building blocks. The CALYPSO cluster prediction method combined with other computational techniques have significantly stimulated the development of the cluster-based nanomaterials. In this review, we will summarize some good cases of cluster structure by CALYPSO method, which have also been successfully identified by the photoelectron spectra experiments. Beginning with the alkali-metal clusters, which serve as benchmarks, a series of studies are performed on the size-dependent elemental clusters which possess relatively high stability and interesting chemical physical properties. Special attentions are paid to the boron-based clusters because of their promising applications. The NbSi12 and BeB16 clusters, for example, are two classic representatives of the silicon-and boron-based clusters, which can be viewed as building blocks of nanotubes and borophene. This review offers a detailed description of the structural evolutions and electronic properties of medium-sized pure and doped clusters, which will advance fundamental knowledge of cluster-based nanomaterials and provide valuable information for further theoretical and experimental studies.

Introduction to AVS2 Scene Video Coding Techniques

The second generation Audio Video Coding Standard (AVS2) is the most recent video coding standard. By introducing several new coding techniques, AVS2 can provide more efficient compression for scene videos such as surveillance videos, conference videos, etc. Due to the limited scenes, scene videos have great redundancy especially in background region. The new scene video coding techniques applied in AVS2 mainly focus on reducing redundancy in order to achieve higher compression. This paper introduces several important AVS2 scene video coding techniques. Experimental results show that with scene video coding tools, AVS2 can save nearly 40%BD?rate (Bj?ntegaard?Delta bit?rate) on scene videos.

Parsing Objects at a Finer Granularity: A Survey

Fine-grained visual parsing, including fine-grained part segmentation and fine-grained object recognition, has attracted considerable critical attention due to its importance in many real-world applications, e.g., agriculture, remote sensing, and space technologies. Predominant research efforts tackle these fine-grained sub-tasks following different paradigms, while the inherent relations between these tasks are neglected. Moreover, given most of the research remains fragmented, we conduct an in-depth study of the advanced work from a new perspective of learning the part relationship. In this perspective, we first consolidate recent research and benchmark syntheses with new taxonomies. Based on this consolidation, we revisit the universal challenges in fine-grained part segmentation and recognition tasks and propose new solutions by part relationship learning for these important challenges. Furthermore, we conclude several promising lines of research in fine-grained visual parsing for future research.

Large-scale Multi-modal Pre-trained Models: A Comprehensive Survey

With the urgent demand for generalized deep models,many pre-trained big models are proposed,such as bidirectional encoder representations(BERT),vision transformer(ViT),generative pre-trained transformers(GPT),etc.Inspired by the success of these models in single domains(like computer vision and natural language processing),the multi-modal pre-trained big models have also drawn more and more attention in recent years.In this work,we give a comprehensive survey of these models and hope this paper could provide new insights and helps fresh researchers to track the most cutting-edge works.Specifically,we firstly introduce the background of multi-modal pre-training by reviewing the conventional deep learning,pre-training works in natural language process,computer vision,and speech.Then,we introduce the task definition,key challenges,and advantages of multi-modal pre-training models(MM-PTMs),and discuss the MM-PTMs with a focus on data,objectives,network architectures,and knowledge enhanced pre-training.After that,we introduce the downstream tasks used for the validation of large-scale MM-PTMs,including generative,classification,and regression tasks.We also give visualization and analysis of the model parameters and results on representative downstream tasks.Finally,we point out possible research directions for this topic that may benefit future works.In addition,we maintain a continuously updated paper list for large-scale pre-trained multi-modal big models:https://github.com/wangxiao5791509/MultiModal_BigModels_Survey.

数字视网膜:智慧城市系统演进的关键环节

本文阐述了作者对智慧城市建设和发展的主要观点:(1)如何实时聚合各类城市大数据,特别是来自视频监控网络的图像视频数据,并通过构建基于云计算的"城市大脑"来分析和挖掘大数据价值并服务于城市运营与管理,是智慧城市发展中亟待解决的一个关键问题.(2)现阶段智慧城市建设的现状是"有眼、有脑",但作为"眼睛"的摄像头功能过于单一使得"脑强眼弱",其根源在于传统监控摄像机网络所采用的技术体系是为存储而不是分析设计的.尽管近期有些智能摄像头具有车牌或人脸识别功能,但是这种单纯强调"边缘计算"的方案仍然无法解决"眼脑合一"的问题.(3)为了解决目前阻碍智慧城市系统功能快速演进的难题,我们应借鉴人类进化了数十万年的视觉系统之"人类视网膜同时具有影像编码与特征编码功能"这一特性,研究与设计数字视网膜,使之具有统一时间戳和精确地理位置,能同时进行高效视频编码和紧凑特征表达的联合优化,并有效支持云端大规模监控视频分析与快速视觉搜索等功能.(4)为利用数字视网膜来构筑智慧城市的"慧眼",应积极布局与推进相关标准制定、芯片与硬件实现、支撑软件开发与软硬件开源社区,并开展大规模测试与应用.

Perspective:Societally connected multimedia across cultures

The advance of the Internet in the past decade has radically changed the way people communicate and col- laborate with each other. Physical distance is no more a barrier in online social networks, but cultural differences (at the individual, community, as well as societal levels) still govern human-human interactions and must be con- sidered and leveraged in the online world. The rapid de-ployment of high-speed Internet allows humans to interact using a rich set of multimedia data such as texts, pictures, and videos. This position paper proposes to define a new research area called 'connected multimedia', which is the study of a collection of research issues of the super-area social media that receive little attention in the literature. By connected multimedia, we mean the study of the social and technical interactions among users, multimedia data, and devices across cultures and explicitly exploiting the cultural differences. We justify why it is necessary to bring attention to this new research area and what benefits of this new research area may bring to the broader scientific research community and the humanity.