An automated end-to-end pipeline for fine-grained video annotation using deep neural networks

Baptist Vandersmissen, Lucas Sterckx, Thomas Demeester, Azarakhsh Jalalvand, Wesley De Neve, Rik Van De Walle

Research output: Chapter in Book/Report/Conference proceedingConference contribution

2 Scopus citations

Abstract

The searchability of video content is often limited to the descriptions authors and/or annotators care to provide. The level of description can range from absolutely nothing to fine-grained annotations at the level of frames. Based on these annotations, certain parts of the video content are more searchable than others. Within the context of the STEAMER project, we developed an innovative end-to-end system that attempts to tackle the problem of unsupervised retrieval of news video content, leveraging multiple information streams and deep neural networks. In particular, we extracted keyphrases and named entities from transcripts, subsequently refining these keyphrases and named entities based on their visual appearance in the news video content. Moreover, to allow for fine-grained frame-level annotations, we temporally located high-confidence keyphrases in the news video content. To that end, we had to tackle challenges such as the automatic construction of training sets and the automatic assessment of keyphrase imageability. In this paper, we discuss the main components of our end- To-end system, capable of transforming textual and visual information into fine-grained video annotations.

Original languageEnglish (US)
Title of host publicationICMR 2016 - Proceedings of the 2016 ACM International Conference on Multimedia Retrieval
PublisherAssociation for Computing Machinery, Inc
Pages409-412
Number of pages4
ISBN (Electronic)9781450343596
DOIs
StatePublished - Jun 6 2016
Externally publishedYes
Event6th ACM International Conference on Multimedia Retrieval, ICMR 2016 - New York, United States
Duration: Jun 6 2016Jun 9 2016

Publication series

NameICMR 2016 - Proceedings of the 2016 ACM International Conference on Multimedia Retrieval

Conference

Conference6th ACM International Conference on Multimedia Retrieval, ICMR 2016
Country/TerritoryUnited States
CityNew York
Period6/6/166/9/16

All Science Journal Classification (ASJC) codes

  • Computer Graphics and Computer-Aided Design
  • Human-Computer Interaction
  • Software

Keywords

  • Deep neural networks
  • Fine-grained video annotation
  • Video retrieval

Fingerprint

Dive into the research topics of 'An automated end-to-end pipeline for fine-grained video annotation using deep neural networks'. Together they form a unique fingerprint.

Cite this