Identifying visible actions in lifestyle vlogs

Oana Ignat; Laura Burdick; Jia Deng; Rada Mihalcea

Identifying visible actions in lifestyle vlogs

Oana Ignat, Laura Burdick, Jia Deng, Rada Mihalcea

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

We consider the task of identifying human actions visible in online videos. We focus on the widely spread genre of lifestyle vlogs, which consist of videos of people performing actions while verbally describing them. Our goal is to identify if actions mentioned in the speech description of a video are visually present. We construct a dataset with crowdsourced manual annotations of visible actions, and introduce a multimodal algorithm that leverages information derived from visual and linguistic clues to automatically infer which actions are visible in a video. We demonstrate that our multimodal algorithm outperforms algorithms based only on one modality at a time.

Original language	English (US)
Title of host publication	ACL 2019 - 57th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference
Publisher	Association for Computational Linguistics (ACL)
Pages	6406-6417
Number of pages	12
ISBN (Electronic)	9781950737482
State	Published - 2020
Event	57th Annual Meeting of the Association for Computational Linguistics, ACL 2019 - Florence, Italy Duration: Jul 28 2019 → Aug 2 2019

Publication series

Name	ACL 2019 - 57th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference

Conference

Conference	57th Annual Meeting of the Association for Computational Linguistics, ACL 2019
Country/Territory	Italy
City	Florence
Period	7/28/19 → 8/2/19

All Science Journal Classification (ASJC) codes

Language and Linguistics
General Computer Science
Linguistics and Language

Cite this

Ignat, O., Burdick, L., Deng, J., & Mihalcea, R. (2020). Identifying visible actions in lifestyle vlogs. In ACL 2019 - 57th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference (pp. 6406-6417). (ACL 2019 - 57th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference). Association for Computational Linguistics (ACL).

Ignat, Oana ; Burdick, Laura ; Deng, Jia et al. / Identifying visible actions in lifestyle vlogs. ACL 2019 - 57th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference. Association for Computational Linguistics (ACL), 2020. pp. 6406-6417 (ACL 2019 - 57th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference).

@inproceedings{ab28227c54e34dbe9694d5741a4f0983,

title = "Identifying visible actions in lifestyle vlogs",

abstract = "We consider the task of identifying human actions visible in online videos. We focus on the widely spread genre of lifestyle vlogs, which consist of videos of people performing actions while verbally describing them. Our goal is to identify if actions mentioned in the speech description of a video are visually present. We construct a dataset with crowdsourced manual annotations of visible actions, and introduce a multimodal algorithm that leverages information derived from visual and linguistic clues to automatically infer which actions are visible in a video. We demonstrate that our multimodal algorithm outperforms algorithms based only on one modality at a time.",

author = "Oana Ignat and Laura Burdick and Jia Deng and Rada Mihalcea",

note = "Publisher Copyright: {\textcopyright} 2019 Association for Computational Linguistics; 57th Annual Meeting of the Association for Computational Linguistics, ACL 2019 ; Conference date: 28-07-2019 Through 02-08-2019",

year = "2020",

language = "English (US)",

series = "ACL 2019 - 57th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference",

publisher = "Association for Computational Linguistics (ACL)",

pages = "6406--6417",

booktitle = "ACL 2019 - 57th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference",

}

Ignat, O, Burdick, L, Deng, J & Mihalcea, R 2020, Identifying visible actions in lifestyle vlogs. in ACL 2019 - 57th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference. ACL 2019 - 57th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference, Association for Computational Linguistics (ACL), pp. 6406-6417, 57th Annual Meeting of the Association for Computational Linguistics, ACL 2019, Florence, Italy, 7/28/19.

Identifying visible actions in lifestyle vlogs. / Ignat, Oana; Burdick, Laura; Deng, Jia et al.
ACL 2019 - 57th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference. Association for Computational Linguistics (ACL), 2020. p. 6406-6417 (ACL 2019 - 57th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - Identifying visible actions in lifestyle vlogs

AU - Ignat, Oana

AU - Burdick, Laura

AU - Deng, Jia

AU - Mihalcea, Rada

PY - 2020

Y1 - 2020

N2 - We consider the task of identifying human actions visible in online videos. We focus on the widely spread genre of lifestyle vlogs, which consist of videos of people performing actions while verbally describing them. Our goal is to identify if actions mentioned in the speech description of a video are visually present. We construct a dataset with crowdsourced manual annotations of visible actions, and introduce a multimodal algorithm that leverages information derived from visual and linguistic clues to automatically infer which actions are visible in a video. We demonstrate that our multimodal algorithm outperforms algorithms based only on one modality at a time.

AB - We consider the task of identifying human actions visible in online videos. We focus on the widely spread genre of lifestyle vlogs, which consist of videos of people performing actions while verbally describing them. Our goal is to identify if actions mentioned in the speech description of a video are visually present. We construct a dataset with crowdsourced manual annotations of visible actions, and introduce a multimodal algorithm that leverages information derived from visual and linguistic clues to automatically infer which actions are visible in a video. We demonstrate that our multimodal algorithm outperforms algorithms based only on one modality at a time.

UR - http://www.scopus.com/inward/record.url?scp=85084050022&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85084050022&partnerID=8YFLogxK

M3 - Conference contribution

AN - SCOPUS:85084050022

T3 - ACL 2019 - 57th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference

SP - 6406

EP - 6417

BT - ACL 2019 - 57th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference

PB - Association for Computational Linguistics (ACL)

T2 - 57th Annual Meeting of the Association for Computational Linguistics, ACL 2019

Y2 - 28 July 2019 through 2 August 2019

ER -

Ignat O, Burdick L, Deng J, Mihalcea R. Identifying visible actions in lifestyle vlogs. In ACL 2019 - 57th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference. Association for Computational Linguistics (ACL). 2020. p. 6406-6417. (ACL 2019 - 57th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference).

Identifying visible actions in lifestyle vlogs

Abstract

Publication series

Conference

All Science Journal Classification (ASJC) codes

Other files and links

Fingerprint

Cite this