Speaker naming in movies

Mahmoud Azab; Mingzhe Wang; Max Smith; Noriyuki Kojima; Jia Deng; Rada Mihalcea

Speaker naming in movies

Mahmoud Azab, Mingzhe Wang, Max Smith, Noriyuki Kojima, Jia Deng, Rada Mihalcea

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

We propose a new model for speaker naming in movies that leverages visual, textual, and acoustic modalities in an unified optimization framework. To evaluate the performance of our model, we introduce a new dataset consisting of six episodes of the Big Bang Theory TV show and eighteen full movies covering different genres. Our experiments show that our multimodal model significantly outperforms several competitive baselines on the average weighted F-score metric. To demonstrate the effectiveness of our framework, we design an end-To-end memory network model that leverages our speaker naming model and achieves state-of-The-Art results on the subtitles task of the MovieQA 2017 Challenge.

Original language	English (US)
Title of host publication	Long Papers
Publisher	Association for Computational Linguistics (ACL)
Pages	2206-2216
Number of pages	11
ISBN (Electronic)	9781948087278
State	Published - 2018
Externally published	Yes
Event	2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL HLT 2018 - New Orleans, United States Duration: Jun 1 2018 → Jun 6 2018

Publication series

Name	NAACL HLT 2018 - 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies - Proceedings of the Conference
Volume	1

Conference

Conference	2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL HLT 2018
Country/Territory	United States
City	New Orleans
Period	6/1/18 → 6/6/18

All Science Journal Classification (ASJC) codes

Linguistics and Language
Language and Linguistics
Computer Science Applications

Cite this

@inproceedings{11fa32a90681422ca15a6bed4b32b3ac,

title = "Speaker naming in movies",

abstract = "We propose a new model for speaker naming in movies that leverages visual, textual, and acoustic modalities in an unified optimization framework. To evaluate the performance of our model, we introduce a new dataset consisting of six episodes of the Big Bang Theory TV show and eighteen full movies covering different genres. Our experiments show that our multimodal model significantly outperforms several competitive baselines on the average weighted F-score metric. To demonstrate the effectiveness of our framework, we design an end-To-end memory network model that leverages our speaker naming model and achieves state-of-The-Art results on the subtitles task of the MovieQA 2017 Challenge.",

author = "Mahmoud Azab and Mingzhe Wang and Max Smith and Noriyuki Kojima and Jia Deng and Rada Mihalcea",

note = "Publisher Copyright: {\textcopyright} 2018 The Association for Computational Linguistics.; 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL HLT 2018 ; Conference date: 01-06-2018 Through 06-06-2018",

year = "2018",

language = "English (US)",

series = "NAACL HLT 2018 - 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies - Proceedings of the Conference",

publisher = "Association for Computational Linguistics (ACL)",

pages = "2206--2216",

booktitle = "Long Papers",

}

Azab, M, Wang, M, Smith, M, Kojima, N, Deng, J & Mihalcea, R 2018, Speaker naming in movies. in Long Papers. NAACL HLT 2018 - 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies - Proceedings of the Conference, vol. 1, Association for Computational Linguistics (ACL), pp. 2206-2216, 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL HLT 2018, New Orleans, United States, 6/1/18.

Speaker naming in movies. / Azab, Mahmoud; Wang, Mingzhe; Smith, Max et al.
Long Papers. Association for Computational Linguistics (ACL), 2018. p. 2206-2216 (NAACL HLT 2018 - 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies - Proceedings of the Conference; Vol. 1).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - Speaker naming in movies

AU - Azab, Mahmoud

AU - Wang, Mingzhe

AU - Smith, Max

AU - Kojima, Noriyuki

AU - Deng, Jia

AU - Mihalcea, Rada

PY - 2018

Y1 - 2018

N2 - We propose a new model for speaker naming in movies that leverages visual, textual, and acoustic modalities in an unified optimization framework. To evaluate the performance of our model, we introduce a new dataset consisting of six episodes of the Big Bang Theory TV show and eighteen full movies covering different genres. Our experiments show that our multimodal model significantly outperforms several competitive baselines on the average weighted F-score metric. To demonstrate the effectiveness of our framework, we design an end-To-end memory network model that leverages our speaker naming model and achieves state-of-The-Art results on the subtitles task of the MovieQA 2017 Challenge.

AB - We propose a new model for speaker naming in movies that leverages visual, textual, and acoustic modalities in an unified optimization framework. To evaluate the performance of our model, we introduce a new dataset consisting of six episodes of the Big Bang Theory TV show and eighteen full movies covering different genres. Our experiments show that our multimodal model significantly outperforms several competitive baselines on the average weighted F-score metric. To demonstrate the effectiveness of our framework, we design an end-To-end memory network model that leverages our speaker naming model and achieves state-of-The-Art results on the subtitles task of the MovieQA 2017 Challenge.

UR - http://www.scopus.com/inward/record.url?scp=85061738416&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85061738416&partnerID=8YFLogxK

M3 - Conference contribution

AN - SCOPUS:85061738416

T3 - NAACL HLT 2018 - 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies - Proceedings of the Conference

SP - 2206

EP - 2216

BT - Long Papers

PB - Association for Computational Linguistics (ACL)

T2 - 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL HLT 2018

Y2 - 1 June 2018 through 6 June 2018

ER -

Speaker naming in movies

Abstract

Publication series

Conference

All Science Journal Classification (ASJC) codes

Other files and links

Fingerprint

Cite this