Transformer Working Memory Enables Regular Language Reasoning And Natural Language Length Extrapolation

Ta Chung Chi; Ting Han Fan; Alexander I. Rudnicky; Peter J. Ramadge

Transformer Working Memory Enables Regular Language Reasoning And Natural Language Length Extrapolation

Ta Chung Chi, Ting Han Fan, Alexander I. Rudnicky, Peter J. Ramadge

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

Unlike recurrent models, conventional wisdom has it that Transformers cannot perfectly model regular languages. Inspired by the notion of working memory, we propose a new Transformer variant named RegularGPT. With its novel combination of Weight-Sharing, Adaptive-Depth, and Sliding-Dilated-Attention, RegularGPT constructs working memory along the depth dimension, thereby enabling efficient and successful modeling of regular languages such as PARITY. We further test RegularGPT on the task of natural language length extrapolation and surprisingly find that it rediscovers the local windowed attention effect deemed necessary in prior work for length extrapolation.

Original language	English (US)
Title of host publication	Findings of the Association for Computational Linguistics
Subtitle of host publication	EMNLP 2023
Publisher	Association for Computational Linguistics (ACL)
Pages	5972-5984
Number of pages	13
ISBN (Electronic)	9798891760615
State	Published - 2023
Event	2023 Findings of the Association for Computational Linguistics: EMNLP 2023 - Singapore, Singapore Duration: Dec 6 2023 → Dec 10 2023

Publication series

Name	Findings of the Association for Computational Linguistics: EMNLP 2023

Conference

Conference	2023 Findings of the Association for Computational Linguistics: EMNLP 2023
Country/Territory	Singapore
City	Singapore
Period	12/6/23 → 12/10/23

All Science Journal Classification (ASJC) codes

Computational Theory and Mathematics
Computer Science Applications
Information Systems
Language and Linguistics
Linguistics and Language

Cite this

Chi, T. C., Fan, T. H., Rudnicky, A. I., & Ramadge, P. J. (2023). Transformer Working Memory Enables Regular Language Reasoning And Natural Language Length Extrapolation. In Findings of the Association for Computational Linguistics: EMNLP 2023 (pp. 5972-5984). (Findings of the Association for Computational Linguistics: EMNLP 2023). Association for Computational Linguistics (ACL).

Chi, Ta Chung ; Fan, Ting Han ; Rudnicky, Alexander I. et al. / Transformer Working Memory Enables Regular Language Reasoning And Natural Language Length Extrapolation. Findings of the Association for Computational Linguistics: EMNLP 2023. Association for Computational Linguistics (ACL), 2023. pp. 5972-5984 (Findings of the Association for Computational Linguistics: EMNLP 2023).

@inproceedings{1c33db482c98407f83b24180aa8a3f15,

title = "Transformer Working Memory Enables Regular Language Reasoning And Natural Language Length Extrapolation",

abstract = "Unlike recurrent models, conventional wisdom has it that Transformers cannot perfectly model regular languages. Inspired by the notion of working memory, we propose a new Transformer variant named RegularGPT. With its novel combination of Weight-Sharing, Adaptive-Depth, and Sliding-Dilated-Attention, RegularGPT constructs working memory along the depth dimension, thereby enabling efficient and successful modeling of regular languages such as PARITY. We further test RegularGPT on the task of natural language length extrapolation and surprisingly find that it rediscovers the local windowed attention effect deemed necessary in prior work for length extrapolation.",

author = "Chi, {Ta Chung} and Fan, {Ting Han} and Rudnicky, {Alexander I.} and Ramadge, {Peter J.}",

note = "Publisher Copyright: {\textcopyright} 2023 Association for Computational Linguistics.; 2023 Findings of the Association for Computational Linguistics: EMNLP 2023 ; Conference date: 06-12-2023 Through 10-12-2023",

year = "2023",

language = "English (US)",

series = "Findings of the Association for Computational Linguistics: EMNLP 2023",

publisher = "Association for Computational Linguistics (ACL)",

pages = "5972--5984",

booktitle = "Findings of the Association for Computational Linguistics",

}

Chi, TC, Fan, TH, Rudnicky, AI & Ramadge, PJ 2023, Transformer Working Memory Enables Regular Language Reasoning And Natural Language Length Extrapolation. in Findings of the Association for Computational Linguistics: EMNLP 2023. Findings of the Association for Computational Linguistics: EMNLP 2023, Association for Computational Linguistics (ACL), pp. 5972-5984, 2023 Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore, Singapore, 12/6/23.

Transformer Working Memory Enables Regular Language Reasoning And Natural Language Length Extrapolation. / Chi, Ta Chung; Fan, Ting Han; Rudnicky, Alexander I. et al.
Findings of the Association for Computational Linguistics: EMNLP 2023. Association for Computational Linguistics (ACL), 2023. p. 5972-5984 (Findings of the Association for Computational Linguistics: EMNLP 2023).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - Transformer Working Memory Enables Regular Language Reasoning And Natural Language Length Extrapolation

AU - Chi, Ta Chung

AU - Fan, Ting Han

AU - Rudnicky, Alexander I.

AU - Ramadge, Peter J.

PY - 2023

Y1 - 2023

N2 - Unlike recurrent models, conventional wisdom has it that Transformers cannot perfectly model regular languages. Inspired by the notion of working memory, we propose a new Transformer variant named RegularGPT. With its novel combination of Weight-Sharing, Adaptive-Depth, and Sliding-Dilated-Attention, RegularGPT constructs working memory along the depth dimension, thereby enabling efficient and successful modeling of regular languages such as PARITY. We further test RegularGPT on the task of natural language length extrapolation and surprisingly find that it rediscovers the local windowed attention effect deemed necessary in prior work for length extrapolation.

AB - Unlike recurrent models, conventional wisdom has it that Transformers cannot perfectly model regular languages. Inspired by the notion of working memory, we propose a new Transformer variant named RegularGPT. With its novel combination of Weight-Sharing, Adaptive-Depth, and Sliding-Dilated-Attention, RegularGPT constructs working memory along the depth dimension, thereby enabling efficient and successful modeling of regular languages such as PARITY. We further test RegularGPT on the task of natural language length extrapolation and surprisingly find that it rediscovers the local windowed attention effect deemed necessary in prior work for length extrapolation.

UR - http://www.scopus.com/inward/record.url?scp=85183293682&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85183293682&partnerID=8YFLogxK

M3 - Conference contribution

AN - SCOPUS:85183293682

T3 - Findings of the Association for Computational Linguistics: EMNLP 2023

SP - 5972

EP - 5984

BT - Findings of the Association for Computational Linguistics

PB - Association for Computational Linguistics (ACL)

T2 - 2023 Findings of the Association for Computational Linguistics: EMNLP 2023

Y2 - 6 December 2023 through 10 December 2023

ER -

Transformer Working Memory Enables Regular Language Reasoning And Natural Language Length Extrapolation

Abstract

Publication series

Conference

All Science Journal Classification (ASJC) codes

Other files and links

Fingerprint

Cite this