Institute Homepage Sign In
SHARE
Empirical Inference Conference Paper 2026

On the Emergence and Test-Time Use of Structural Information in Large Language Models

arXiv

Learning structural information from observational data is central to producing new knowledge outside the training corpus. This holds for mechanistic understanding in scientific discovery as well as flexible test-time compositional generation. We thus study how language models learn abstract structures and utilize the learnt structural information at test-time. To ensure a controlled setup, we design a natural language dataset based on linguistic structural transformations. We empirically show that the emergence of learning structural information correlates with complex reasoning tasks, and that the ability to perform test-time compositional generation remains limited.

Empirical Inference
Moritz Miller
Empirical Inference
Bernhard Schölkopf
Empirical Inference
Siyuan Guo
Author(s): Chen, M. C. and Miller, M. and Schölkopf, B. and Guo, S.
Links:
Book Title: Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
Pages: 1449--1465
Year: 2026
Month: July
Editors: Liakata, Maria and Moreira, Viviane P. and Zhang, Jiajun and Jurgens, David
Publisher: Association for Computational Linguistics
BibTeX Type: Conference Paper (conference)
Event Name: 64th Annual Meeting of the Association for Computational Linguistics (ACL 2026)
Event Place: San Diego, California
State: Published
URL: https://aclanthology.org/2026.acl-long.65/

BibTeX 

@conference{CheMilSchGuo26,
  title = {On the Emergence and Test-Time Use of Structural Information in Large Language Models},
  booktitle = {Proceedings of the 64th Annual Meeting of the {A}ssociation for {C}omputational {L}inguistics (Volume 1: Long Papers)},
  abstract = {Learning structural information from observational data is central to producing new knowledge outside the training corpus. This holds for mechanistic understanding in scientific discovery as well as flexible test-time compositional generation. We thus study how language models learn abstract structures and utilize the learnt structural information at test-time. To ensure a controlled setup, we design a natural language dataset based on linguistic structural transformations. We empirically show that the emergence of learning structural information correlates with complex reasoning tasks, and that the ability to perform test-time compositional generation remains limited.},
  pages = {1449--1465},
  editors = {Liakata, Maria and Moreira, Viviane P. and Zhang, Jiajun and Jurgens, David},
  publisher = {Association for Computational Linguistics},
  month = jul,
  year = {2026},
  author = {Chen, M. C. and Miller, M. and Sch{\"o}lkopf, B. and Guo, S.},
  url = {https://aclanthology.org/2026.acl-long.65/},
  month_numeric = {7}
}