Grammaticality and Language Modelling

TL;DR

We advocate the point-biserial correlation (PBC) for evaluating language models as grammaticality classifiers, since it directly compares a discrete variable (acceptability judgements) with a continuous one (LM probabilities). With the PBC in hand, we reappraise recent syntactically targeted evaluations (Hu et al., 2020), argue that GLUE’s use of the Matthews correlation coefficient on CoLA is problematic, and conduct PBC studies comparing popular language models, including the effects of normalization and data homogeneity.

How to Cite

@inproceedings{niu-penn-2020-grammaticality,
   title = "Grammaticality and Language Modelling",
   author = "Niu, Jingcheng and Penn, Gerald",
   booktitle = "Proceedings of the First Workshop on Evaluation and Comparison of NLP Systems",
   month = nov,
   year = "2020",
   address = "Online",
   publisher = "Association for Computational Linguistics"
}