Attenuating Catastrophic Forgetting of Token-Level Knowledge in Contrastive Sentence Embeddings with Attentive Pooling

TL;DR

Contrastive post-training à la SimCSE improves Semantic Textual Similarity (STS) but causes “catastrophic forgetting of token-level knowledge,” hurting transfer tasks. GLOOCoSE, a layer- and token-wise attentive pooling mechanism, eliminates this trade-off: it boosts transfer-task performance while maintaining — and in some cases improving — STS performance, and it is compatible with various SimCSE variants.

How to Cite

@article{Saifei2025Attenuating,
  author = {Liao, Saifei and Niu, Jingcheng and Penn, Gerald},
  journal = {Proceedings of the Canadian Conference on Artificial Intelligence},
  year = {2025},
  month = {may 19},
  note = {https://caiac.pubpub.org/pub/n7sbt42t},
  publisher = {Canadian Artificial Intelligence Association (CAIAC)},
  title = {Attenuating {Catastrophic} {Forgetting} of {Token}-{Level} {Knowledge} in {Contrastive} {Sentence} {Embeddings} with {Attentive} {Pooling}},
}