LM Pub Quiz
Measure and compare the factual knowledge of language models!
LM Pub Quiz
We sometimes need to better understand a language model (LM). What kind of knowledge does it contain? How does it compare to other LMs?
With LM Pub Quiz, we present a new approach for automatically probing LMs for their factual knowledge.
LM Pub Quiz uses facts from WikiData, such as that Kampala is the capital of Uganda. For each fact, we create plausible detractors, i.e. false statements such as that Thimpu is the capital of Uganda. We use LMs to rank sentence expressing a verbalization of the true fact and all detractors. If the LM models knowledge well, it should rank the true fact the highest.
This gives us an idea of what type of knowledge they contain, and allows us to compare different LMs with regards to their knowledge. Our current work is leveraging this probe to investigate the sample-efficiency of LMs (how quickly they learn new information) and how much they forget in continual learning settings.
Getting Started
- Check out the LM Pub Quiz pageand our leaderboard of popular LMs!
- Check out our github repo!
Publications
BEAR: A Unified Framework for Evaluating Relational Knowledge in Causal and Masked Language Models.Jacek Wiland, Max Ploner and Alan Akbik. NAACL 2024.
LM-PUB-QUIZ: A Comprehensive Framework for Zero-Shot Evaluation of Relational Knowledge in Language Models.Max Ploner, Jacek Wiland, Sebastian Pohl and Alan Akbik.