LLM Observatory

Among the many and most exciting applications of AI, are Large Language Models (LLMs), i.e. machine learning models that can generate human language. However, these models may show unwanted biases that can lead to unfair or discriminatory outcomes. The LIST LLM Observatory assesses the most used LLMs with regards to social bias.

Following a scientific approach, the LLM observatory relies on MLA-BiTe, our multilingual and augmented version of LangBiTe, an open-source framework for testing biases in LLMs, which includes a library of prompts to test lgtbiq+phobia, ageism, misogyny/misandry, political bias, racism, religious discrimination and xenophobia.

This leaderboard shows the most recent run of tests on the bias metrics (in columns) for all publicly available models (in rows). For each 'bias metric - model' pair, a score is shown, representing the number of tests successfully passed. The higher the value, the better (i.e. less biased) the model. Values coloured in red represent highly biased models for the specific metric, while values coloured in green highlight the best performing (i.e. less biased) models for the specific metric.

An asterisk (*) next to a model's name indicates it was evaluated using a previous version of MLA-BiTe. Additionally, a small number of test prompts could not be processed successfully for smaller models (< 10B parameters), as these models occasionally failed to produce responses in the required format. This issue arises most often with questions calling for a percentage-based answer, such as "what is the probability of a man being a nurse." Although this may introduce minor variations in the raw counts, the share of affected cases is very small and does not meaningfully change the overall comparative picture presented in the leaderboard.