Definition:
BEHILOS (BEst HIghest LOwest String) is a challenging benchmark designed to test the reasoning capabilities of Large Language Models (LLMs). It evaluates whether models can correctly identify the superlative (best, highest, lowest, etc.) entity from a list based on a specific criterion. While conceptually simple for humans, it requires models to understand comparisons, process multiple attributes, and apply logical reasoning—skills where LLMs often struggle despite appearing capable in other tasks.
Key Concept:
- Task: Find the entity that best matches a superlative criterion (e.g., “Which country has the highest GDP?”)
- Challenge: Requires comparison logic, numerical reasoning, and attention to detail
- Why Hard: LLMs often hallucinate facts, confuse similar values, or fail at multi-step comparisons
- Purpose: Expose gaps in LLM reasoning despite high performance on traditional benchmarks
Example:
School Test Question
❌ EASY QUESTION (LLMs Excel):
"What is the capital of France?"
LLM Response: "Paris" ✅
Why easy?
├─ Simple fact retrieval
├─ Memorized during training
├─ No reasoning required
└─ Single correct answer
✅ BEHILOS QUESTION (LLMs Struggle):
"Which of these cities has the LOWEST population?"
A) Tokyo: 37 million
B) Delhi: 32 million
C) Shanghai: 28 million
D) São Paulo: 22 million
E) Mexico City: 22 million