2608.07411-GeoBenchLLM: A Comprehensive Benchmark for Evaluating LLMs on Geo-Related Tasks

Introduction to the show: ident: Paper Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Next we'll be talking about the paper "GeoBenchLLM: A Comprehensive Benchmark for Evaluating LLMs on Geo-Related Tasks".

Jane: The paper was written by Rodrigo Ferreira Rodrigues, Karim Radouane, Jose G Moreno and Lynda Tamine from University of Toulouse.

Tom: Stay tuned as we take you through the paper and discuss its implications.

Paper summary: Tom: So we've got a new benchmark paper in the geo-eye space, and I have to say, this one feels like it fills a real gap. The team from Toulouse pulled together a benchmark called GeoBenchLLM that spans twelve datasets and over four hundred thousand questions, all about geographic tasks.

Jane: Four hundred twenty-one thousand questions, and they cover everything from simple factoid questions like coordinates to complex pathfinding. That's a serious scale jump compared to earlier benchmarks, where you'd see maybe a few thousand examples at best.

Lu: What I find striking is the three-level cognitive structure they borrowed from earlier work — Knowledge, Reasoning, Application. It's not just about whether a model knows Paris is north of Toulouse, it's about whether it can reason through spatial relationships and then apply that to real tasks like recommending a point of interest.

Meng: And the key finding? They compared Qwen models from 0 point 6 billion parameters up to 8 billion, with and without a "thinking" mode, and then compared against GPT-OSS models at 20 and 120 billion. The result shows that reasoning matters enormously — an 8-billion-parameter model with thinking enabled actually beats a 120-billion-parameter model on several reasoning-heavy tasks.

Tom: Right, on the PPNL multi-pathfinding subdataset, the 8-billion Qwen with thinking hit 0 point 62 accuracy while the 120-billion model only reached 0 point 57. That's a remarkable result, because the usual story is that bigger models just win. Here, for tasks that really require spatial-temporal reasoning, thinking mode can close — and sometimes overturn — the size gap.

Jane: But it's not uniform. On knowledge-level tasks like coordinates prediction, the larger GPT-OSS models still dominate by a wide margin. That tells us that remembering geographic facts is largely about how much world knowledge is baked into the parameters, while reasoning is a skill that smaller models can activate if they're allowed to think step by step.

Lalam: The broader implication here is that benchmarks need to separate these cognitive levels carefully. If you lump factual recall and spatial reasoning together, you get a muddy picture of what models can actually do. This paper's taxonomy and its new evaluation metrics for open-ended generation really push the field toward a more granular understanding.

Lu: And they made the whole thing public, with metrics available through Hugging Face and the code on GitHub. That's important — a benchmark is only useful if the community can build on it, reproduce the numbers, and extend it.

Meng: The paper also introduces custom metrics for things like coordinates accuracy with a tolerance radius, precision and recall for place lists, and compliance ratios for pathfinding tasks. Those are genuinely needed because existing evaluation tools were designed for closed-form answers, while many geo-tasks require open-ended generation.

Tom: Exactly — and that's where a lot of earlier benchmarks fell short. They restricted themselves to yes/no or multiple-choice questions because those are easy to grade. This work deliberately embraces the messy, generative setting and builds the tools to grade it fairly.

Jane: So we're looking at a benchmark that not only scales up the data, but scales up the difficulty of evaluation itself. The next question is how they actually assembled these twelve datasets and what transformations were needed — that's where the real work shows.

Page 1 of the paper: Lu: So we've set the stage with the big numbers and the main finding. Let's pull back the first page and look at how they actually positioned the work against what came before, because the motivation is carefully argued.

Tom: Right, they start with the claim that geo-tasks are genuinely hard for question-answering systems. It's not just about knowing facts, but about geometric uncertainty and the vagueness of everyday language about locations. If I ask "is Paris north of Toulouse," that's easy for a human, but the model has to handle the spatial relationship between two real entities.

Jane: And the existing benchmarks each have their own weakness. GeoBenchmark only covers knowledge and reasoning, with no application level. CityEval is limited to an urban context and mostly multiple-choice. STBench is large but too focused on spatial reasoning tasks and misses coordinates prediction and regression. And the Xu benchmark is diverse but only has nine hundred questions.

Lu: That's the key gap — either too narrow in scope, too small in scale, or locked into formats that don't reflect how people actually ask questions. The paper's comparison table makes that crystal clear, showing which benchmarks cover which cognitive levels, formats, and scales.

Meng: I also notice they specifically point out that several benchmarks only use yes/no or multiple-choice formats. That's a serious limitation, because it gives the model a tiny set of options and doesn't test its ability to generate a free-form answer. Real users don't get four choices when they ask where something is or how far apart two cities are.

Tom: And that's why they emphasize their benchmark includes all four formats — generative, regression, yes/no, and multiple-choice. That breadth forces the evaluation to be more honest about what models can and cannot do.

Jane: The scale difference is worth dwelling on. Their benchmark has 421,041 examples, while the largest existing benchmark, STBench, has around 80,000. That's a five-fold increase, and it matters because evaluation on a bigger, more diverse set reduces the variance of the results and gives you more confidence that a model's performance is real.

Lalam: There's also a careful choice in how they define the three cognitive levels. Knowledge tasks are factoid questions answerable by querying a geographic database — coordinates, real numbers, yes/no, place names. Reasoning tasks require applying those concepts, like spatial reasoning with distance and topology, or complex scenario QA. Application tasks go further, requiring recommendation or pathfinding in real-world settings.

Lu: That taxonomy gives the benchmark an internal structure that allows for a more nuanced analysis. You can ask whether thinking mode helps equally across levels, or whether model size matters more at certain levels. And indeed, their results show exactly that — thinking closes the gap most dramatically on reasoning and application tasks, not on knowledge recall.

Meng: The authors also mention that they adopted this classification from the Xu et al. benchmark, but they expanded it, added more tasks, and scaled it up. So it's a continuity with prior work rather than a brand-new framework, which makes comparison across benchmarks more meaningful.

Tom: And they previewed the key finding on that very first page — that models up to 120 billion parameters can succeed, but smaller models can close the gap when thinking is enabled. It's a bold claim to put right up front, and the rest of the paper is them backing it up.

Jane: Now, the next page takes us into the related work and the detailed motivation, where they go through the individual datasets and benchmarks one by one. There's a lot of texture there about why each existing resource falls short.

Page 2 of the paper: Lu: We've covered the big-picture motivation. Now the paper walks through the related work in detail, and that's where it gets interesting because you can see the history of how geo-evaluation evolved.

Meng: They start with GeoBenchmark, which focuses on direction, distance, and topology using data from YAGO2geo and Ordnance Survey geometries. That's a solid knowledge and reasoning test, but it has no application tasks at all — no recommendation, no pathfinding.

Tom: Then there's STBench, which they credit with around 80,000 author-generated questions derived from the Yelp dataset. The focus there is on temporal characteristics of geographic questions, but it lacks variety and misses coordinates prediction and complex scenario QA.

Jane: The Xu benchmark is more diverse in task coverage but has only 900 questions. Even though those are high-quality questions drawn from Wikipedia and geography textbooks, that's just too few to provide a comprehensive assessment. You can't make strong claims about a model's geographic capability based on nine hundred examples.

Lu: And CityEval, part of the CityGPT framework, has a large set of questions covering most tasks except regression and complex scenario QA, but everything is stuck in an urban context. That limits its generalizability to world scale.

Meng: The paper also discusses the individual datasets that they eventually incorporated. There's bAbI, which is a classic from Facebook designed to test language models on various QA tasks — but only two of its twenty tasks are geographic, namely positional reasoning and pathfinding.

Tom: And then there's MapQA, which is interesting because it's focused on places of interest retrieved from OpenStreetMap for Southern California and Illinois. It asks questions like predicting a place name given its amenity type and spatial relationship to another place, plus regression questions about distances between POIs.

Jane: What I appreciate here is that they're not just criticizing — they're carefully identifying what each resource can contribute. They're building their benchmark from twelve existing datasets, some of which were never originally designed for LLM evaluation, and they're transforming them to make them suitable.

Lu: For example, GeoQuestions1089 was originally designed for natural language to SPARQL translation. They processed and cleaned the raw query responses to extract useful data for their coordinate, yes/no, regression, and place prediction subdatasets.

Meng: And there's a transparency note about TourismQA — the original code couldn't be used to regenerate the dataset, so they retrieved it from a subsequent work that used it. That kind of honest reporting about provenance is important for reproducibility.

Tom: They also applied transformations from prior works where necessary and partitioned some datasets into subdatasets to prevent overlap across tasks. That's a crucial design choice, because if a dataset contains both yes/no questions and place prediction questions, you don't want to contaminate your task definitions.

Jane: So on this page, we're seeing the nuts and bolts of what it takes to assemble a comprehensive benchmark. It's not just collecting files — it's about understanding each dataset's original purpose, adapting it to a new evaluation setting, and being transparent about what was changed and why.

Lu: And the outcome is seventeen textual subdatasets covering eight tasks across three cognitive levels. That granularity is what allows the paper's analysis to show which cognitive skills respond to model size versus thinking mode.

Meng: Now, the next page gets into the actual datasets beyond knowledge tasks — the reasoning and application ones — and that's where things get innovative, especially with synthetic spatial reasoning datasets like StepGame.

Page 3 of the paper: Lu: So the previous page set up the datasets for knowledge tasks — coordinates, yes/no, regression, and place prediction. Now the paper introduces the reasoning and application datasets, and that's where the benchmark gets really demanding.

Tom: For reasoning, we have GeoSQA and GKMC, both drawn from the Chinese Gaokao geography exams. These are scenario-based multiple-choice questions, and they were translated into English using Google Translate. You have to appreciate the scale of that — GeoSQA has over 4,000 questions, and GKMC has tens of thousands.

Meng: But the more novel additions are SpatialEvalLLM, SpartUN, and StepGame. SpatialEvalLLM places the model in a grid of objects and asks it to identify the object at the end of a described path. SpartUN builds scenarios with objects connected topologically or directionally and asks either boolean or relational questions.

Jane: StepGame is particularly clever — it's synthetic, and the model has to infer the directional relationship between two points from intermediate placements. The questions are categorized by how many reasoning hops are required, so you can literally see where models start to fail as the chain of reasoning gets longer.

Lu: Now the application level is where the paper pushes into real-world scenarios. TourismQA is a POI recommendation dataset built from tourist reviews across fifty cities worldwide. Given a tourist question and available reviews, the model has to predict relevant points of interest.

Meng: And NY-POI is derived from Foursquare check-ins, with the task being to predict the next POI in a user's trajectory based on their habits and visit history. That's a spatial-temporal prediction problem that requires understanding user behavior alongside geography.

Tom: The pathfinding datasets are the real test of applied reasoning though. GridRoute asks the model to return a valid sequence of adjacent grid coordinates from point A to point B, avoiding obstacles and without diagonal movements. PPNL extends that to a multi-objective setting where you must pass through an unordered list of intermediate points.

Jane: That's where their evaluation metrics become essential, because grading a path isn't a simple right-or-wrong judgment. They distinguish between a path being feasible — staying within grid boundaries and avoiding obstacles — versus successful, meaning it reaches the goal, versus optimal, meaning it does so in the minimum number of moves.

Lu: The optimal ratio is their main metric because it's the most discriminating. A model can produce lots of feasible paths that wander around, but only a truly competent reasoner will find the shortest route.

Meng: And they also handle the edge case of an unreachable goal. The unreachable accuracy metric measures how often the model correctly detects that a goal is impossible to reach, which tests whether the model truly understands the grid constraints rather than just pattern-matching on training data.

Tom: There's also a compliance ratio for whether the model even outputs the right format — a list of grid coordinates. That's a subtle but important point. If a model gives you a perfectly reasoned path in prose but not in the requested format, it should be marked down, because the task explicitly asks for a structured output.

Jane: And for the place prediction tasks, they adapted precision and recall to handle lists of geographic coordinates. It's not just about whether one predicted point matches one reference — you need to account for multiple reference points and multiple predictions, and the distance between them matters.

Lu: They introduce a metric called the compliance ratio, and separately they use Bleu-1 and BERT-Score for the text-based recommendation tasks like TourismQA. Bleu-1 is their main metric because it's more discriminating than BERT-Score, which tends to give high scores even when the model is using different words.

Meng: The beauty of these metrics is that they can be reused. They put them in a Hugging Face collection so other researchers can evaluate their own models with the same tools, which is a real contribution to the community beyond just the benchmark data.

Tom: So the third page gives us the full landscape of tasks and the evaluation machinery. The tables there spell out the exact numbers — trains, devs, tests, average word counts — for all seventeen subdatasets. It's a lot of detail, but it makes the benchmark concrete and reproducible.

Jane: Now we have the full picture of what was built and how it was measured. The remaining question is what the results actually tell us, and that's where the paper delivers its central insight about thinking versus size.

Conclusion: Tom: Alright, we've walked through the benchmark itself, the datasets, and the metrics. Now let's pull together what they actually found, because that's the part that will stick with people.

Jane: The central result is captured in that chart showing mean improvement gain by cognitive level. For knowledge-level tasks, the gap between the largest model and the 8-billion Qwen with thinking is about twenty-four percent. For reasoning and application tasks, that gap narrows dramatically to thirteen and eighteen percent respectively.

Lu: And in specific reasoning-heavy tasks, the small model with thinking actually wins outright. On PPNL_multi, the 8-billion Qwen with thinking got 0 point 62 accuracy while the 120-billion GPT-OSS only managed 0 point 57. That's a direct counterexample to the assumption that bigger is always better.

Meng: The pattern across all seventeen subdatasets is consistent. The thinking-enabled Qwen models almost always beat their non-thinking versions, sometimes matching the performance of the next model size up. That suggests that giving a model time to reason can be a substitute for raw parameter count — at least for geo-reasoning tasks.

Tom: But the knowledge tasks tell a different story. On GeoQuestions1089_coord, GPT-OSS-120B outperforms the best Qwen model by 0 point 29 in coordinates accuracy. That's a massive margin, and it shows that factual geographic knowledge — like where a city is or what its coordinates are — is still largely a function of model size.

Jane: Even GPT-OSS models were only given limited thinking budgets in this evaluation. So the authors are careful not to overclaim — they suggest that using a larger model with full thinking capabilities would likely produce marked improvements across all subdatasets. But that's left as future work.

Lalam: The broader implication is that the eye community needs to distinguish between knowledge and reasoning when evaluating models. This benchmark provides the tools to do exactly that, and the finding that thinking can substitute for scale on reasoning tasks has practical implications — smaller models are cheaper to run and deploy.

Lu: And they've made everything accessible. The benchmark is on GitHub, the metrics are on Hugging Face, and there's a notebook for reproducing the results. That lowers the barrier for other researchers to test their own models and extend the benchmark to new tasks.

Meng: There are limitations, of course. The benchmark is text-based, so it doesn't test visual-geographic understanding. And the translation of the Chinese exam questions using Google Translate could introduce some noise. But those are natural starting points for future work.

Tom: The takeaway for me is that this paper gives us a clearer map of where LLMs stand on geographic abilities. They're far from perfect, but with the right prompting strategy — thinking mode — even modest models can handle complex spatial reasoning that we might have thought required massive scale.

Jane: And that's genuinely useful information for anyone building geo-eye applications, whether it's navigation, urban planning, or location-based recommendation. So we'll be watching how the community uses this benchmark and what results come out of it.

Lu: Thanks to the authors for putting this together and making it public. That's the kind of contribution that moves the field forward.

Tom: We're wrapping up this paper and getting ready for the next one. Thanks for listening, and we'll be back soon with more research to talk through.