GeoBenchLLM: A Comprehensive Benchmark for Evaluating LLMs on Geo-Related Tasks

arXiv:2608.07411 · cs.AI, cs.CL, cs.IR, cs.LG · Submitted 2026-08-07 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: Paper Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Next we'll be talking about the paper "GeoBenchLLM: A Comprehensive Benchmark for Evaluating LLMs on Geo-Related Tasks".

Jane: The paper was written by Rodrigo Ferreira Rodrigues, Karim Radouane, Jose G Moreno and Lynda Tamine from University of Toulouse.

Tom: Stay tuned as we take you through the paper and discuss its implications.

Paper summary: Tom: So we've got a new benchmark paper in the geo-eye space, and I have to say, this one feels like it fills a real gap. The team from Toulouse pulled together a benchmark called GeoBenchLLM that spans twelve datasets and over four hundred thousand questions, all about geographic tasks.

Jane: Four hundred twenty-one thousand questions, and they cover everything from simple factoid questions like coordinates to complex pathfinding. That's a serious scale jump compared to earlier benchmarks, where you'd see maybe a few thousand examples at best.

Lu: What I find striking is the three-level cognitive structure they borrowed from earlier work — Knowledge, Reasoning, Application. It's not just about whether a model knows Paris is north of Toulouse, it's about whether it can reason through spatial relationships and then apply that to real tasks like recommending a point of interest.

Meng: And the key finding? They compared Qwen models from 0 point 6 billion parameters up to 8 billion, with and without a "thinking" mode, and then compared against GPT-OSS models at 20 and 120 billion. The result shows that reasoning matters enormously — an 8-billion-parameter model with thinking enabled actually beats a 120-billion-parameter model on several reasoning-heavy tasks.

Tom: Right, on the PPNL multi-pathfinding subdataset, the 8-billion Qwen with thinking hit 0 point 62 accuracy while the 120-billion model only reached 0 point 57. That's a remarkable result, because the usual story is that bigger models just win. Here, for tasks that really require spatial-temporal reasoning, thinking mode can close — and sometimes overturn — the size gap.

Jane: But it's not uniform. On knowledge-level tasks like coordinates prediction, the larger GPT-OSS models still dominate by a wide margin. That tells us that remembering geographic facts is largely about how much world knowledge is baked into the parameters, while reasoning is a skill that smaller models can activate if they're allowed to think step by step.

Lalam: The broader implication here is that benchmarks need to separate these cognitive levels carefully. If you lump factual recall and spatial reasoning together, you get a muddy picture of what models can actually do. This paper's taxonomy and its new evaluation metrics for open-ended generation really push the field toward a more granular understanding.

Lu: And they made the whole thing public, with metrics available through Hugging Face and the code on GitHub. That's important — a benchmark is only useful if the community can build on it, reproduce the numbers, and extend it.

Meng: The paper also introduces custom metrics for things like coordinates accuracy with a tolerance radius, precision and recall for place lists, and compliance ratios for pathfinding tasks. Those are genuinely needed because existing evaluation tools were designed for closed-form answers, while many geo-tasks require open-ended generation.

Tom: Exactly — and that's where a lot of earlier benchmarks fell short. They restricted themselves to yes/no or multiple-choice questions because those are easy to grade. This work deliberately embraces the messy, generative setting and builds the tools to grade it fairly.

Jane: So we're looking at a benchmark that not only scales up the data, but scales up the difficulty of evaluation itself. The next question is how they actually assembled these twelve datasets and what transformations were needed — that's where the real work shows.

Page 1 of the paper: Lu: So we've set the stage with the big numbers and the main finding. Let's pull back the first page and look at how they actually positioned the work against what came before, because the motivation is carefully argued.

Tom: Right, they start with the claim that geo-tasks are genuinely hard for question-answering systems. It's not just about knowing facts, but about geometric uncertainty and the vagueness of everyday language about locations. If I ask "is Paris north of Toulouse," that's easy for a human, but the model has to handle the spatial relationship between two real entities.

Jane: And the existing benchmarks each have their own weakness. GeoBenchmark only covers knowledge and reasoning, with no application level. CityEval is limited to an urban context and mostly multiple-choice. STBench is large but too focused on spatial reasoning tasks and misses coordinates prediction and regression. And the Xu benchmark is diverse but only has nine hundred questions.

Lu: That's the key gap — either too narrow in scope, too small in scale, or locked into formats that don't reflect how people actually ask questions. The paper's comparison table makes that crystal clear, showing which benchmarks cover which cognitive levels, formats, and scales.

Meng: I also notice they specifically point out that several benchmarks only use yes/no or multiple-choice formats. That's a serious limitation, because it gives the model a tiny set of options and doesn't test its ability to generate a free-form answer. Real users don't get four choices when they ask where something is or how far apart two cities are.

Tom: And that's why they emphasize their benchmark includes all four formats — generative, regression, yes/no, and multiple-choice. That breadth forces the evaluation to be more honest about what models can and cannot do.

Jane: The scale difference is worth dwelling on. Their benchmark has 421,041 examples, while the largest existing benchmark, STBench, has around 80,000. That's a five-fold increase, and it matters because evaluation on a bigger, more diverse set reduces the variance of the results and gives you more confidence that a model's performance is real.

Lalam: There's also a careful choice in how they define the three cognitive levels. Knowledge tasks are factoid questions answerable by querying a geographic database — coordinates, real numbers, yes/no, place names. Reasoning tasks require applying those concepts, like spatial reasoning with distance and topology, or complex scenario QA. Application tasks go further, requiring recommendation or pathfinding in real-world settings.

Lu: That taxonomy gives the benchmark an internal structure that allows for a more nuanced analysis. You can ask whether thinking mode helps equally across levels, or whether model size matters more at certain levels. And indeed, their results show exactly that — thinking closes the gap most dramatically on reasoning and application tasks, not on knowledge recall.

Meng: The authors also mention that they adopted this classification from the Xu et al. benchmark, but they expanded it, added more tasks, and scaled it up. So it's a continuity with prior work rather than a brand-new framework, which makes comparison across benchmarks more meaningful.

Tom: And they previewed the key finding on that very first page — that models up to 120 billion parameters can succeed, but smaller models can close the gap when thinking is enabled. It's a bold claim to put right up front, and the rest of the paper is them backing it up.

Jane: Now, the next page takes us into the related work and the detailed motivation, where they go through the individual datasets and benchmarks one by one. There's a lot of texture there about why each existing resource falls short.

Page 2 of the paper: Lu: We've covered the big-picture motivation. Now the paper walks through the related work in detail, and that's where it gets interesting because you can see the history of how geo-evaluation evolved.

Meng: They start with GeoBenchmark, which focuses on direction, distance, and topology using data from YAGO2geo and Ordnance Survey geometries. That's a solid knowledge and reasoning test, but it has no application tasks at all — no recommendation, no pathfinding.

Tom: Then there's STBench, which they credit with around 80,000 author-generated questions derived from the Yelp dataset. The focus there is on temporal characteristics of geographic questions, but it lacks variety and misses coordinates prediction and complex scenario QA.

Jane: The Xu benchmark is more diverse in task coverage but has only 900 questions. Even though those are high-quality questions drawn from Wikipedia and geography textbooks, that's just too few to provide a comprehensive assessment. You can't make strong claims about a model's geographic capability based on nine hundred examples.

Lu: And CityEval, part of the CityGPT framework, has a large set of questions covering most tasks except regression and complex scenario QA, but everything is stuck in an urban context. That limits its generalizability to world scale.

Meng: The paper also discusses the individual datasets that they eventually incorporated. There's bAbI, which is a classic from Facebook designed to test language models on various QA tasks — but only two of its twenty tasks are geographic, namely positional reasoning and pathfinding.

Tom: And then there's MapQA, which is interesting because it's focused on places of interest retrieved from OpenStreetMap for Southern California and Illinois. It asks questions like predicting a place name given its amenity type and spatial relationship to another place, plus regression questions about distances between POIs.

Jane: What I appreciate here is that they're not just criticizing — they're carefully identifying what each resource can contribute. They're building their benchmark from twelve existing datasets, some of which were never originally designed for LLM evaluation, and they're transforming them to make them suitable.

Lu: For example, GeoQuestions1089 was originally designed for natural language to SPARQL translation. They processed and cleaned the raw query responses to extract useful data for their coordinate, yes/no, regression, and place prediction subdatasets.

Meng: And there's a transparency note about TourismQA — the original code couldn't be used to regenerate the dataset, so they retrieved it from a subsequent work that used it. That kind of honest reporting about provenance is important for reproducibility.

Tom: They also applied transformations from prior works where necessary and partitioned some datasets into subdatasets to prevent overlap across tasks. That's a crucial design choice, because if a dataset contains both yes/no questions and place prediction questions, you don't want to contaminate your task definitions.

Jane: So on this page, we're seeing the nuts and bolts of what it takes to assemble a comprehensive benchmark. It's not just collecting files — it's about understanding each dataset's original purpose, adapting it to a new evaluation setting, and being transparent about what was changed and why.

Lu: And the outcome is seventeen textual subdatasets covering eight tasks across three cognitive levels. That granularity is what allows the paper's analysis to show which cognitive skills respond to model size versus thinking mode.

Meng: Now, the next page gets into the actual datasets beyond knowledge tasks — the reasoning and application ones — and that's where things get innovative, especially with synthetic spatial reasoning datasets like StepGame.

Page 3 of the paper: Lu: So the previous page set up the datasets for knowledge tasks — coordinates, yes/no, regression, and place prediction. Now the paper introduces the reasoning and application datasets, and that's where the benchmark gets really demanding.

Tom: For reasoning, we have GeoSQA and GKMC, both drawn from the Chinese Gaokao geography exams. These are scenario-based multiple-choice questions, and they were translated into English using Google Translate. You have to appreciate the scale of that — GeoSQA has over 4,000 questions, and GKMC has tens of thousands.

Meng: But the more novel additions are SpatialEvalLLM, SpartUN, and StepGame. SpatialEvalLLM places the model in a grid of objects and asks it to identify the object at the end of a described path. SpartUN builds scenarios with objects connected topologically or directionally and asks either boolean or relational questions.

Jane: StepGame is particularly clever — it's synthetic, and the model has to infer the directional relationship between two points from intermediate placements. The questions are categorized by how many reasoning hops are required, so you can literally see where models start to fail as the chain of reasoning gets longer.

Lu: Now the application level is where the paper pushes into real-world scenarios. TourismQA is a POI recommendation dataset built from tourist reviews across fifty cities worldwide. Given a tourist question and available reviews, the model has to predict relevant points of interest.

Meng: And NY-POI is derived from Foursquare check-ins, with the task being to predict the next POI in a user's trajectory based on their habits and visit history. That's a spatial-temporal prediction problem that requires understanding user behavior alongside geography.

Tom: The pathfinding datasets are the real test of applied reasoning though. GridRoute asks the model to return a valid sequence of adjacent grid coordinates from point A to point B, avoiding obstacles and without diagonal movements. PPNL extends that to a multi-objective setting where you must pass through an unordered list of intermediate points.

Jane: That's where their evaluation metrics become essential, because grading a path isn't a simple right-or-wrong judgment. They distinguish between a path being feasible — staying within grid boundaries and avoiding obstacles — versus successful, meaning it reaches the goal, versus optimal, meaning it does so in the minimum number of moves.

Lu: The optimal ratio is their main metric because it's the most discriminating. A model can produce lots of feasible paths that wander around, but only a truly competent reasoner will find the shortest route.

Meng: And they also handle the edge case of an unreachable goal. The unreachable accuracy metric measures how often the model correctly detects that a goal is impossible to reach, which tests whether the model truly understands the grid constraints rather than just pattern-matching on training data.

Tom: There's also a compliance ratio for whether the model even outputs the right format — a list of grid coordinates. That's a subtle but important point. If a model gives you a perfectly reasoned path in prose but not in the requested format, it should be marked down, because the task explicitly asks for a structured output.

Jane: And for the place prediction tasks, they adapted precision and recall to handle lists of geographic coordinates. It's not just about whether one predicted point matches one reference — you need to account for multiple reference points and multiple predictions, and the distance between them matters.

Lu: They introduce a metric called the compliance ratio, and separately they use Bleu-1 and BERT-Score for the text-based recommendation tasks like TourismQA. Bleu-1 is their main metric because it's more discriminating than BERT-Score, which tends to give high scores even when the model is using different words.

Meng: The beauty of these metrics is that they can be reused. They put them in a Hugging Face collection so other researchers can evaluate their own models with the same tools, which is a real contribution to the community beyond just the benchmark data.

Tom: So the third page gives us the full landscape of tasks and the evaluation machinery. The tables there spell out the exact numbers — trains, devs, tests, average word counts — for all seventeen subdatasets. It's a lot of detail, but it makes the benchmark concrete and reproducible.

Jane: Now we have the full picture of what was built and how it was measured. The remaining question is what the results actually tell us, and that's where the paper delivers its central insight about thinking versus size.

Conclusion: Tom: Alright, we've walked through the benchmark itself, the datasets, and the metrics. Now let's pull together what they actually found, because that's the part that will stick with people.

Jane: The central result is captured in that chart showing mean improvement gain by cognitive level. For knowledge-level tasks, the gap between the largest model and the 8-billion Qwen with thinking is about twenty-four percent. For reasoning and application tasks, that gap narrows dramatically to thirteen and eighteen percent respectively.

Lu: And in specific reasoning-heavy tasks, the small model with thinking actually wins outright. On PPNL multi, the 8-billion Qwen with thinking got 0 point 62 accuracy while the 120-billion GPT-OSS only managed 0 point 57. That's a direct counterexample to the assumption that bigger is always better.

Meng: The pattern across all seventeen subdatasets is consistent. The thinking-enabled Qwen models almost always beat their non-thinking versions, sometimes matching the performance of the next model size up. That suggests that giving a model time to reason can be a substitute for raw parameter count — at least for geo-reasoning tasks.

Tom: But the knowledge tasks tell a different story. On GeoQuestions1089 coord, GPT-OSS-120B outperforms the best Qwen model by 0 point 29 in coordinates accuracy. That's a massive margin, and it shows that factual geographic knowledge — like where a city is or what its coordinates are — is still largely a function of model size.

Jane: Even GPT-OSS models were only given limited thinking budgets in this evaluation. So the authors are careful not to overclaim — they suggest that using a larger model with full thinking capabilities would likely produce marked improvements across all subdatasets. But that's left as future work.

Lalam: The broader implication is that the eye community needs to distinguish between knowledge and reasoning when evaluating models. This benchmark provides the tools to do exactly that, and the finding that thinking can substitute for scale on reasoning tasks has practical implications — smaller models are cheaper to run and deploy.

Lu: And they've made everything accessible. The benchmark is on GitHub, the metrics are on Hugging Face, and there's a notebook for reproducing the results. That lowers the barrier for other researchers to test their own models and extend the benchmark to new tasks.

Meng: There are limitations, of course. The benchmark is text-based, so it doesn't test visual-geographic understanding. And the translation of the Chinese exam questions using Google Translate could introduce some noise. But those are natural starting points for future work.

Tom: The takeaway for me is that this paper gives us a clearer map of where LLMs stand on geographic abilities. They're far from perfect, but with the right prompting strategy — thinking mode — even modest models can handle complex spatial reasoning that we might have thought required massive scale.

Jane: And that's genuinely useful information for anyone building geo-eye applications, whether it's navigation, urban planning, or location-based recommendation. So we'll be watching how the community uses this benchmark and what results come out of it.

Lu: Thanks to the authors for putting this together and making it public. That's the kind of contribution that moves the field forward.

Tom: We're wrapping up this paper and getting ready for the next one. Thanks for listening, and we'll be back soon with more research to talk through.

Rodrigo Ferreira Rodrigues, Karim Radouane, Jose G Moreno, Lynda Tamine

University of Toulouse

cs.AI, cs.CL, cs.IR, cs.LG

Submitted: 2026-08-07

Updated: 2026-08-10

Comments: Accepted at CIKM2026

Code: https://github.com/Rfr2003/GeoBenchLLM

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 66/100

The gist: The paper is authored by Rodrigo Ferreira Rodrigues, Karim Radouane, Jose G Moreno, and Lynda Tamine, all affiliated with the University of Toulouse, France.

Key concepts

GeoBenchLLM: A benchmark for evaluating large language models on geographic tasks. It includes 421,041 questions across 12 datasets, covering knowledge, reasoning, and application levels. It uses multiple formats like generative, regression, yes/no, and multiple-choice, and provides custom metrics for open-ended answers.
Cognitive levels: The benchmark categorizes tasks into three levels: Knowledge (factoid questions like coordinates), Reasoning (spatial relationships and complex QA), and Application (recommendation and pathfinding). This structure helps analyze whether model size or thinking mode affects performance differently across levels.
Thinking mode: A feature in some LLMs that allows step-by-step reasoning before answering. The episode highlights that enabling thinking mode in an 8-billion-parameter model can outperform a 120-billion-parameter model on reasoning-heavy tasks, closing the size gap.

Terminology

Benchmark — 46× in this paper · explained in 2 episodes
Accuracy — 13× in this paper · explained in 1 episode
Generalization — 4× in this paper · explained in 13 episodes
Text-to-SQL — 3× in this paper · explained in 3 episodes
Benchmarking — 2× in this paper · explained in 4 episodes
Core — 2× in this paper · explained in 2 episodes
Thinking mode — 2× in this paper · explained in 2 episodes
Hypergraph — 1× in this paper · explained in 2 episodes
IEEE Conference on Artificial Intelligence — 1× in this paper
Large Language Models (LLMs) — 1× in this paper · explained in 6 episodes
Transfer Learning — 1× in this paper · explained in 10 episodes
Transformer — 1× in this paper · explained in 2 episodes
Zero-Shot Generalization — 1× in this paper · explained in 5 episodes

Summary

The paper is authored by Rodrigo Ferreira Rodrigues, Karim Radouane, Jose G Moreno, and Lynda Tamine, all affiliated with the University of Toulouse, France. The paper is published on arXiv (arXiv:2608.07411v1 [cs.AI], 7 Aug 2026).

The paper introduces GeoBenchLLM, a comprehensive benchmark for probing LLMs on geo-related tasks. The authors state that In the context of geodata, existing Large Language Models have often been studied in a homogeneous setting, which has considerably limited insights into their generalization capabilities. They leverage a careful selection of twelve publicly available datasets from diverse geo-related tasks and domains, and evaluate a set of LLMs on geo-spatial and temporal understanding using our benchmark. Their results show that reasoning and size have a strong impact on overall performance. The benchmark is publicly available at https://github.com/Rfr2003/GeoBenchLLM.

The authors define geo-related tasks as "problems involving geographic entities or concepts, usually requiring spatial operations to be solved. These tasks range from factoid questions (e.g., Is Paris north of Toulouse?) to pathfinding problems (e.g., Show me a route from Paris to Toulouse). They attribute challenges in these tasks to geometric uncertainty as well as language variability and vagueness," citing Mai et al.

While prior work has evaluated LLMs on geo-related tasks (e.g., Manvi et al. on geographical regression, Li et al. on place prediction), the authors argue existing benchmarks are insufficient: they are either too small (Xu et al. [21]), do not cover enough major tasks (STBench [14]), or are limited to a specific context (CityEval [5]). To address this gap, they introduce GeoBenchLLM, a highly accessible and comprehensive benchmark for assessing the intrinsic knowledge of LLMs on geo-related tasks.

GeoBenchLLM brings together twelve datasets, further divided and transformed into seventeen purely textual subdatasets... covering eight different tasks. Table 1 provides a comparison with existing benchmarks (GeoBenchmark, CityEval, Xu et al., STBench, bAbI tasks 17 & 19, MapQA), showing that GeoBenchLLM uniquely covers Knowledge, Reasoning, and Application cognitive levels at world scale, with all formats (Generative, MCQ, Regression, Yes/No), totaling 421,041 examples.

The taxonomy is inspired by Xu et al. [21], organizing our benchmark into three cognitive levels — Knowledge, Reasoning, and Application. The tasks are described as follows:

Knowledge tasks: comprise factoid questions answerable by querying a geographical database, split into four subtasks by answer type: coordinates, real numbers, Yes/No, or place names.
Reasoning tasks: require the model to apply those concepts: Complex Scenario QA involves processing knowledge and facts in an MCQ format, while Spatial Reasoning requires understanding distance, topology, and direction.
Application tasks: "build further on these skills for real-world use cases: POI Recommendation combines world knowledge, spatial reasoning, and user preference processing, while Pathfinding additionally incorporates a temporal factor to construct paths between points."
GeoQuestions1089: 1089 geographical factoid questions covering four regions: the United States, the United Kingdom, Ireland, and Greece, originally designed for natural language to SPARQL translation, with raw query responses processed and cleaned.
GeoQuery: simple questions about geographical facts concerning the United States, drawn from the GeoBase knowledge base (1996), using the version made available by Finegan-Dollak et al. for text-to-SQL evaluation.
Ms-Marco: a large-scale dataset of 1,000,000 question-answer pairs collected from Bing, from which they retained a geographical subset of the QA task (version 2.1), retaining only LOCATION-category questions that mention at least one real geographical entity.
GeoSQA: 4,110 multiple-choice questions from Gaokao geography examinations, each accompanied by a scenario and a diagram annotated in text by the authors, originally in Chinese and translated to English using Google Translate.
GKMC: also a scenario-based multiple-choice dataset drawn from the Gaokao, but without diagrams, similarly translated from Chinese to English.
SpatialEvalLLM: evaluates spatial understanding by placing the LLM within a shaped grid of objects and asking it to identify the object at the end of a described path.
SpartUN: "evaluates spatial reasoning by constructing scenarios of objects connected topologically or directionally, then asking either Boolean (truthfulness of a relationship) or Relational (choice of best-fitting relationship) questions."
StepGame: a synthetic dataset where the model must infer the directional relationship between two points from a scenario describing intermediate placements, with questions categorised by the number of reasoning hops required, with increasing difficulty.
TourismQA: a POI recommendation dataset built from tourist site reviews across fifty cities worldwide, retrieved from the repository of Li et al. as the original code could not regenerate it.
NY-POI: derived from Foursquare check-ins and contains geolocated POIs with user–POI interaction sequences and timestamps, where the task is to predict the next POI in a trajectory given the user's habits and visit history.
GridRoute: assesses pathfinding by asking the model to return a valid sequence of adjacent grid coordinates from a starting point A to an end point B, avoiding obstacles and without diagonal movements.
PPNL: extends the pathfinding task to a multi-objective setting, where the model must pass through an unordered list of intermediate points to produce a correct solution, making it more difficult than GridRoute.

The authors introduced custom metrics for open-ended tasks since the evaluation of text generation can be complex as they are open questions. Key metrics include:

Coordinates Accuracy: a predicted coordinate p is correct if it falls within a circle of radius r centred at the gold coordinate g using haversine distance.
Precision, Recall, P-R Mean and Median: used for Place prediction and Regression where reference and predicted answers are lists of values. The median of P-R Mean across questions is our main metric, chosen to reduce the impact of outliers.
Compliance Ratio: measures how often the model's output follows the required format (list of grid coordinates) in the Pathfinding task.
Feasible, Success and Optimal Ratios: a path is feasible if it stays within grid boundaries and avoids obstacles, successful if it also reaches the goal, and optimal if it does so in the minimum number of moves. Optimal Ratio is our main metric as the most discriminating.
Unreachable Accuracy: measures how often the model correctly detects that a goal is unreachable.
Distance: the minimum number of additional moves needed to turn a feasible path into a successful one.
Accuracy: used for closed-answer tasks (Yes/No questions, Complex Scenario QA, Spatial Reasoning, and NY-POI).
Bleu-1 and Bert-Score: used for Ms-Marco place and TourismQA. Bleu-1 is our main metric as it is more discriminating than Bert-Score.

All metrics are available in a Hugging Face Collection usable with the evaluate library.

The authors evaluated models from the Qwen3 family with various sizes (Qwen3-0.6B, Qwen3-1.7B, Qwen3-8B), with and without thinking, as well as two larger models in the GPT-OSS-20B and 120B versions, noting that we only allowed low thinking budgets to GPT-OSS-20B and 120B.

The main results (Table 3) show that GPT-OSS models manage to achieve the best results in nine of the seventeen subdatasets. However, "a relatively small model, Qwen3-8B, is capable of achieving similar performance when thinking mode is activated. It even surpasses the largest model on five of the ten subdatasets belonging to the Reasoning and Application cognitive levels, achieving 0.62 in accuracy on PPNL multi while the largest model only attains 0.57, for example."

The authors conclude that for tasks where reasoning plays a central role, thinking is more important than size. Conversely, GPT-OSS models lead in four out of the seven subdatasets of the Knowledge level, often by a wide margin. On GeoQuestions1089 coord, GPT-OSS-120B outperforms the best Qwen model by 0.29 in coordinates accuracy. This indicates that without external tools and databases, the size of the model plays a major role in embedding real-world knowledge, which is required to answer questions at the Knowledge cognitive level.

Figure 1 shows that "for the Reasoning and Application cognitive levels, the gap between the largest models and Qwen3-8B in thinking mode is very narrow compared to the one at the Knowledge level. This gap goes from around 24% for the Knowledge level to 13% and 18% for the Reasoning and Application levels respectively. The authors also note the power of reasoning, as the thinking version of the same model almost always surpasses the non-thinking one, sometimes reaching the same performance as the next model in size, suggesting that using a larger model with thinking would yield a marked improvement in performance across all subdatasets."

The paper presents GeoBenchLLM, a general benchmark containing twelve datasets for assessing the abilities of large language models in geo-related tasks at different cognitive levels, containing eight tasks ranging from Coordinates Prediction to Pathfinding. The authors emphasize accessibility and introduced new metrics for evaluating open questions in a generation context. They present baselines with small Qwen3 models with and without thinking, plus larger models from 20B to 120B parameters. Their central finding is that the parameter gap could be closed using thinking, especially for Reasoning and Application tasks.

Improvements for AI systems

Use GeoBenchLLM as a standard evaluation suite for geo-related LLM capabilities, covering 17 textual subdatasets across 8 tasks (coordinates prediction, numeric regression, yes/no factoids, place-name generation, scenario-based QA, spatial reasoning, POI recommendation, and pathfinding).
Diagnose model weaknesses by cognitive level (Knowledge, Reasoning, Application) rather than by overall accuracy, so training and compute can be targeted where the model actually fails.
For geo Reasoning and Application tasks, enable an explicit “thinking” / chain-of-thought reasoning mode before scaling parameters. Evidence: Qwen3-8B with thinking outperforms larger non-thinking models on 5/10 reasoning/application subdatasets, e.g., 0.62 vs 0.57 accuracy on PPNL multi.
For geo Knowledge tasks, do not rely on reasoning alone—scale model size or augment with external geo-database retrieval/tool use, because factual knowledge embedded in parameters strongly depends on size (e.g., GPT-OSS-120B beats the best Qwen model by 0.29 on GeoQuestions1089 coord).
Fine-tune small models on GeoBenchLLM’s diverse datasets (GeoQuestions1089, GeoQuery, Ms-Marco geographic subset, GeoSQA, GKMC, SpatialEvalLLM, SpartUN, StepGame, TourismQA, NY-POI, GridRoute, PPNL) to improve geo-spatial and temporal understanding in a multi-task setting.
Train models to output structured geo answers (coordinates, lists, grid paths) and to self-check format compliance, coordinate-radius tolerance, PR-median, feasibility, success, optimality, and unreachable-goal detection.
Teach models to explicitly recognize impossible pathfinding scenarios (Unreachable Accuracy) and to repair near-valid paths (Distance metric), improving reliability in open-ended route generation.
Use GridRoute and PPNL to train models in feasible step-by-step route generation, obstacle avoidance, multi-objective waypoint traversal, and temporal sequencing.
Answer geo factoid questions with higher accuracy across coordinates, real-number values, yes/no questions, and place-name generation.
Perform complex scenario-based geo reasoning from exam-style multiple-choice questions, including spatial relationships and diagram-described text.
Solve spatial reasoning tasks involving distance, topology, direction, and multi-hop directional inference.
Recommend points of interest based on user preferences, visit history, timestamps, and geolocated trajectories.
Generate valid, feasible, and near-optimal grid paths from start to goal, avoiding obstacles, without diagonal moves, and optionally passing through intermediate points.
Correctly detect when a requested route is unreachable instead of hallucinating an invalid path.
Match or exceed the geo task performance of much larger models when small models are used with thinking mode, while scaling to larger models with thinking yields further gains across all geo cognitive levels.