Geospatial Knowledge of LLMs
Probing what large language models know about places, distances and maps
Before building agents that move through cities, we asked a basic question: what do language models actually know about geography? Using only prompts, with no training, we probed open models (OPT, LLaMA and the instruction-tuned Alpaca) for three abilities:
- Knowledge: can a model tell where a city is?
- Awareness: when a model writes “near”, does it name a place that is actually nearby?
- Reasoning: can the distances a model believes in be turned back into a map?
1 · Knowledge: where is this city?
We asked each model for the coordinates of 3,527 cities with more than 100,000 people, using two wordings, with no examples (zero-shot) and with three solved examples (3-shot):
Template 1: The geo-coordinates of <city> are … Template 2: The latitude and longitude of <city> are …
The geo-coordinates of Peoria are 40.69 and -89.58.
The geo-coordinates of Oldham are 53.55 and -2.11.
The geo-coordinates of Plzen are 49.75 and 13.36.
The geo-coordinates of Kathmandu are … | Model | Template | Zero-shot error (km) | Answered % | 3-shot error (km) | Answered % |
|---|---|---|---|---|---|
| Earlier baselines (Liétard et al.) | |||||
| Word2Vec · BERT-L · GPT-2 | – | 2612 · 3077 · 4498 | – | – | – |
| Open LLMs, prompted | |||||
| OPT-6.7B | 1 | 3085 | 21 | 7714 | 98 |
| OPT-6.7B | 2 | 3009 | 43 | 7304 | 99 |
| OPT-13B | 1 | 2452 | 18 | 6830 | 98 |
| OPT-13B | 2 | 2780 | 22 | 6052 | 98 |
| LLaMA-7B | 1 | 1082 | 84 | 1634 | 99 |
| LLaMA-7B | 2 | 521 | 10 | 1469 | 99 |
| LLaMA-13B | 1 | 864 | 89 | 1069 | 99 |
| LLaMA-13B | 2 | 386 | 31 | 1634 | 99 |
| Alpaca-7B (instruction) | 1 · 2 | 1799 · 2158 | 76 · 99 | – | – |
Mean distance between predicted and true coordinates (Table 1 of the full paper). "Answered" is the share of cities for which the model produced coordinates at all; for example, a model may reply "Lobito is located in Angola" instead.
Three lessons stand out. Bigger and better-trained models know more: LLaMA is far more accurate than OPT at the same size. Wording matters: the two templates give quite different results. And examples change behavior: without them, models answer less often but more accurately; with three examples they almost always answer, sometimes with a guess.
2 · Awareness: does “near” mean near?
For 93 cities across the contiguous U.S., LLaMA-13B completed sentences such as these, 500 times per city and phrase:
Albany, New York is near … Albany, New York is close to … Albany, New York is far from … Albany, New York and …
If the model understands these phrases, the places it names after “near” and “close to” should be close by, those after “far from” far away, and those after “and” anywhere. That is exactly what happens:
Where the places LLaMA-13B names for Albany, New York, end up (Fig. 3 of the full paper). Map tiles by Stamen Design (CC BY 3.0); map data © OpenStreetMap contributors (ODbL).
The state name matters too. Without it, the model confuses Albany, New York, with Albany, California, and a second hotspot appears on the West Coast:
Is this real awareness or just words that often appear together? We used the CC100 web corpus as a stand-in for the training data and correlated how often each city is generated with distance, co-occurrence and popularity:
| Spearman correlation of generation counts with… | near | close to | far from | and |
|---|---|---|---|---|
| Distance to the prompt city | −0.33 | −0.32 | 0.03 | −0.12 |
| Co-occurrence of the two cities in text | 0.24 | 0.24 | 0.09 | 0.26 |
| How common the generated city is | 0.13 | 0.14 | 0.13 | 0.30 |
Selected rows of Table 2 in the full paper.
“Near” and “close to” track distance far more strongly than the control word, while “and” leans on co-occurrence and popularity. The model’s spatial words reflect geography, not just which names show up together.
3 · Reasoning: redraw the map
Finally, we asked whether the model’s sense of distance is coherent enough to locate cities. We prompted for the distance between every pair of cities, placed the cities with multidimensional scaling (MDS), aligned the result to cities with known positions, and measured how far each held-out city landed from its true location.
The distance in kilometers between Albany, New York and Dallas, Texas is …
| Distances used to place cities | Whole contiguous U.S. (km) | Within census division (km) |
|---|---|---|
| True distances (best possible) | 190.41 | 56.78 |
| Random distances | 1440.01 | 483.15 |
| Co-occurrence counts | 1237.01 | 425.64 |
| “and” generation counts | 1359.51 | 453.73 |
| “near” generation counts | 750.66 | 328.91 |
| “close to” generation counts | 782.22 | 324.06 |
| “far from” generation counts | 1383.23 | 455.76 |
| Distances predicted by the LLM | 346.65 | 177.41 |
Mean error in placing each city (Table 3 of the full paper).
Asking the model for distances directly works best, and its distances correlate with the true ones at r = 0.92. The placements are not precise enough to replace coordinates, but they are often close. For Albany and Havre both placements land near the true location; for Dallas the placement from the model’s distances is off, while for Indianapolis it is the better of the two:
Green: true location; blue: placed from true distances; red: placed from distances predicted by the LLM (Fig. 4 of the full paper). Map tiles by Stamen Design (CC BY 3.0); map data © OpenStreetMap contributors (ODbL).
What came next
Language models carry a surprising amount of geography and use spatial words sensibly, though not yet with map-level precision. That made them worth trying where human behavior and places meet: generating travel diaries, and later running whole city simulations.
