Seeing the City or Recognizing the Place? What Street-View Imagery Adds Beyond Existing Urban Data in VLM-Based Urban Sensing

Street-view imagery adds most information when an urban attribute is visible and existing records are sparse, providing guidance for image collection and map provenance.

The gist

Street-view imagery adds the most information when an urban attribute is visible and existing records are sparse.

Study Design and Comparison Framework

The study compares image-based predictions with existing urban data across seven attributes from five public resources (Road damage, Curb ramp, Floor count, Building type, Building function, Population, House price) using three Vision-Language Models (VLMs). The evaluation methodology includes comparing the target image against alternatives based on prevalence, nearby labels, or a public record. Specific comparisons include:

1. Within-model image gain: Comparing a VLM with and without the target image in addition to its task context.

2. Standalone image advantage: Comparing an image-only prediction with the strongest available non-image alternative tested for that task.

3. Image replacement effect: Changing the target photograph or adding conflicting records to observe which source shapes the model’s answer.

Urban Attributes and Source Performance

The analysis reveals that the cross-task comparison shows that image value varies more by attribute than by model. The results show distinct patterns for different attributes:

-Road damage, Curb ramp, and House price:

Existing urban data matched or exceeded image-only models for road damage, curb ramps, and house price, while neighbouring official statistics nearly matched the best image result for population. Nearby road observations predicted road damage better than image-only models.

-Building attributes:

Images were more informative for building type, building function, and low-rise floor count. All three image-only models exceeded the non-image alternative for building type. Building function also benefited from imagery, although the difference was smaller and depended on the model.

The Role of Visual Legibility and Data Density

The value of an image is determined by two properties: visual legibility and non-image predictability. Visual legibility describes how much of the reference value is physically represented in a facade-level photograph at the resolution supplied to the model, while non-image predictability assesses how well existing urban data can predict the value without the target photograph. The study found that Floor count advantage increased by 5.7 percentage points per doubling of distance to the nearest labelled building and declined for tall buildings whose rooflines often fell outside the frame. This indicates that an image can be valuable in one part of a city and largely redundant in another, as an image can therefore be valuable in one part of a city and largely redundant in another.

Influence of Conflicting Records and Annotations

When sources disagree, conflicts help distinguish model behavior. The analysis showed that supplied records often shape the prediction, with predictions moving towards the supplied value across infrastructure, building, and socioeconomic tasks. Specifically for floor count, Models followed the supplied record most often for tall buildings, while records had greater influence when the image carried less usable evidence. Furthermore, released annotations informed by reference data showed an opposite error pattern compared to image-only reruns for floor count in tall buildings. This suggests that machine annotations can reflect a combined image-and-record process.

Implications for Urban Data Collection

The findings guide future practices by suggesting that cities should compare candidate imagery with existing sources before large-scale acquisition. The appropriate source depends on the attribute: Images are well suited to visible change and condition, while stable attributes may already be represented in administrative data. The study concludes that Street-view collection is therefore most informative in places where visible attributes are poorly represented in the available map, and emphasizes that Map provenance should record the information used for each prediction. This allows users to select labels based on whether they were image-derived or annotation informed by reference data.

Limitations and Future Directions

The study notes limitations, including temporal mismatches between image capture dates and record retrieval times, which can affect mutable assets. The results are specific to the evaluated public resources, models (GPT-4o-mini, Gemini 2.5 Flash Lite, Qwen3.5 Flash), and task definitions. Future work should consider specialized models or different viewpoints to change the balance between sources for specific urban sensing tasks. The core takeaway is that Evaluating images within their local data environment provides a clearer basis for collection decisions and for describing the provenance of urban maps.

References

Biljecki, F. and Ito, K. (2021). Street view imagery in urban analytics and GIS: A review. Landscape and Urban Planning, 215:104217.

Chen, L., Li, J., Dong, X., Zhang, P., Zang, Y., Chen, Z., Duan, H., Wang, J., Qiao, Y., Lin, D., and Zhao, F.

Based on the scientific paper, here are specific ways to improve AI systems (specifically Vision-Language Models or VLMs) in urban sensing, and what those improved systems can achieve:

---

1. Improving Urban Attribute Inference by Contextual Source Comparison:

The improved AI system should not rely solely on image input for prediction. It must be designed to perform a source-based evaluation by simultaneously processing the target street-view image alongside multiple existing urban data sources (e.g., nearby observations, public registers, official statistics).

2. Improving Predictive Accuracy Across Diverse Urban Attributes:

The system can achieve superior accuracy in inferring specific attributes based on the local information environment:

• For attributes where existing data is strong (like road damage and curb ramps), the AI will maintain high accuracy even when only using an image, but its performance will be benchmarked against the superior non-image alternatives.

• For visible architectural features (building type and building function), the AI can leverage imagery to achieve higher predictive accuracy than relying on mere prevalence or labels alone.

3. Improving Floor Count Estimation in Data-Sparse Areas:

The system can provide a significant advantage in estimating building floor counts, particularly for low-rise structures, when neighboring labeled data is sparse. The improvement is quantified by the image advantage, which increases significantly as the distance to the nearest labeled building grows (e.g., +5.7 percentage points per doubling of distance).

4. Improving Model Robustness Against Data Conflicts:

The system can be trained to recognize and weigh conflicting information sources intelligently. Specifically, when an image contradicts a record-style statement (e.g., a street sign or register entry), the improved system should be able to determine whether to trust the visual scene or the accompanying text/record, leading to more accurate final predictions.

5. Improving Annotation Quality in Multimodal Datasets:

For automated annotation workflows (like OpenFACADES), the system can be enhanced by incorporating reference data directly into its training or inference pipeline. This allows it to generate machine labels that are informed by both visual evidence and structured record values (image-and-record annotations), leading to higher accuracy, especially for tall buildings where visual roofline cues are often lost.

6. Improving Spatial Generalization in Urban Mapping:

The system can be designed with an awareness of its local data environment. By understanding the adjusted reference-density gradient, the AI can better predict attribute values in areas where existing records are weak, guiding map collection efforts to target the most informative photographic gaps.