Click2Poly: A VLM for vector mapping buildings and walls

Click2Poly is a human-in-the-loop AI assistant designed to speed up the manual correction and editing of building and wall vector layers in geospatial mapping. It extends the Florence-2 Vision Language Model (VLM) and is implemented as a QGIS plugin. The system responds to user clicks by directly editing the building or wall vector layer, with no post-processing.

The paper states: "We present Click2Poly, a human-in-the-loop AI assistant designed to speed up this manual step. Extending the Florence-2 [1] Vision Language Model (VLM), Click2Poly responds to user clicks by editing the building or wall vector layer directly. Implemented as a QGIS [2] plugin, Click2Poly speeds up the manual editing of building and wall vector layers in a real-world production environment."

The method trains Florence-2 on three new tasks (tools): , , and . The prompt consists of the task plus the user click location. For training, the dataset includes 377,207 building polygons across 202 geographic areas (159 training, 43 validation) covering 1,344 km2, and 197,234 wall linestrings across 117 geographic areas (94 training, 23 validation) covering 201 km2, all using 30 cm/px satellite imagery. Images are patched to 768×768 px to match Florence-2's input size. Training uses LoRA with rank 256, reducing trainable parameters to 70M, with one adapter per task. Training times were 7 days for on 8 A100 GPUs, 27 days for on 8 A100 GPUs, and 14 days for on a single L40S GPU.

The system is implemented in a server-client fashion, working on a grid of overlapping 768×768 px cells. Image encodings are cached per cell for faster subsequent predictions. Performance: an image encoding request round-trip takes about 1.3 s and a polygon prediction request round-trip takes about 0.8 s (the server running on a RTX 3060 GPU).

The experimental setup measured speedup in manual correction of automatic building extractions across 4 test areas (Luanda, São Paulo, Rosario, Kampala) using a Latin square design with 4 operators. Results show a total speed-up of ×1.53 when using Click2Poly compared to standard QGIS tools. Area-specific speedups were: ×1.78 (Area A), ×0.93 (Area B, slower), ×2.30 (Area C), ×1.38 (Area D). The paper notes: "Area B in a city center with multi-level complex buildings on an off-nadir image was slower when using Click2Poly because some of its results had to be manually fixed by standard tools. Area C had the best speed-up, showing that the model works very well in a dense built-up area as long as the buildings are not multi-level and the image is closer to nadir."

Operator agreement (IoU) between operators ranged from 89.1% (Area B) to 96.5% (Area C), with the paper stating: mapping is inherently ambiguous as 4 different operators produce 4 slightly different maps, making it difficult to define a true ground truth for deep learning training.

Comparisons with GeoAI plugin (running SAM 3) and SAMPolyBuild show Click2Poly follows contours more accurately with a single click, while other methods could not do the same even with multiple clicks. For walls, the segmentation-based methods cannot output meaningful results, while Click2Poly predicts accurate walls directly. The model also generalized well to higher-resolution aerial imagery (7.5 cm/px) despite being trained on 30 cm/px, capturing non-rectangular buildings without over-regularization.

Limitations include: "Florence-2's input size being 768 × 768 px, only the visible part of large buildings extending beyond this size is extracted. The deep learning model also does not snap predicted geometry and thus can result in overlapping geometries."

The contributions listed are: (i) a model predicting polygon buildings on satellite imagery, (ii) a system able to split building blocks into individual buildings, (iii) a model predicting linear features for walls on satellite imagery, and (iv) human-in-the-loop for better control. Future work aims to solve snapping of predicted geometries and the issue of large buildings, and to extend toward a fully-automated mode.

Improvements to AI Systems:

1. Add Click-Based Interactive Correction to Segmentation Models

• Integrate a human-in-the-loop mechanism where a single user click on an image directly edits the output polygon or line (e.g., draw, split, or refine) without requiring post-processing.

• The improved system can correct mis-segmented buildings or walls in real time, reducing manual redrawing effort by 1.5× in production mapping workflows.

2. Multi-Task Tool-Specific Adapters on a Shared Vision-Language Backbone

• Use a single VLM (like Florence-2) with separate low-rank adapters (LoRA) for different geometric tasks (e.g., draw building, split building, draw wall).

• The improved system can switch between tasks via a text prompt (e.g., `` + click coordinates) without retraining the entire model, enabling flexible, on-demand geometry generation for diverse mapping needs.

3. Cached Image Encoding for Repeated Local Edits

• Precompute and cache image embeddings per grid cell (768×768 px) to avoid redundant forward passes when a user clicks multiple times in the same area.

• The improved system reduces per-click latency from 2.1 s to 0.8 s for polygon prediction, enabling near-interactive editing speeds on modest GPUs (e.g., RTX 3060).

4. Grid-Based Overlap Handling for Large Objects

• Process images in overlapping tiles and merge predictions across tiles, using the click location to select the correct tile’s output.

• The improved system can handle buildings larger than the model’s input size by predicting partial geometries per tile and stitching them seamlessly, avoiding truncation errors.

5. Ambiguity-Aware Evaluation and Training

• Train and evaluate using multiple operator annotations (e.g., 4 different maps per area) and report IoU ranges (e.g., 89–96%) instead of a single ground truth.

• The improved system can learn to produce multiple plausible geometry hypotheses, and the human-in-the-loop click can disambiguate which one to finalize, reducing errors in ambiguous urban scenes.

6. Adaptive Regularization Control

• Add a confidence or complexity flag to the model output, allowing it to switch between over-regularized (e.g., rectangular) and fine-grained (non-rectangular) geometry based on image resolution or building complexity.

• The improved system generalizes from 30 cm/px to 7.5 cm/px aerial imagery without forcing right angles, preserving irregular building footprints.

7. Snapping and Topology Correction Module

• Post-process predicted geometries with a lightweight snapping algorithm that aligns shared edges and prevents overlaps, triggered only when the user confirms a click.

• The improved system outputs clean, topologically consistent vector layers directly, eliminating manual cleanup of overlapping polygons or misaligned walls.

8. Fully-Automated Mode with Click Fallback

• Extend the interactive system to run in batch mode (no clicks) for initial predictions, then allow users to click on any mispredicted area to trigger a targeted correction.

• The improved system can pre-label entire city blocks automatically, then let operators fix only the hardest cases (e.g., multi-level buildings), combining speed with human oversight.