A natural language processing-based approach: mapping human perception by understanding deep semantic features in street view images

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Hi, I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "A natural language processing-based approach".

Jane: Here's the gist of the paper. The gist This study proposes a new framework based on a pre-train natural language model to understand the relationship between human perception and the sense of a scene by using deep semantic features in street view images, showing that this approach performs better than previous methods with shallow features Previous research has demonstrated the importance of measuring human perception in comprehending urban spatial variability and elucidating the influence of urban function Urban experience has always been captured, expressed, and disseminated through human language description or digital photography and recording However, this approach using only image-shallow information makes it difficult to comprehensively understand the deep semantic features of human perception of a scene To close the gap in this effort, thus, using vision language description, we explore participants' emotional changes in city scenes and propose a natural language processing-based framework for the scoring of human perceptions in this study The study is divided into three essential steps 1.

Tom: The first step is transform the deep semantic features of each SVIs in the Place Pulse 2.0 dataset into natural language using the image caption model 2.

Jane: Step 2: A fine-tuned pre-trained BERT model was run as a regression network, and we took each street view image's description and associated emotional label from the Place Pulse dataset to create our urban vision language training dataset, which consists of six human perceptual dimensions 3.

Tom: Step 3: A text topic algorithm was used to investigate semantic information in order to better understand urban settings and their significance The Place Pulse 2.0 dataset includes a variety of human-perceived labels, namely, beautiful, safe, wealthy, depressing, boring, and lively The dataset includes 110988 street view images from 56 cities in 28 nations/regions on 6 continents that were captured between 2007 and 2012 The image captioning network is built on an encoder and a decoder, where the first part focuses on image feature encoding, which processes through the Faster R-CNN network to extract the image feature to vector Furthermore, in the second part, an LSTM with top-down attention decoding each feature’s global weight and input to a language LSTM network during caption generation to describe the urban scene and generate multiple graphic-text The pre-trained BERT model is used for scoring by adding a new function layer after the last transformer encoder layer to scoring the six human perceptions in urban scene The score calculation equation is given by P i = 1/s i(L + b) where s i(L) is averaged score for each perception Results and Comparison The algorithm creates five of the most pertinent Street View descriptions for each image The results show that human perception scoring by deep semantic features performed better than previous studies by machine learning methods with shallow features When compared to LightGBM, XGBoots, Decision Tree, Random Forest, and GBDT the five methods use the same parameters and the training dataset to test dataset ratio is 9:1 Fine-tuned BERT performs best in MAE and R2 Our model's accuracy is improved by about 42% overall (Table 2) In the migration experiment in Hong Kong, the findings demonstrate the highly consistency(R 2 = 0.42) between the predictions made using Fine-tuned BERT and the results of manual scoring Discussion and Conclusion The results demonstrate that (i) the image caption model is more accurate for the text description of street view images, and these texts contain the entity attributes and location attributes of scene objects (ii) the approach of combining BERT embedding vector and LDA topic model is beneficial to mining urban scenes deep semantic characteristics of the nine categories of urban scenes, all of which have distinct category features (iii) compared with traditional measuring methods such as machine learning, the six perceptual dimension measurement tasks based on the pre-trained BERT model have a high SOTA Of course, there are some limitations on this study The first is that our vision language model is limited by its inability to recognize the intangible urban content, which limits its ability to distinguish scenes without visual differences The second is that urban scenes are constantly changing and that street view images can only capture urban scenes at a particular moment An expected future work direction is to further expand the analysis of the causes of urban perception phenomena by combining richer multivariate urban data References ANDERSON, P., HE, X., BUEHLER, C., TENEY, D., JOHNSON, M., GOULD, S.

Paper summary: Tom: So we're looking at this paper called "A natural language processing-based approach: mapping human perception by understanding deep semantic features in street view images." Basically, they’re saying that using just the image itself to measure how people feel about a place isn't good enough anymore because it only gives you shallow information.

Jane: Right, so instead of just looking at the picture, they want to use natural language processing to actually grasp the deep meaning behind what people perceive in those city scenes. The paper claims that this new framework uses a pre-trained natural language model to connect human perception with the actual sense of a scene.

Lu: The core idea here is using vision language description to close that gap, because previous work often focused only on the physical characteristics of space when measuring emotional responses <ref:2311.17354#pg2>. They're proposing a framework where they take the descriptions from street view images and feed them into a pre-trained BERT model for scoring sentiment in the language of those urban scenes <ref:2311.17354#pg3>.

Meng: So, what's the starting point for this? Where do they get their data to train this whole system up?

Tom: They use a dataset called Place Pulse two point zero, which has a variety of human-perceived labels like beautiful, safe, wealthy, depressing, boring and lively <ref:2311.17354#pg1>. They also use an image captioning network to extract those description details from each street view image <ref:2311.17354#pg2>.

Jane: And then they fine-tune a pre-trained BERT model, adding a regression function for those six human perceptual dimensions, using the descriptions and labels from that Place Pulse dataset to build their urban vision language training dataset <ref:2311.17354#pg3>. That’s how they create the input for their scoring mechanism.

Lu: Before they even get to the final scoring step, they use a text topic algorithm to investigate semantic information, trying to better understand the different urban settings and what makes them significant <ref:2311.17354#pg3>. That helps map out those deep semantic characteristics of the scenes <ref:2311.17354#pg3>.

Meng: Okay, so that sounds like a pretty involved process. How much computational overhead is this thing bringing? What's the real cost to run this at scale?

Tom: They compared their approach against five traditional regression methods, including LightGBM, XGBoots, Decision Tree, Random Forest and GBDT <ref:2311.17354#pg3>. And their fine-tuned BERT model actually performed better in terms of MAE and R squared scores <ref:2311.17354#pg4>.

Jane: The results show that human perception scoring by deep semantic features performed better than previous studies that used machine learning methods with shallow features <ref:2311.17354#pg4>. It seems like the way they’re extracting the description information from the images is what's making such a difference for accuracy.

Paper summary: Lu: The paper shows that their algorithm created five of the most pertinent Street View descriptions for each image, and when compared against those other methods, it was performing better in MAE and R squared scores <ref:2311.17354#pg4>. That’s a strong indication of the value of using these deep semantic features <ref:2311.17354#pg4>.

Meng: But what does that mean when we try to actually deploy this? If you look at their migration experiment in Hong Kong, they found a consistency with R squared equal to zero point four two between the predictions made using Fine-tuned BERT and the results of manual scoring <ref:2311.17354#pg4>. That’s a bit low for production if we're expecting perfect scores.

Jane: It shows that even in a real-world setting like Hong Kong, where things change constantly, there is still a reasonably high consistency between the AI's prediction and what people actually scored manually <ref:2311.17354#pg4>. That gives us some confidence that this mapping approach has real explanatory power in the face of spatial heterogeneity.

Tom: So, to wrap up on this paper, "A natural language processing-based approach: mapping human perception by understanding deep semantic features in street view images," they are using a pre-trained BERT model to score six perceptual dimensions based on deep semantic features extracted via image captioning from the Place Pulse two point zero dataset <ref:2311.17354#pg1>.

Jane: It’s about moving beyond just looking at the physical layout of a scene and instead using language models to understand the actual emotional context embedded in those street view images <ref:2311.17354#pg2>. The authors are suggesting that this deep semantic feature approach provides new ideas for subsequent human perception research <ref:2311.17354#pg4>.

Lu: The limitation they mention is that their vision language model is limited by its inability to recognize the intangible urban content, which limits its ability to distinguish scenes without visual differences <ref:2311.17354#pg6>. They also point out that urban scenes are constantly changing and street view images can only capture urban scenes at a particular moment <ref:2311.17354#pg6>.

Meng: So, the system works well on the data they used, but it's tied to those specific descriptions and snapshots in time <ref:2311.17354#pg6>. It won't magically understand a scene that hasn't been described or captured in that way yet.

Tom: Exactly. The implication for us is that this method of combining BERT embeddings with LDA topic models is beneficial for mining those deep semantic characteristics of urban scenes, especially when you have nine categories of urban scenes <ref:2311.17354#pg6>. It’s a new way to measure things compared to the traditional machine learning methods they benchmarked against <ref:2311.17354#pg4>.

Paper summary: Jane: So, what does this mean for someone just listening to the show? It means that when we talk about how people feel about a place in a city, we can start looking at deeper semantic features derived from language models rather than just surface-level image data <ref:2311.17354#pg2>.

Lu: The real value is in how this approach provides better explanatory power when dealing with spatial heterogeneity, which means understanding why different areas of the city feel different <ref:2311.17354#pg6>. It opens up avenues for future work that combines richer multivariate urban data to understand the causes of urban perception phenomena <ref:2311.17354#pg6>.

Meng: From a deployment standpoint, it’s interesting how they structured the training dataset specifically around those six human perceptual dimensions using both image descriptions and labels <ref:2311.17354#pg3>. That structured approach is what makes the BERT fine-tuning work so well on that specific task.

Tom: The authors are clearly excited about how this NLP-based framework can be used to understand participants' emotional changes in city scenes, which is a big step forward from just looking at physical characteristics <ref:2311.17354#pg2>.

Jane: So, the title of the paper, "A natural language processing-based approach: mapping human perception by understanding deep semantic features in street view images," really captures that shift toward using language models to get past shallow image information to understand the true meaning of a scene <ref:2311.17354#pg1>.

Lu: It’s about how we use pre-trained models like BERT not just for text generation, but for structured regression tasks on complex urban perception data <ref:2311.17354#pg3>. That's a significant application of the model's ability to understand context <ref:2311.17354#pg6>.

Meng: For production, the biggest hurdle is that as Lu mentioned, street view images are only snapshots in time, and the model learns from those specific descriptions <ref:2311.17354#pg6>. So deployment would need to account for that temporal aspect carefully.

Tom: It’s a solid approach because it's moving away from shallow features toward deep semantic features, and the empirical results show it beats traditional regression methods in terms of MAE and R squared scores <ref:2311.17354#pg4>.

Jane: So, to summarize, this paper suggests that by using a pre-trained natural language model on image descriptions from street view images, we can better map human perception by capturing deep semantic features about the scene <ref:2311.17354#pg2>.

Lu: This framework gives us a new tool for analyzing urban settings because it helps us understand the distinct category features of different urban scenes <ref:2311.17354#pg6>. It's an improvement over previous methods that relied on only physical space <ref:2311.17354#pg2>.

Meng: And for anyone building something like this, the key seems to be that structured training dataset built from image descriptions and labels, which is what they did with the Place Pulse dataset <ref:2311.17354#pg3>. That's where you need to focus your effort.

Paper summary: Tom: We’re seeing promising results here because the fine-tuned BERT model performs well against established methods like LightGBM and XGBoots <ref:2311.17354#pg4>. That number, an improvement of about forty-two percent overall in accuracy, is what gives this approach traction.

Jane: It’s a framework that uses vision language description to propose a natural language processing-based framework for the scoring of human perceptions in city scenes <ref:2311.17354#pg2>. It’s about connecting the visual input to a richer linguistic understanding of what makes a place feel beautiful or safe <ref:2311.17354#pg6>.

Lu: The paper shows that combining BERT embedding vectors with an LDA topic model is beneficial for mining those deep semantic characteristics of urban scenes <ref:2311.17354#pg6>. It’s a powerful combination for extracting meaning from the text descriptions <ref:2311.17354#pg6>.

Meng: So, in short, it uses a pre-trained language model to score human perception using deep semantic features instead of just shallow image information <ref:2311.17354#pg1>. It’s an interesting direction for urban science because it addresses the limitations of previous studies <ref:2311.17354#pg2>.

Tom: That’s where we are with this paper on "A natural language processing-based approach: mapping human perception by understanding deep semantic features in street view images." It moves the field toward using language models for deeper understanding of urban experience <ref:2311.17354#pg2>.

Jane: It suggests that by focusing on the linguistic perspective, we can better capture how people feel about a place compared to just measuring physical characteristics of space <ref:2311.17354#pg2>.

Lu: The implication is that future research should expand on combining this with richer multivariate urban data to understand the causes of urban perception phenomena <ref:2311.17354#pg6>. That’s where the real potential lies for expanding this idea.

Meng: For now, it’s a solid result showing that deep semantic features from language models can outperform shallow feature methods in measuring these six perceptual dimensions <ref:2311.17354#pg4>. That's what we need to see before we think about scaling it up <ref:2311.17354#pg6>.

Tom: So, the paper is demonstrating that this new framework based on a pre-train natural language model offers better performance than previous machine learning methods with shallow features <ref:2311.17354#pg4>.

Jane: It’s about using vision language description to propose a natural language processing-based framework for the scoring of human perceptions in city scenes <ref:2311.17354#pg2>.

Lu: And it shows that this approach provides new ideas for subsequent human perception research, along with better explanatory power in the face of spatial heterogeneity <ref:2311.17354#pg6>.

Meng: So, the big picture is that we can start using language models to understand the emotional context of urban scenes in a way that was previously difficult <ref:2311.17354#pg2>.

Conclusion: Tom: So we're wrapping up this paper about mapping human perception using deep semantic features from street view images, and it’s titled "A natural language processing-based approach: mapping human perception by understanding deep semantic features in street view images."

Jane: Basically, this study is using a pre-trained natural language model to understand the actual meaning behind what people feel when they look at city scenes. It's trying to go beyond just looking at the physical layout of the picture.

Tom: Right, so we’re talking about taking those raw street view images and using an AI to extract deep semantic features from them, and then using that language model to score things like how beautiful or safe a place feels.

Jane: Yes, it suggests that by focusing on the linguistic side—the descriptions of the scenes—we can get a much richer understanding of human emotion in those urban spaces.

Tom: The authors are showing that this deep semantic feature approach performs better than previous methods that only used shallow image information, and they compare it against five traditional regression techniques.

Jane: It's interesting because the paper points out that this method gives a new way to study human perception, especially when there’s a lot of different types of urban scenes present.

Tom: Yeah, what I find compelling is that they use the Place Pulse dataset for training and fine-tuning BERT specifically for those six perceptual dimensions.

Jane: That structured training setup is key, because it shows how combining image descriptions with human labels helps build a really accurate scoring system.

Tom: And the results show that this approach improved accuracy by about forty-two percent overall when tested against those other machine learning methods.

Jane: It suggests that for researchers looking at urban experience, you can start thinking about using language models to capture the actual feeling of a city rather than just its physical attributes.

Tom: Exactly. It's moving the focus from what the picture *looks* like to how the picture is *described*, which is a big shift for how we analyze perception.

Jane: But it also leaves us wondering about those limitations they mention, especially with street view images only capturing a scene at one specific moment in time.

Tom: True. They flag that the vision language model can't recognize things that aren't visually different, which limits its ability to distinguish scenes without a clear visual difference.

Jane: So the next thing we need to look into is how this system handles those changes over time, and what kind of richer urban data we could combine with it for more robust results. **Outro Music begins to fade in**