LLM-powered Query Expansion for Enhancing Boundary Prediction in Language-driven Action Localization

The gist: This paper proposes an LLM-powered query expansion method to enhance boundary prediction in language-driven action localization by generating detailed textual descriptions of action start and end boundaries and modeling boundary probabilities using semantic similarities and temporal distances.

How it works

1. The method first expands the original language query by prompting a Large Language Model (LLM) to generate fine-grained textual descriptions of the action start and end boundaries, denoted as expanded queries Qs = 5, Qe = 6. These prompts are crafted to guide the LLM in producing descriptions that are semantically aligned with the original query and explicitly focus on the temporal boundaries of action, such as “Please describe the beginning and ending process in one sentence of the following action”. The expanded queries help capture fine-grained temporal information that is overlooked by the original coarse query.

2. A query-guided temporal modeling module is proposed to enhance video features by focusing on boundary-specific temporal details while maintaining a holistic understanding of the action progression. This module consists of two branches: a local branch and a global branch. The Local Branch enhances the video feature Fv by performing fine-grained temporal modeling for each action stage, using the start query feature Fs, the original query feature Fq, and the end query feature Fe as guidance. The Global Branch enhances the video feature Fv by capturing holistic temporal dependencies across the entire action sequence, using the concatenated query feature F′q = [Fs; Fq; Fe] as guidance. Finally, these enhanced features are weighted fused to obtain the final video feature F′v: F′v = aFg v + b(F s v + F q v + F e v).

How it works

1. To enhance the tolerance to boundary uncertainty during training, a boundary probability modeling module is designed to estimate probability scores of potential action boundaries based on the expanded queries. This module consists of two steps: a pseudo boundary generation step and a probability score estimation step.

2. The Pseudo Boundary Generation step identifies the most confident action boundaries relevant to the expanded queries by calculating scores S p s (i) and S p e (i) for each frame i using semantic similarity sim(Fv,i, Fs) and temporal distance dis(i, τs), and similarly for the end boundary. The pseudo boundaries are determined by maximizing these scores: s′ = arg maxi S p s (i), e′ = arg maxi S p e (i).

3. The Probability Score Estimation step converts rigid boundary annotations into probability scores to better capture boundary uncertainty. For each frame i < s′, its score Ss(i) is calculated based on its temporal distance to the pseudo boundary s′ and its semantic similarity to the start query Qs: Ss(i) = sim(Fv,i, Fs) − dis(i, s′). A probability score ps(i) is then generated using min-max normalization based on these scores.

Results and Contributions

The method's main contributions are summarized as follows:

We propose a query expansion method that expands the language query through LLM to capture more details of action boundaries, effectively reducing the impact of boundary uncertainty in action localization

"We propose a boundary probability modeling module that utilizes the expanded query to transform rigid boundary annotations into probability scores for training, successfully improving the robustness to boundary uncertainty"

The experimental results show that all methods integrated with our modules consistently achieve better performance on all three datasets, highlighting the effectiveness of query expansion and probability score modeling in action boundary prediction. The combination of the two modules achieves the best results. Furthermore, ablation studies confirm that both modules independently contribute to improving performance in all metrics, and their combination demonstrates synergistic roles. The method exhibits robustness to boundary uncertainty, showing relatively smaller performance drop as the level of annotation noise increases compared to QD-DETR. Qualitative results show consistent localization across similar queries, with our method predicting start boundaries consistently across examples for similar queries on the Charades-STA dataset.

Conclusion

We have presented a large language model (LLM)-powered query expansion method to enhance boundary prediction for language-driven action localization by leveraging LLMs to generate expanded boundary queries that provide cues for localization and modeling boundary probabilities using semantic similarities and temporal distances. The proposed modules are modelagnostic and can be seamlessly integrated into existing models of language-driven action localization in an off-the-shelf manner. Extensive experiments on five state-of-the-art models across three datasets validate the effectiveness of our method in reducing the impact of boundary uncertainty and enhancing boundary prediction.

Data Availibility All datasets used in this study are open access and have been cited in the paper. The references list includes works on various aspects such as SlowFast networks for video recognition and LLaMa3-8B Dubey et al (2024) for query expansion. The paper also details hyperparameter tuning, observing that an intermediate value τ = 0.8 achieves the optimal balance for the probability modeling module. The work concludes by showing high quality and consistency of generated queries through user studies, indicating the overall high quality and consistency of the queries generated by our method. The paper also includes qualitative results visualizing boundary probability scores, showing meaningful probability peaks at frames corresponding to the boundary actions described in the expanded queries. The final loss function is defined as L = Lbound + Lorigin, where Lbound is the cross-entropy loss over all frames and Lorigin is the original loss that the base model uses.

--- Page 1 ---

The gist: This paper proposes an LLM-powered query expansion method to enhance boundary prediction in language-driven action localization by generating detailed textual descriptions of action start and end boundaries and modeling boundary probabilities using semantic similarities and temporal distances.

1. Improved Query Expansion via LLMs to capture boundary specifics: The system can generate fine-grained descriptions about motions, poses, or temporal differences that distinguish boundaries, such as The person picks up a cookie with their hand and brings it to their mouth, which provides more detailed boundary cues for localization.

2. Improved Boundary Prediction via Query-Guided Temporal Modeling: The system will enhance video features by focusing on boundary-specific temporal details using the local branch (using start/end query features as key/value) and capturing global structure using the global branch, resulting in a final feature of F′v = aFg v + b(F s v + F q v + F e v).

3. Improved Training Stability via Boundary Probability Modeling: The system will transform rigid boundary annotations into probability scores by calculating semantic similarity between frames and the expanded query as well as the temporal distances between frames and the annotated boundary frames, which reduce over-reliance on a single annotated boundary by providing more flexible and reliable supervision.

4. Improved Robustness to Uncertainty via Probability Score Estimation: By using pseudo boundaries around the groundtruth boundaries, based on two key factors: temporal distance and semantic similarity, the system can generate scores that focus on frames most likely to be action boundaries, even when human annotations are inconsistent.

5. Improved Query Quality via LLM Selection: The system can select an LLM (like LLaMa3-8B) that achieves better performance, as different LLMs have little impact on performance but LLaMa3-8B achieves the best overall performance.