3D Weighted Geometric Graph Neural Networks for Sheep Facial Pain Assessment

This paper presents the 3D Sheep Pain Facial Expression System (3D-SPFES), a novel, monocular depth-aware geometric graph neural network system that integrates each SPFES facial landmark, such as the ears, eyes, and nose, into 3D Euclidean space estimated from a single RGB camera by using VideoDepthAnything, thus preventing the need for specialized depth hardware. Each landmark node includes a feature vector containing its 3D spatial coordinates, estimated surface normal, and facial attribute class embedding. Edges linked to nodes are assigned weights based on an aggregate metric that combines both Euclidean distance and surface co-planarity in a 3D space. A Weighted Geometric Graph Neural Network (WG-GNN) studies this graph using K = 3 geometry-aware message-passing layers enhanced by a scaled dot-product attention method that selectively enhances anatomically relevant inter-landmark messages. The resultant node embeddings are combined into O = 3 pain-level clusters and integrated into a Normalized Pain Score (NPS) within the range of [0, 100%] a confidence-weighted, SPFES-derived scoring method. We additionally propose bilateral detection averaging, which integrates left and right instances of symmetric face features into a singular SPFES-consistent representative node, enhancing test accuracy by 6.7% compared to single-detection baselines. In a particular sheep facial landmark dataset, 3DSPFES WG-GNN system model achieved a validation accuracy of 73.2% and an accuracy of 78.33% ± 2.2%. The proposed system can be deployed on any standard RGB camera platform, including UAVs and mobile devices, without the need for specialist sensing hardware.

The paper addresses limitations of previous work, noting that "Deep learning systems perform mainly within the 2D for a single image domain and take the face as a single-dimension representation, losing sight of the 3D anatomy of sheep and cross-landmark spatial relationships that are intrinsic to the clinically proven Sheep Pain Facial Expression Scale (SPFES)." The authors' previous work introduced a 2D Weighted Graph Neural Network (WGNN) with a cluster classification accuracy of 92.71%, but it was limited by performing primarily within the 2D image plane and depending on a singular camera viewpoint without any 3D geometric details.

The main contributions are fourfold: (1) 3D facial graph representation with monocular depth, proposing a novel 3D-SPFES graph G = (N, E, O, X) in which each SPFES landmark node is embedded in metric 3D space using VideoDepthAnything; (2) Geometric message passing with edge-attention weighting, proposing a Weighted Geometric GNN (WG-GNN) layer that computes geometry-aware messages; (3) Bilateral detection averaging for SPFES-consistent landmark representation, which optimizes test accuracy by 6.7% compared to single-detection baselines; and (4) 3D-SPFES WG-GNN evaluation with global model design, transforming local models to a single global model with four 3D-domain augmentations, resulting in an accuracy of 78.33%±2.2%.

The system model formulates the graph with nodes containing 9 facial expressions (eyes, ears, nose, cheeks, lips, jaw), with each node's feature vector containing pain intensity pi ∈ 0, 1, 2, facial part type embedding, 3D spatial coordinates, and surface normal vector. The 3D back-projection process translates bounding box centroids from image coordinates to 3D coordinate system using the depth map and camera's intrinsic matrix. Edge construction requires both anatomical proximity in 3D space with threshold τ and pain-level regularity with tolerance ϵ, with edge weights computed using a radial basis function of 3D space combined with a normal-alignment attribute.

The geometric message passing integrates structural priors with data-driven attention, where the message from adjacent node j to node i at layer k is calculated as mij = ωij · ϕm(ψi, ψj, δij, d⊤i dj), with ωij representing the precomputed geometric edge weight and ϕm being a learned edge network. A scaled dot-product attention method allows the system to assign differential weights to communication from various neighbors based on learned feature compatibility.

The experimental work used a sheep facial landmarks dataset developed using SPFES parameters, divided into training, testing, and validation images. The model was trained using PyTorch on a GPU-equipped workstation with a GeForce RTX A3000, with ResNet-18 visual backbone initialized with ImageNet pretrained weights and fine-tuned end-to-end with a learning rate of 5 × 10−5, while the GNN head used a learning rate of 5 × 10−4. Four domain-specific augmenting methods were used: 3D position jitter, horizontal axis flipping, depth scale jitter, and facial part feature dropout.

The evaluation results show that the 3D-SPFES WG-GNN global model gives a significant gain in all consensus evaluation metrics without requiring further training. Cohen's κ rises from 0.180 (slight consistency) to 0.473 (moderate consistency), indicating a 163% relative increase. The ear part shows significant enhancement, increasing from κ = 0.098 to κ = 0.527, but the eyes part improves from a negative κ = −0.053 to a positive κ = 0.470. The detailed breakdown shows 80% accuracy in the ear and eye areas, while 75% is achieved in the nose, with an overall accuracy of 78.33%.

The attention pattern analysis shows that the Eyes node represents the focal point of the facial graph: Ear→Eyes, Eyes→Ear, Eyes→Nose, and Nose→Eyes all exhibit α = 1.0, whereas Ear↔Nose connections are inactive. This hub-and-spoke layout corresponds with the clinical SPFES observation that eye expressions correlate with both auricular and nose pain symptoms.

The NPS analysis shows that no-pain images focus primarily at an NPS of approximately 0%, whereas mild-pain images range from 8% to 17%, with a median of 13.5%. The substantial divergence between no-pain and mild-pain NPS values indicates that the NPS offers a clinically important continuous assessment of pain intensity.

The ablation study shows three major processes: the change from C1 to C2 shows the complexity of the 3D-SPFES WG-GNN system model by eliminating the circular pain-score leakage, resulting in a decrease in accuracy from an apparent 100% to an actual 68.3%; the shift from C5 to C6 represents the integrated impact of local 3D-SPFES WG-GNN models and bilateral detection averaging, resulting in an accuracy enhancement of 11.7 percentage points; and the transition from C6 to C7 shows that the global 3D-SPFES WG-GNN model approach results in the most significant Kappa increase (+0.293), enhancing the model from slight agreement (κ = 0.180) to mild agreement (κ = 0.473) and nearly tripling the mild pain F1-score from 0.22 to 0.58 without retraining.

The paper acknowledges three main limitations: the severe pain category is significantly limited in the dataset with insufficient test samples; the NPS for the individual severe-pain test image (11.1%) falls within the mild-pain spectrum, showing that the cluster-weight parameterization has not yet achieved a strictly monotonic severity hierarchy; and the geometric weights ω = 0.013 are still considerably lower than the learned attention weights (α = 1.0), showing that the integration of camera intrinsic calibration and dense surface reconstruction could further enhance the geometric aspect of the message-passing mechanism.

The conclusion states that the proposed system model is scalable with any standard RGB camera platform and is relevant to a wider range of facial behavioral biometrics, including livestock pain assessment and clinical pain monitoring.

Improvements to AI Systems:

1. Monocular 3D Geometric Graph Construction – Replace 2D landmark-based models with a depth-aware graph neural network that back-projects RGB image landmarks into metric 3D space using a monocular depth estimator (e.g., VideoDepthAnything). This enables the AI to reason about spatial relationships, surface normals, and co-planarity, not just pixel coordinates.

2. Geometry-Aware Message Passing with Edge Attention – Implement a Weighted Geometric GNN (WG-GNN) layer where messages between nodes are weighted by a combination of precomputed geometric edge weights (Euclidean distance + surface normal alignment) and learned scaled dot-product attention. This allows the AI to selectively amplify anatomically relevant connections (e.g., eye↔ear, eye↔nose) while suppressing irrelevant ones, mimicking clinical knowledge.

3. Bilateral Detection Averaging – For symmetric facial features (left/right ears, eyes), fuse detections into a single representative node by averaging their 3D coordinates and features. This reduces noise and improves test accuracy by 6.7% over single-detection baselines, making the system more robust to pose and occlusion.

4. Confidence-Weighted Continuous Pain Scoring – Replace discrete classification with a Normalized Pain Score (NPS) in [0, 100%], derived from cluster probabilities and confidence weights. This provides a continuous, clinically interpretable severity metric, enabling finer-grained monitoring (e.g., distinguishing no-pain vs. mild-pain with clear separation).

5. Global Model with Domain Augmentations – Train a single global model (instead of local per-subject models) using 3D-specific augmentations: position jitter, horizontal flipping, depth scale jitter, and facial part feature dropout. This improves generalization and inter-rater consistency (Cohen’s κ from 0.180 to 0.473) without retraining on new subjects.

6. Circular Pain-Score Leakage Prevention – Avoid using the pain label as an input feature during training (which caused 100% apparent accuracy). Instead, ensure the model learns purely from geometric and appearance features, leading to realistic accuracy (68–78%).

7. Hub-and-Spoke Attention Regularization – Enforce or bias attention patterns to follow clinical anatomical priors (e.g., eyes as central hub connecting ears and nose). This can be done by initializing attention weights or adding a regularization loss that encourages clinically known correlations, improving interpretability and robustness.

---

What the Improved AI System Can Do:

• Deployable on standard RGB cameras (UAVs, smartphones, farm CCTV) without depth sensors, making 3D pain assessment accessible in field settings.

• Provide continuous, real-time pain scores for sheep (and potentially other animals or humans) with moderate inter-rater agreement (κ ≈ 0.47), useful for veterinary triage and welfare monitoring.

• Automatically identify which facial regions contribute most to pain (e.g., eyes, ears) via attention weights, aiding clinical validation and explainability.

• Generalize across individual subjects without retraining, thanks to global model design and 3D augmentations.

• Handle symmetric features robustly under varying head poses and partial occlusions via bilateral averaging.

• Serve as a blueprint for other facial behavior analysis tasks (e.g., human pain, fatigue, emotion) by replacing the SPFES landmarks with task-specific ones and reusing the geometric GNN architecture.