A Cascaded Unsupervised-Supervised NLP Pipeline for Detecting Accusatory Language in Public Procurement

This paper proposes a cascaded hybrid learning strategy that integrates unsupervised clustering and supervised classification to detect accusatory or whistleblowing-style comments in Ecuador's public procurement system. The study analyzes metadata from Ecuador’s Sistema Oficial de Contratación Pública (SOCE), focusing on participant comments generated during the pre-contractual phase. The authors state: "We propose a hybrid modeling framework that integrates unsupervised clustering and supervised classification within a natural language processing (NLP) pipeline to uncover latent patterns and detect potentially irregular procurement processes."

The methodology involves several stages: data wrangling, embedding generation, clustering, cluster identification, classification, and model testing. The datasets were obtained through the Kapak framework, which automates collection of public procurement data from SOCE. The supervised dataset contained 5,005 questions, of which 143 were labeled as accusatory, with inter-annotator agreement measured at κ = 0.72 using Fleiss' Kappa. The unsupervised dataset contained 100,000 rows initially, reduced to 92,579 after cleaning.

Three embedding strategies were evaluated: LLaMA 3.2 (1B parameters), RoBERTa, and a domain-trained Word2Vec model. Mean pooling was applied to generate fixed-length sentence representations. The authors found that the combination of domain-trained Word2Vec embeddings, GMM-based clustering, and a Random Forest classifier achieves high precision and recall, even under severe class imbalance.

For clustering, both k-means and Gaussian Mixture Models (GMMs) were evaluated. The results showed that the combination of Word2Vec embeddings and GMM achieved the best clustering performance, yielding the highest Silhouette score and the greatest concentration of accusatory phrases within a single cluster. The optimal number of clusters was between k = 3 and k = 5. The authors note that Transformer-based embeddings suffered from anisotropy, with average cosine similarity of 0.91 for RoBERTa and 0.79 for LLaMA, while Word2Vec had a significantly lower average cosine similarity of 0.36, indicating a more isotropic space.

For cluster identification, a keyword-based strategy was used with five high-impact keywords: Direccionado (Directed procurement), Imposible (Impossible), Corrupción (Corruption), Ilegal (Illegal), and Prohibido (Prohibited). The cluster with the highest cumulative frequency of at least four of these five keywords was identified as the most accusatory cluster. This approach proved robust across both labeled and unlabeled datasets.

For classification, three classifiers were evaluated: Random Forest, Gaussian Naive Bayes (GNB), and Support Vector Machine (SVM). The classification pipeline included cluster filtering, SMOTE for handling class imbalance, and Stratified K-Fold Cross-Validation. The results showed that the Random Forest classifier with SMOTE achieved the best overall performance, maintaining a strong balance between precision and recall and yielding an F1-score of 0.87 for the positive (accusatory) class. The Precision–Recall curve demonstrated robustness with an Average Precision (AP) score of 0.93.

The final performance metrics showed that the complete pipeline based on Word2Vec required approximately 157 minutes for full training, with conservative CPU and RAM usage and no GPU requirements. When using pre-trained models, predicting whether an unlabeled sentence is accusatory requires approximately 1 to 3 seconds, making the pipeline suitable for near real-time deployment.

In the unsupervised testing stage, the pipeline identified 1,892 accusatory candidates out of 92,579 unlabeled samples, compared to 122 out of 4,841 in the supervised dataset. Manual validation of a random subset of positive predictions confirmed an empirical precision of approximately 0.84, consistent with the supervised evaluation.

The paper also discusses cross-country generalizability, noting that while the methodological framework is broadly generalizable, practical deployment across Latin America is constrained primarily by data accessibility rather than modeling limitations. The authors conclude that transparent and effective NLP pipelines for corruption risk detection could be built with modest computational cost and that lightweight, domain-adapted NLP architectures can effectively support risk identification and enhance transparency in public procurement systems without requiring large-scale computational infrastructure.

Improvements to AI Systems:

1. Hybrid Unsupervised-Supervised Pipeline Integration

• Implement a cascaded framework where unsupervised clustering (GMM) first groups text embeddings to isolate high-risk clusters, then supervised classification (Random Forest with SMOTE) refines detection within those clusters. This reduces noise and improves precision under extreme class imbalance (e.g., 2.8% positive rate).

• Capability: Detect rare, irregular patterns (e.g., fraud indicators) in large-scale text corpora without requiring massive labeled datasets.

2. Anisotropy-Aware Embedding Selection

• Use domain-trained Word2Vec embeddings instead of Transformer-based models (RoBERTa, LLaMA) when text is short and domain-specific, because Transformers exhibit high anisotropy (cosine similarity 0.91 vs. 0.36 for Word2Vec), which degrades clustering quality.

• Capability: Build lightweight, interpretable NLP systems that maintain semantic separation in specialized domains (e.g., legal, procurement) without GPU-heavy infrastructure.

3. Keyword-Driven Cluster Identification for Labeling

• Automatically identify the most relevant cluster for a target class using a small set of high-impact keywords (e.g., corruption, illegal) and cumulative frequency thresholds (≥4 of 5 keywords). This replaces manual labeling and scales to unlabeled datasets.

• Capability: Rapidly bootstrap labeled data for new domains or languages by leveraging domain-specific lexicons, reducing annotation effort.

4. Class-Imbalance Robust Classification

• Combine cluster filtering with SMOTE oversampling and Stratified K-Fold Cross-Validation to train classifiers that achieve high F1 (0.87) and Average Precision (0.93) even when positive class is <3% of data.

• Capability: Deploy reliable risk-detection systems in real-world settings where fraudulent or accusatory instances are rare, minimizing false negatives.

5. Resource-Efficient Deployment

• Use CPU-only, low-memory pipelines (157 minutes training, 1–3 seconds per prediction) with no GPU requirements, enabling near-real-time inference on modest hardware.

• Capability: Run continuous monitoring of public procurement, social media, or whistleblower platforms in resource-constrained government agencies or NGOs.

6. Cross-Domain Generalizability via Framework, Not Data

• Reuse the methodological pipeline (embedding → clustering → keyword identification → classification) across different countries or sectors, swapping only domain-specific embeddings and keyword lexicons.

• Capability: Adapt corruption-risk detection to other Latin American or global procurement systems with minimal retraining, limited only by data access.

7. Empirical Precision Validation for Unsupervised Outputs

• Incorporate manual validation of a random subset of positive predictions (empirical precision 0.84) to calibrate confidence thresholds and provide human-auditable outputs.

• Capability: Generate trustworthy alerts that can be legally or administratively actionable, with quantified uncertainty for downstream investigation.

Public procurement involves the allocation of substantial financial resources; therefore, continuous oversight through audits, controls, and monitoring mechanisms is essential. However, stakeholder comments and publicly available government data are often underutilized, despite their potential to reveal procedural irregularities. To address this gap, this paper analyzes metadata from Ecuador's Sistema Oficial de Contrataci'on P'ublica (SOCE, Official Public Procurement System), with particular emphasis on participant comments generated during the pre-contractual phase. We propose a hybrid modeling framework that integrates unsupervised clustering and supervised classification within a natural language processing (NLP) pipeline to uncover latent patterns and detect potentially irregular procurement processes. Semantic embeddings are generated using Word2Vec, LLaMA, and RoBERTa, followed by Gaussian Mixture Models (GMMs) for unsupervised clustering. A supervised classification stage is then applied to identify accusatory or whistleblowing-style comments. Experimental results show that the combination of domain-trained Word2Vec embeddings, GMM-based clustering, and a Random Forest classifier achieves high precision and recall, even under severe class imbalance. These findings demonstrate that lightweight, domain-adapted NLP architectures can effectively support risk identification and enhance transparency in public procurement systems without requiring large-scale computational infrastructure.