SCPP: A Unified Python Library for Soft Clustering

Summary

SCPP (Soft Clustering Python Package) is an open-source Python framework for soft clustering, which assigns each observation a vector of fractional memberships rather than a single cluster label. The paper states: "SCPP establishes a canonical, scikit-learn-compatible estimator interface that standardizes model training, prediction, membership representation, evaluation, and benchmarking across heterogeneous soft clustering methods, including fuzzy, probabilistic, graph-based, matrix factorization, and deep learning methods."

The framework currently integrates 40 representative algorithms and provides a comprehensive benchmarking comprising datasets, clustering quality metrics, and standardized runtime, memory, and scalability evaluation. The source code is publicly available at https://github.com/soft-clustering/soft-clustering.

The paper identifies a problem: "Despite extensive methodological advances, the software ecosystem for soft clustering remains fragmented. Existing implementations are distributed across independent Python libraries that differ in algorithm coverage, APIs, parameter conventions, membership representations, and benchmarking support." Most libraries focus on a single family of methods rather than providing a common abstraction across heterogeneous paradigms, hindering reproducibility, systematic benchmarking, and fair empirical comparison.

The contributions of the work are fourfold:

1. A unified software abstraction for soft clustering: SCPP introduces a scikit-learn-compatible estimator interface that enables diverse soft clustering algorithms to be trained, evaluated, and compared through the same workflow, independent of their underlying methodology.

2. Comprehensive algorithmic coverage within a common framework: The library unifies 40 representative soft clustering algorithms spanning the principal methodological families under a consistent API.

3. An integrated benchmarking and reproducibility ecosystem: SCPP provides standardized datasets, clustering quality metrics, runtime, memory, and scalability benchmarking modules that facilitate fair and reproducible empirical evaluation.

4. Research-grade software engineering infrastructure: "To support reliable scientific software, the package is distributed through PyPI under the MIT license and is accompanied by comprehensive documentation, extensive examples, automated testing comprising 241 unit tests across 41 modules, and continuous integration workflows."

Design and Implementation: The central design principle is that heterogeneous soft clustering methods should expose a common software abstraction despite substantial differences in their mathematical formulations and optimization procedures. This abstraction standardizes the complete lifecycle of a soft clustering estimator, including model configuration, training, prediction, membership representation, evaluation, and benchmarking.

The architecture is organized into five software layers: end users interact exclusively through a canonical, scikit-learn-compatible estimator interface; the underlying implementation encapsulates multiple families of soft clustering algorithms within a common API; a dedicated benchmarking layer provides standardized evaluation infrastructure; and the framework is complemented by documentation, automated testing, and seamless integration with the scientific Python ecosystem.

Every estimator implements the standard methods fit, predict, predict proba, and fit predict, while exposing consistent outputs through attributes such as membership, labels, and, when available, cluster centers and weights. The paper provides an example workflow:

from sklearn.datasets import load iris

from soft clustering import FCM

X, = load iris(return X y=True)

model = FCM(n clusters=3, random state=0)

model.fit(X)

U = model.membership

labels = model.labels

Benchmarking: The benchmarking layer comprises 20 benchmark datasets, 12 clustering quality metrics, standardized experiment runners, and dedicated modules for runtime, memory, and scalability analysis. Because every estimator conforms to the same software abstraction, identical evaluation pipelines can be executed across all supported methods.

Algorithm coverage: The 40 algorithms span the following categories:

• Foundational Methods: FCM, GMM, PCM, Gustafson–Kessel, Gath–Geva

• Classical Extensions: Rough K-Means, Latent Dirichlet Allocation (LDA), Probabilistic Latent Semantic Analysis (PLSA), Subtractive Clustering

• Modern Variants: Kernelized FCM, Fuzzy-Possibilistic C-Means, Adaptive FCM with Spatial Regularization, Soft Kernel Spectral Clustering, Soft DBSCAN-GM, Fuzzy Competitive Clustering, Multivariate Beta Mixture Models, Beta-Gaussian Mixture Models, Evidential C-Means, EVCLUS

• Graph and Community Detection: BigCLAM, Bayesian Nonnegative Matrix Factorization, Mixed Membership Stochastic Block Models, Gumbel-Softmax Community Detection, NOCD, Deep Modularity Networks (DMoN)

• Ensemble and Advanced Methods: Soft Clustering Ensembles, Adaptive FCM with Graph Embedding, Robust Projected Fuzzy K-Means, Deep Fuzzy K-Means, Federated Fuzzy Clustering, Semi-supervised Fuzzy Clustering, Fuzzy Subspace Clustering, Fuzzy Soft Set Clustering

• Specialized Techniques: Entropy C-Means, CAF-HFCM, Collaborative Annealing FCM, Fuzzy Color Clustering, Word-Based Soft Clustering, Similarity-Based Soft Clustering, KMART

Datasets: The benchmark dataset collection includes 20 datasets organized into three groups: real-world (iris, wine, digits, breast cancer, olivetti faces), synthetic (blobs, moons, circles, anisotropic blobs, varied blobs, high dimensional blobs), and OpenML (glass, vehicle, ecoli, yeast, segment, satimage, letter, pendigits, optdigits).

Evaluation Framework: SCPP provides a unified evaluation framework for both hard and soft clustering methods. In addition to conventional clustering metrics, the framework includes soft-clustering-specific measures that directly operate on the membership matrix U. These include partition coefficient, modified partition coefficient, partition entropy, fuzzy hypervolume, Xie-Beni index, fuzzy compactness, and fuzzy separation. For hard cluster assignments, SCPP integrates widely used internal and external evaluation metrics from scikit-learn, including silhouette, Calinski-Harabasz, Davies-Bouldin, adjusted Rand index (ARI), and normalized mutual information (NMI).

Computational Performance Benchmarking: The framework evaluates runtime, scalability, and memory consumption using a unified interface. The runtime benchmark repeatedly executes the training procedure and reports average execution time and standard deviation across multiple runs. Scalability is assessed on progressively larger subsets of the same dataset, with sample sizes ranging from 100 to 10,000 observations. Memory profiling measures resident memory usage before and after model fitting, computes memory overhead, and records peak observed memory footprint.

Testing Infrastructure: The testing framework consists of 241 unit and integration tests distributed across 41 dedicated test modules, validating every one of the 40 clustering algorithms. The verification strategy includes output validation, mathematical consistency (non-negativity of memberships, normalization constraints, parameter ranges), model state verification, parameter robustness, edge cases and failure modes, and reproducibility (repeated executions with identical random seeds produce identical outputs).

Mo2oM Migration Case Study: The paper demonstrates a real-world use case integrating SCPP into Mo2oM, a monolithic-to-microservice migration pipeline. Originally hardwired to a single graph community detection method (NOCD), after integration Mo2oM can select from multiple graph-based soft clustering methods (NOCD, BayesianNMF, BIGCLAM) through a unified API, enabling pluggable clustering method selection, rapid experimentation, and preserving soft membership semantics for overlap-aware service decomposition.

Integration with Python Ecosystem: The paper provides guidance for integrating SCPP with the PyData stack, covering tabular data preprocessing (pandas, scikit-learn), text data embedding (sentence-transformers, TF-IDF), image feature extraction (torchvision ResNet), graph/relational data construction (networkx, scipy sparse matrices), and evaluation metrics. The recommended end-to-end workflow is: (1) load and preprocess raw data into a feature matrix X or affinity matrix A; (2) fit the model with the unified SCPP interface; (3) extract and interpret the membership matrix U to obtain hard labels, confidence scores, entropy, and cluster masses; (4) evaluate comprehensively using both soft-specific internal metrics (PC, PE, XB) and standard external metrics (ARI, NMI).

The framework adopts a lightweight dependency strategy, relying primarily on NumPy and SciPy while requiring PyTorch only for graph-based and deep learning models. The architecture is designed to facilitate long-term extensibility and integration with the broader scientific Python ecosystem.

Based on the SCPP paper, here are the specific improvements I can make to AI systems and what the improved systems can do:

• Implement a scikit-learn-compatible estimator interface with standardized fit, predict, predict proba, and fit predict methods across all clustering algorithms

• Standardize membership matrix output (U ∈ [0,1](n×K)) with consistent attribute naming (membership, labels, centers)

• Add automatic parameter validation for fuzzification coefficients, convergence tolerances, and initialization strategies

• Integrate 40 algorithms across 7 methodological families: fuzzy (FCM, PCM, GK), probabilistic (GMM, LDA, PLSA), graph-based (BigCLAM, NOCD, DMoN), matrix factorization (BayesianNMF), deep learning, ensemble, and specialized methods

• Implement a pluggable architecture allowing new algorithms to inherit the unified interface and immediately work with existing benchmarking infrastructure

• Add soft-clustering-specific metrics that operate directly on membership matrices: partition coefficient, partition entropy, fuzzy hypervolume, Xie-Beni index, fuzzy compactness, fuzzy separation

• Integrate standard hard clustering metrics (ARI, NMI, silhouette, Calinski-Harabasz, Davies-Bouldin) for defuzzified labels

• Provide a unified evaluation interface that automatically computes all applicable metrics from labels, memberships, and prototypes

• Implement standardized runtime benchmarking with repeated fits, mean/std reporting, and prediction time measurement

• Add memory profiling with tracemalloc for peak memory, memory overhead, and resident memory tracking

• Create scalability analysis across sample sizes (100 to 10,000) measuring both runtime and memory growth

• Implement 241 unit tests across 41 modules validating: output dimensions, mathematical invariants (membership normalization, non-negativity), parameter robustness, edge cases, and deterministic behavior with fixed random seeds

• Add continuous integration workflows and comprehensive documentation for reproducible experimentation

• Build preprocessing pipelines for tabular data (scaling, encoding, imputation), text data (LLM embeddings, TF-IDF), image data (ResNet features), and graph data (affinity matrix construction)

• Integrate with PyData ecosystem: pandas/polars, scikit-learn, sentence-transformers, torchvision, networkx, faiss

• Compare 40+ clustering algorithms with identical code — swap between FCM, GMM, NOCD, and BayesianNMF by changing one line

• Run complete benchmark suites (quality, runtime, memory, scalability) with a single API call

• Evaluate soft clustering quality using both fuzzy-specific metrics (PC, PE, XB) and standard metrics (ARI, NMI)

• Reproduce experiments exactly with deterministic random seeds and standardized evaluation protocols

• Handle heterogeneous data types (tabular, text, image, graph) through unified preprocessing pipelines

• Extract richer insights from membership matrices: confidence scores, uncertainty entropy, cluster mass, and ambiguous sample identification

• Deploy in production with validated numerical stability, edge-case handling, and predictable failure modes

• Extend with new algorithms by implementing a single estimator interface — new methods immediately work with benchmarking, evaluation, and visualization tools

• Integrate into existing pipelines (e.g., Mo2oM migration tool) with pluggable clustering method selection

• Run automated testing with 241 unit tests ensuring mathematical correctness and robustness

• Improve software modularization by identifying overlapping components with cross-cutting responsibilities (as demonstrated in the Mo2oM case study)

• Enhance community detection in social networks with soft memberships capturing overlapping community structures

• Improve topic modeling with semantic-aware clustering that preserves uncertainty in document assignments

• Process datasets from 100 to 10,000+ samples with linear memory scaling (0.26 KB per sample for FCM)

• Handle sparse matrices and disconnected graph structures without failure

• Validate mathematical constraints automatically: membership normalization (sum to 1), non-negativity, parameter ranges

• Provide confidence-calibrated outputs where high-entropy points are flagged for manual review or active learning

In this paper, we present SCPP (Soft Clustering Python Package), an open-source Python framework for soft clustering. SCPP establishes a canonical, scikit-learn-compatible estimator interface that standardizes model training, prediction, membership representation, evaluation, and benchmarking across heterogeneous soft clustering methods, including fuzzy, probabilistic, graph-based, matrix factorization, and deep learning methods. The framework currently integrates 40 representative algorithms together with a comprehensive benchmarking comprising datasets, clustering quality metrics, and standardized runtime, memory, and scalability evaluation. SCPP further provides extensive documentation, practical examples, automated testing, and seamless integration with the scientific Python ecosystem, enabling reproducible experimentation and straightforward extension with new algorithms. The source code is publicly available at https://github.com/soft-clustering/soft-clustering.