The COTe score: A decomposable framework for evaluating Document Layout Analysis models

Document Layout Analysis (DLA) models typically rely on general object detection metrics like IoU, F1, or mAP, which are ill-suited for printed media because they treat images as 2D projections of 3D space rather than natively 2D tessellations. This discrepancy leads to misleading performance interpretations; therefore, the paper introduces the Structural Semantic Unit (SSU) and the Coverage, Overlap, Trespass, and Excess (COTe) score as a decomposable framework to provide a more robust and informative evaluation of DLA model quality.

The Core Problem: Limitations of Traditional Metrics

Traditional metrics are often inadequate because they fail to capture the specific challenges inherent in parsing printed media. For instance, IoU is sensitive to the arbitrary choice of granularity at which text is labelled (e.g., line or paragraph), resulting in misleading interpretations of model quality. Furthermore, these metrics do not account for critical failure modes such as whether predictions overlap or trespass across semantic boundaries. This lack of comprehensive evaluation frameworks leaves practitioners with generalist Object Detection methods that cannot provide meaningful insight into model performance, leading to a gap between prediction and actual task-relevant outcomes.

The Structural Semantic Unit (SSU)

The SSU is introduced as a relational labelling approach designed to address the practical manifestation of incommensurability arising from differing text-labelling granularities. The SSU shifts the focus from the physical position of text to its meaning by grouping labels into semantic and layout units. A single SSU is defined as any number of labelled text regions which share a common layout and semantic unit. For two regions to belong to the same SSU, they must satisfy five conditions: (1) same class, (2) same structural unit, (3) same semantic unit, (4) adjacent in reading order, and (5) adjacency in reading order. This approach provides robustness to the level at which the text itself was labelled.

The COTe Score: A Decomposable Metric

The COTe score is a decomposable metric for measuring page parsing quality that is natively 2D. It measures performance across four distinct elements:

1. Coverage (C): Measures how much of the ground truth is covered by predictions, defined as the normalized area of intersection between prediction and SSU masks.

2. Overlap (O): Concerns repeated phrases and sentences, penalizing stacked prediction areas.

3. Trespass (T): Penalizes breaches of tessellation logic by measuring how much a prediction covers an SSU that does not belong to its assigned ground truth SSU.

4. Excess (E): Acts as a support metric, measuring the area outside the bounds of any SSU, contextualizing core metrics and showing how well the predictions fit the SSUs.

The overall COTe score is calculated additively: COTe score = C − O − T. This additive structure means that 1 is a perfect score, and negative scores indicate that the sum of Trespass and Overlap exceeds coverage.

Granularity Robustness and Pragmatic Competence

A key finding is the granularity robustness of the COTe score, which largely holds even without explicit SSU labelling. This suggests that practitioners can gain much of the benefit without retraining models on custom SSU labels. The paper demonstrates this by showing that when ground truth is at line level but predictions are at paragraph level (or vice versa), traditional metrics yield poor results, whereas the COTe score remains highly informative and reliable. This robustness stems from the metric's pragmatic competence, which allows it to track task-relevant outcomes across a wider range of granularities and schemas without requiring model retraining.

Model Evaluation and Diagnostics

The authors evaluate five common DLA models (DocLayout-YOLO, Heron, PPDoclayout) on three datasets (NCSEV2, HNLA2013, DocLayNet). The results show that no single model is universally best; performance varies by dataset. For example, DocLayout-YOLO showed high coverage but also extensive Overlap and Trespass in one case. Crucially, the COTe score reveals distinct failure modes across models—such as breaching semantic boundaries or repeatedly parsing the same region—allowing practitioners to debug models through visual inspection and identify specific weaknesses related to training data or architecture. The paper concludes that the COTe framework provides a more insightful evaluation than IoU or F1, as it penalizes errors in spatial and semantic structure.

Practical Release

To support community adoption, the authors release an SSU labelled dataset and a Python library called cotescore for applying the COTe framework in DLA projects. This allows researchers to use the Coverage, Overlap, Trespass, and Excess to identify system weaknesses leading to more robust and accurate models.

Here are the specific improvements that an AI system can achieve by implementing the concepts from this paper:

1. Improve model evaluation robustness against annotation granularity mismatch: The system will no longer rely solely on IoU/F1/mAP, which are sensitive to differing label levels (e.g., line vs. paragraph). Instead, it will use the deomposable COTe score to quantify performance across different structural and semantic units (SSUs), providing a more reliable measure of true parsing quality, especially when ground truth is loosely labeled or predictions are at a finer granularity than expected.

2. Diagnose specific model failure modes: The COTe score's decomposed metrics (Coverage, Overlap, Trespass) allow the system to pinpoint exactly why a model fails. It can distinguish between:

3. Identify and quantify semantic boundary breaches (high Trespass).

4. Detect redundant phrase parsing (high Overlap).

5. Determine poor content identification in sparse areas (low Coverage/Excess).

6. Improve post-processing strategies: By identifying high Trespass or Overlap, the system can trigger targeted mitigation steps, such as re-evaluating adjacent regions or applying context-aware filtering before final text extraction.

7. Enhance model selection and training feedback: The paper shows how different model architectures (e.g., DocLayout-YOLO vs. Heron) exhibit distinct failure modes across datasets (NCSE, HNLA2013). This diagnostic capability allows researchers to select models based on the specific error profile needed for their application or to guide targeted fine-tuning efforts toward correcting identified weaknesses (e.g., if a model consistently shows high Overlap, it suggests issues with phrase segmentation).

8. Create a more pragmatically competent evaluation framework: The COTe score shifts the focus from the physical projection to the semantic tessellation of printed media, making the metric inherently more relevant to human reading and document understanding tasks, thus reducing the performance interpretation gap relative to traditional metrics.

9. Develop a versatile, community-supported toolkit: By releasing an open-source Python library (cotescore), practitioners can integrate this robust framework into their existing DLA pipelines immediately, lowering the barrier to entry for advanced evaluation without requiring complete retraining of models.

10. Facilitate cross-dataset generalization: The SSU concept provides a relational structure that is less dependent on exact label consistency across datasets, allowing for better comparison and transfer of knowledge between documents with different annotation schemas.

Document Layout Analysis (DLA) is the process by which a page is parsed into meaningful elements, often using machine learning models. Typically, the quality of a model is judged using general machine vision metrics such as IoU, F1 or mAP. However, these metrics are designed for images that are 2D projections of 3D space, not for the natively 2D imagery of printed media. This discrepancy can result in misleading or uninformative interpretation of model performance. To encourage more robust, comparable, and nuanced DLA, we introduce: The Structural Semantic Unit (SSU), a relational labelling approach that shifts the focus from the physical to the semantic structure of the content; and the Coverage, Overlap, Trespass, and Excess (COTe) score, a decomposable metric for measuring page parsing quality. We demonstrate the value of these methods through case studies and by evaluating 5 common DLA models on 3 DLA datasets. We show that the COTe score is more informative than traditional metrics and reveals distinct failure modes across models, such as breaching semantic boundaries or repeatedly parsing the same region. We find that, under granularity differences between model and ground truth, the COTe score is substantially more robust than the F1. Even in the worst case, comparing character-level predictions against paragraph-level ground truth with otherwise perfect parsing, COTe returns 0.68 where F1 returns 0. Notably, we find that, on real datasets, the COTe's granularity robustness largely holds even without explicit SSU labelling, reducing the barrier to entry. Finally, we release an SSU labelled dataset and a Python library for applying COTe in DLA projects.