Designing AI Pipelines for Decision-Ready ITSM Intelligence

This paper reframes ITSM data use as an IS problem of transformation, abstraction, and decision support. The implemented AI pipeline converts heterogeneous ticket exports into a multilevel, decision-ready report through schema normalization, HDBSCAN-based sub-topic clustering, and HAC-based main-topic abstraction. Preliminary evaluation across six artifacts and five stakeholders shows that all four decision support metrics exceed 4.0, with Trust emerging as the most consistent signal (mean = 4.33, std = 0.61), providing early evidence that the artifact is perceived as credible and adoption-worthy. In doing so, this paper transforms raw ITSM tickets into high-fidelity insights for sales and executive decision making.

Improvements to AI Systems:

1. Hierarchical Topic Abstraction for Noisy, Heterogeneous Text Data

• Implement a two-stage clustering pipeline (density-based sub-topic detection → agglomerative main-topic merging) to handle unstructured, multi-source ticket data without fixed taxonomies.

• The improved AI system can automatically generate multi-level topic hierarchies (e.g., printer issue → hardware failure → customer infrastructure problem) from raw logs, reducing manual labeling effort.

2. Schema-Normalization Layer for Cross-System Data Integration

• Add a preprocessing module that maps heterogeneous ticket fields (e.g., different date formats, priority scales, custom fields) into a unified schema before analysis.

• The improved AI system can ingest data from multiple ITSM tools (ServiceNow, Jira, Zendesk) and produce consistent, comparable outputs without custom per-source code.

3. Trust-Calibrated Decision Support Metrics

• Integrate a confidence-scoring mechanism that outputs not only recommendations but also a trust score (mean 4.33, std 0.61 in the paper) based on cluster stability, data completeness, and stakeholder feedback loops.

• The improved AI system can flag low-confidence insights (e.g., sparse clusters, ambiguous sub-topics) and explain why, enabling users to decide when to act vs. request more data.

4. Multilevel Report Generation for Role-Specific Consumption

• Build a report generator that produces both granular sub-topic details (for operational staff) and abstracted main-topic summaries (for executives), with drill-down/roll-up navigation.

• The improved AI system can automatically tailor the level of abstraction to the user's role, reducing cognitive load and improving decision speed (all four decision support metrics > 4.0).

5. Iterative Stakeholder Feedback Integration

• Add a feedback loop where users rate the usefulness of each insight (e.g., 1–5 scale), and the system re-weights clustering parameters or re-ranks topics based on that signal.

• The improved AI system can continuously adapt its abstraction granularity to match evolving business priorities, increasing adoption likelihood over time.

What the Improved AI System Can Do:

• Automatically transform raw, messy IT tickets from any source into a clean, hierarchical, decision-ready report within minutes.

• Provide executives with high-level trend summaries (e.g., top 3 recurring infrastructure issues this quarter) while allowing support teams to drill down to specific ticket clusters for root-cause analysis.

• Quantify its own reliability per insight, so users trust the output enough to act on it (as evidenced by the paper's trust score).

• Learn from user feedback to refine topic boundaries and report structure, making it more relevant with each use.

IT service management (ITSM) systems accumulate large volumes of heterogeneous ticket data that are difficult for sales and executive stakeholders to convert into actionable intelligence. This paper presents a sociotechnical AI pipeline, designed and evaluated following design science research principles, that transforms raw ITSM exports into a multilevel decision-support artifact. The pipeline combines LLM-based schema normalization, HDBSCAN sub-topic clustering, and hierarchical agglomerative clustering to generate executive-facing Main-topics and granular Sub-topics. A stakeholder evaluation across six artifacts and five raters from Sales Engineering and customer success roles shows that all four decision-support metrics, interpretability, actionability, trust, and likelihood of use, on average exceed 4.0 out of 5.0, with trust as the most consistent signal. The findings position ITSM analytics as an Information Systems (IS) problem of transformation, abstraction, and human-centered design.