Data readiness for AI: test whether history supports the decision

A dataset can be complete, consistent, and still unsuitable for an AI task. It may record the outcome after the decision the model is meant to support. Its labels may reflect a process that has since changed. Or it may describe only cases selected by the existing workflow, leaving the intended deployment population largely unseen.

A meaningful data readiness assessment evaluates data against a specific decision. Generic quality scores are useful for locating defects, but readiness depends on timing, meaning, coverage, permission, and the ability to reproduce inputs in production. More historical rows cannot compensate for a target that answers the wrong question.

Consider a hypothetical service business planning to predict which support requests need specialist attention. It has years of tickets, final resolution notes, and escalation labels. Those records look rich. Yet many fields were added after specialist involvement, and the historical escalation policy varied between teams. The dataset requires careful interpretation before it can support an early routing model.

Define the decision and when it occurs

State when the prediction will be made and what action it can influence. Routing at ticket creation differs from routing after an initial conversation. The latter has more information available but may lose some opportunity to reduce delay. Choose the timing deliberately rather than building a model first and finding a place to insert it afterward.

Identify the intended outcome. Historical escalation is not necessarily equivalent to specialist need. A ticket may have escalated because the initial team lacked capacity, while another genuinely difficult ticket remained unresolved without escalation. If the model learns every historical routing choice as correct, it can reproduce process limitations rather than improve them.

Define what the system is allowed to recommend. The service business might use a score to prioritize review rather than move tickets automatically. That boundary affects evaluation and data requirements. A model proposing candidates for human review can tolerate different uncertainty from one that silently changes ownership and customer expectations.

Set a baseline for the existing decision. Measure current routing behavior, delays, and correction effort with enough context to understand the population. The model should be assessed against that practical alternative. A high classification score does not establish improvement if the proposed workflow creates more specialist work without helping customers reach resolution.

Reconstruct inputs available at prediction time

Classify fields by when they become available. Final resolution notes and specialist responses clearly occur after initial routing, but subtler leakage can enter through status updates, derived durations, or retrospective category corrections. A dataset exported today may contain values that did not exist when the historical decision was made.

Use event timestamps and revision history where available. If history is missing, document which inputs cannot be reliably reconstructed. Excluding those fields may produce a less impressive offline score, but it creates a more credible evaluation. Keeping them because they improve accuracy tests a capability the live system will not possess.

Include preprocessing in the timing boundary. Learned normalization, feature selection, and imputation should be fitted using the training population rather than the entire dataset. This keeps information from the evaluation population out of learned preprocessing choices and preserves a clearer test of behavior on previously unseen cases. The business assessment still needs to establish whether each input is available and meaningful at the intended decision point.

Test the actual production input path. Historical exports may contain clean, joined records while live requests arrive with missing identifiers or delayed enrichment. Compare these representations directly. A readiness finding should distinguish an available historical feature from an input the application can deliver reliably within the required response time.

Audit labels before treating them as truth

Review a sample of labels with domain specialists using an explicit rubric. Include disagreements, ambiguous tickets, and cases handled by different teams or periods. The purpose is to understand what the label captures and whether specialists can consistently interpret it, not to assume that a stored category has authoritative meaning.

Separate outcome labels from process labels. Escalation records describe an action taken; resolution quality describes a result. Either may be useful, but their relationship must be established rather than assumed. A prediction of historical escalation should not be presented as a prediction of customer benefit without further evidence.

Preserve uncertainty rather than forcing every case into a clean category. Some tickets legitimately require contextual judgment. Record the adjudication rule and allow unresolved labels where necessary. A smaller defensible evaluation set can be more informative than a larger set whose apparent certainty was created by arbitrary relabeling.

Examine coverage and selection effects

Historical data reflects the workflow that produced it. Tickets reaching specialists may have richer notes and better outcome records than tickets closed by the initial team. Training on only those well-documented cases changes the population. The model may then appear strong while lacking evidence about the requests it will actually route.

Compare coverage across relevant groups, channels, products, and periods. These dimensions should follow the task rather than a generic demographic checklist. A newly introduced product or support channel may require explicit limits on use because it has little comparable history. Report that limitation instead of hiding it inside an overall evaluation result.

Check duplicates and linked records before splitting datasets. Several tickets from one incident or customer can contain nearly identical content. Randomly spreading them across training and evaluation can overstate generalization. Choose a split that reflects the intended use and prevents closely related records from supplying unfair clues about the evaluation cases.

Time also matters. A historical policy change can alter labels and workload characteristics. Evaluate whether older records remain relevant and whether the model must generalize to a later operating period. The reasoning in forecast evaluation applies here: the test must respect the information and environment of the future decision, even when the task is not forecasting.

Establish appropriate access and data handling

Determine which information is necessary for routing and which merely happens to be present. Support conversations can include personal, confidential, or unrelated material. Minimizing inputs can reduce exposure while simplifying the production integration. Richer context should earn its place through a task-specific reason rather than being collected by default.

Review the organization’s permissions and data-use requirements with the responsible owners. Access to a historical export does not automatically establish permission to use it for every model or send it to another processing environment. Document approved uses, restrictions, retention, and the systems allowed to handle the information.

Preserve access boundaries in the deployed workflow. A model score or generated explanation can reveal information drawn from restricted sources even when the original record is hidden. Evaluate output behavior as part of the data-handling design. Security is not complete when the training files have been placed behind a permission check.

Govern the use of AI throughout its lifecycle by recording the approved purpose, evaluating changes, and maintaining clear authority to narrow or suspend the deployed task. For this project, concrete readiness decisions include the permitted inputs, responsible reviewers, and acceptable routing authority. Documenting those decisions should make the operating boundaries inspectable, without implying that a dataset or application has received a certification.

Prioritize defects by their effect on the decision

Not every quality issue deserves equal investment. Inconsistent formatting in an unused field may have little impact, while an unreliable arrival timestamp can undermine the entire evaluation. Rank defects by whether they change the target, leak future information, remove a consequential population, or prevent production input delivery.

Distinguish remediation options. The team might repair a source process, exclude an input, collect new labels, narrow the initial deployment population, or defer the task. A readiness assessment should present these choices with their consequences. Declaring the data bad without a feasible next step is no more useful than declaring it ready because the files load successfully.

Estimate the ongoing cost of the chosen fix. Manual labeling can improve an evaluation set but may not scale as a permanent production requirement. A new enrichment dependency may add latency and failure modes. The data plan should account for how the organization will maintain the input after the initial project team leaves.

Build a small, inspectable evaluation set before expanding model complexity. Use it to compare the baseline and candidate approach under the intended workflow. The pilot-to-production decision becomes clearer when the team can identify specific data limits and the operating arrangement needed to handle them.

Deliver a readiness decision with explicit boundaries

The final assessment should state which task and population the data can support, which findings remain unresolved, and what evidence is needed next. Preserve the target definition, input timing rules, labeling rubric, split method, and approved handling arrangements. These records make the decision reviewable as the model and business evolve.

For the support-routing example, a reasonable conclusion might be that historical escalation alone is insufficient as a target, while a reviewed specialist-need sample can support a bounded trial. That is a useful outcome even if it postpones deployment. It prevents the organization from interpreting an attractive offline score as proof of a better customer workflow.

Continue readiness checks when inputs or policies change. New product categories, revised routing rules, and different data collection can invalidate assumptions that previously held. The assessment is not a one-time badge for a dataset; it is a maintained account of why particular data remains suitable for a particular use.

The strongest foundation for AI is often a clearer decision definition rather than a larger dataset. When the service business knows what specialist need means, what information exists at routing time, and how uncertain cases will be handled, model development becomes a testable engineering choice instead of an exercise in extracting a flattering score from history.