What Your Work Order History Can and Cannot Do
Coordinating the next repair and training a model on the last five years are two different projects with two different data requirements. Here is how to tell which one your records can carry.
Working on a live operating problem? Book Your 20-Minute Assessment
Your maintenance history is usually good enough to coordinate the next repair and usually not good enough to train a reliable model on the last five years. Two projects, two sets of data requirements, and they get confused because both arrive under the same label: “AI on your maintenance data”. In a 2024 study in IET Renewable Power Generation, two organisations labelling the same records produced subsystem failure rates ranging from 0.71 to 3.5 times one another.
Key takeaways
- The objection is correct on the facts. NIST-led work in Manufacturing Letters describes typical work order health data as “messy, unstructured raw text” plus “inconsistent, incomplete, incorrect, or missing codes and data formats”, and names the consequence: a lack of verification, validation “and ultimately, personnel trust”.
- The objection answers the wrong question. The minimum viable input is set per workflow. Coordinating one repair forward needs a named asset, a reachable person, a parts record and a clock. Training a model on your history needs consistent labels across years of records that nobody wrote for that purpose.
- The human gold standard is not stable either. Walgern and colleagues (2024) found that a single wind fleet dataset, labelled to the same published designation guideline by two organisations that discussed ambiguous cases regularly, produced failure rates differing by a factor of 3.5 for one subsystem.
- Model size is not the variable that moved the result. Stewart, Hodkiewicz and Li measured GPT-3.5 at F1 0.46 out of the box on failure mode classification, below a conventional text classifier at 0.60, and the same model reached 0.80 once fine-tuned on domain-annotated data.
- The asset hierarchy is the binding constraint, more than the free text. A functional location list that collapses a whole treatment train into one entry caps every analysis built above it, and no amount of language modelling recovers what was never distinguished.
- Ten questions to your CMMS administrator settle which project you can start. Most are answerable from a database query in an afternoon.
Is the objection right?
Yes. It deserves a straight concession rather than a way around it.
The clearest statement of the problem comes from a NIST-led group writing in Manufacturing Letters. Maintenance work orders, they write, are “the clinical notes” of an asset management system, and “typical MWO health data contain (a) messy, unstructured raw text; and (b) inconsistent, incomplete, incorrect, or missing codes and data formats”. The same paper names the thing that matters most to anyone who has been burned before: pipelines designed with non-technical language in mind, then dropped onto technical text, suffer “from a lack of verification, validation, and ultimately, personnel trust”.
That last clause is the real risk. A model that is wrong in public, in front of a crew that already knew the answer, does not merely fail. It spends credibility you will need later.
A case study from the same NIST group, on historical HVAC work orders, makes the point at the level of arithmetic: when data quality is low, analysis accuracy is reduced “often in hidden ways”. The authors go further and estimate that the common human errors sitting in free-text fields are highly non-random, and therefore cause large errors in KPI calculation “even with large sample sizes”. Estimate is their word, and it is worth keeping. Volume does not rescue you. A hundred thousand work orders with an unreliable failure code field give you a hundred thousand unreliable failure codes.
So the sceptic in the room is right about the data. What follows from that depends entirely on what you intend to do with it.
Which project are you actually being sold?
Two things get called “AI on your maintenance data”. They have almost nothing in common.
| Coordinate the next repair | Train a model on your history | |
|---|---|---|
| Question it answers | What has to happen now, who does it, what is blocking it | What tends to fail, how often, and what it costs |
| Direction in time | Forward from this moment | Backward across years of records |
| Unit of work | One fault, one job, one asset | A population of assets and events |
| Minimum viable input | A named asset, a person who can act, a parts and stores record, a clock that starts | Consistent labels, a stable hierarchy, coded failure modes, and enough events per class |
| What poor history costs you | Some context you would have liked | The entire result |
| Effect on your data | Produces structured records going forward | Consumes the records you already have |
| How you know it worked | The job closed, with evidence, in less elapsed time | Out-of-sample performance against a held-out label set you trust |
The row that settles most arguments is the fourth. Coordination is a workflow problem. It needs to know which pump, who is on shift, whether the seal kit is on the shelf, whether the permit is signed and when the clock started. Almost all of that either sits in your systems today or gets created by the act of doing the work. None of it requires that the record of a 2019 repair was written well.
Training on history is an inference problem. It requires that thousands of past records were labelled consistently by people who had no idea a model would read them. Much stronger requirement. It is the one your history usually fails.
Watch for two moves in an evaluation: a demonstration of the second offered as evidence for the first, and a demand for the data conditions of the second before the first will be delivered. Both are easy to make, and both fall apart when you ask which of the two columns above the demonstration actually sat in.
How stable is the human gold standard?
Walgern and colleagues, writing in IET Renewable Power Generation in 2024, studied wind turbine maintenance reports and their conversion into reliability KPIs. In one part of that work they took a single dataset from one operator and had it categorised by two distinct organisations, both using the RDS-PP reference designation guideline for labelling, with the ZEUS state-event-cause code guiding how a component state or maintenance action was recorded. Both organisations discussed ambiguous categorisations with each other regularly. The authors then computed failure rates from each set of labels and expressed one as a multiple of the other.
The multiples ran from 0.71 for the generator switching system category to a 3.5-fold higher failure rate for the fire extinguishing system category. The drive train system and the lifting gears came out 1.79 and 1.77 times higher. That is a spread of roughly five to one across subsystems, produced by two expert organisations working from identical records under a shared designation guideline.
Two cautions on that number. It is wind fleet data from one operator, not municipal water and not an industrial plant, and the specific ratios do not transfer. And it is a ratio between two labellings of the same records, not an error rate against a known truth. The defensible statement is the ratio itself.
What does transfer is the structural finding. The authors attribute the divergence to three things: guideline categories broad enough that a component can defensibly sit in more than one, ambiguity about whether a given intervention counts as a failure at all when the report lacks detail, and human factors including whether the person labelling specialised in electrical or mechanical components. The paper is blunt about what this means for the literature, naming “the challenges inherent to comparing failure rates across different publications, even when they employ the same taxonomies or failure definitions”.
Now take that into a procurement meeting. Someone says: the model produced a false positive, so it cannot be trusted. That is a fair test only if the thing it was measured against is itself stable. On failure classification from maintenance text, this study shows it need not be. Two expert organisations disagreed by a factor of 3.5 on one subsystem.
None of which defends a bad model. It is an argument for reading an accuracy score carefully. Comparing a model to “the right answer” assumes a right answer exists in the record. Where the record does not settle the question, a single accuracy number scored against one organisation’s labels is measuring agreement with that organisation, not correctness. Two questions survive that problem, and an evaluation can answer both: can an output be traced back to the source record a person can open and check, and what does a wrong output cost in the specific place it is used. A wrong ranking in a five-year reliability study can misdirect a capital plan for years before anyone notices. A wrong label on a single job is in front of the person doing that job.
What do the benchmarks say about language models on maintenance text?
Two findings are worth carrying into a vendor conversation.
Stewart, Hodkiewicz and Li released a preprint investigating large language models for failure mode classification in September 2023. Out of the box, GPT-3.5 scored F1 0.46 on labelling an observation with a failure mode code from a restricted code list. A conventional text classification model trained on the annotated dataset scored 0.60. The same GPT-3.5, fine-tuned on that annotated data, scored 0.80. Their own conclusion is that the work “reinforces the need for high quality fine-tuning data sets for domain-specific tasks”.
Read that ordering carefully. The large general model, unprompted by the domain, lost to a conventional classifier. What moved the result was domain-grounded annotated data, not model size. Releases since 2023 have improved on general benchmarks and the absolute numbers will have moved, so test the ordering on your own records rather than assume it in either direction.
The Walgern study supports the same reading from the other side. Classifiers were assessed on data from wind farms that were not used in training but still belonged to the original operator, and performance dropped. The authors note that the classifier predicting labels up to the second level of the RDS-PP designation guideline had only half of its predictions correct for the entire dataset. Its fallback label, used for descriptions that could not be sorted into any more specific category, produced failure rates seven to nine times higher than the manually labelled data. The model did not fail loudly. It routed the hard cases into a bucket, and the bucket then distorted the statistics.
Against that, the field is starting to build evaluation openly rather than by demonstration. A team led by IBM Research released AssetOpsBench as a preprint in June 2025, a framework and benchmark for evaluating agents on industrial asset operations and maintenance tasks, built around a curated set of more than 140 human-authored natural language queries grounded in real industrial scenarios, with the code public on GitHub. Its existence is the point. If a vendor says their system performs well on maintenance tasks, ask against what, and expect a benchmark rather than a screenshot.
Why does the asset hierarchy cap everything above it?
The free text gets the attention. The asset register decides the ceiling.
Reliability data practice rests on a hierarchical equipment register: broad levels for the site and the process, narrowing until you reach the individual item that work is actually performed on. In a CMMS this is the functional location hierarchy. Everything downstream turns on how far down it goes, and whether a work order can be written against the bottom of it.
There is a published international standard for a register of this kind, and it is worth knowing both what it is and what it is not. ISO 14224:2016, third edition, is titled “Petroleum, petrochemical and natural gas industries: Collection and exchange of reliability and maintenance data for equipment”, and its stated scope is those industries, not water and not the grid. It is also a paid document. This article therefore names it and stops: nothing here reproduces or paraphrases its contents, and a utility deciding whether to adopt it should read the standard rather than a summary of it. For water systems specifically, the EPA guidance cited below is free, sector-specific and a reasonable starting point.
The failure mode in the field is simple and extremely common. The functional location list was built for accounting or for a SCADA screen rather than for maintenance analysis, so an entire treatment train appears as one entry. Every work order for every pump, valve, blower and instrument in that train lands on the same node. Nothing above that node can tell them apart, because the distinction was never recorded.
No language model recovers this. The information is not compressed or badly worded. It is absent. A model reading “replaced seal, unit 3” against a functional location that covers eleven pumps cannot tell you which pump, and neither can you. It will produce an answer anyway.
This is also where the sector data lands. Black and Veatch’s news release for its 2026 Water Report, a survey of more than 600 US water sector stakeholders, states that “seven in 10 (70%) say they collect sufficient data, but only 19% say they leverage it effectively”. The gap is rarely a shortage of sensors. It is that the collected data cannot be joined to a structure that means anything.
The EPA’s asset management guidance for water systems puts the current state of the assets first among its five core questions, and its first listed best practice under that question is preparing an asset inventory and system map. It is also realistic about how that inventory gets built. It tells systems to use estimates where the information is hard to find, and says that over time, as assets are rehabilitated, repaired or replaced, the inventory will become more accurate. The register improves through the work, not only through a separate exercise.
Two projects, sold under one word
Scroll sideways to see the whole drawing.
Figure 1. Two projects, sold under one word. Two columns comparing the data requirements of two different projects. The first column is coordinating the next repair, which needs an asset register good enough to name the thing that failed, a place to write the record back, and no history at all, and which fails when the hierarchy is too shallow to name the asset. The second column is training a model on maintenance history, which needs years of consistently coded records, stable failure codes, and labelled outcomes, and which fails when free text and coding conventions changed between crews. A shared band beneath both states that both are called data readiness in a sales conversation, and that only one of them depends on what already happened.
Ten questions to put to your CMMS administrator
These are answerable from queries against your own database. Ask for the numbers, not the impression.
| # | Ask your CMMS administrator | A weak answer sounds like | What a weak answer caps |
|---|---|---|---|
| 1 | What share of assets in the field exist as records in the CMMS, and when was that last reconciled against a physical walk-down? | “Everything important is in there.” | Any analysis by asset class. Work lands on the wrong record or on none. |
| 2 | How many levels deep does the functional location hierarchy go, and what is the lowest level a work order can be written against? | “We use one level, by facility or by process area.” | Everything. Failure rates, cost per asset, criticality ranking and parts forecasting all collapse together. |
| 3 | What percentage of closed corrective work orders in the last 24 months carry a populated failure code, and how many distinct codes are actually in use? | “The field exists but the crews skip it.” | Failure mode analysis, and any model that would learn from coded outcomes rather than free text. |
| 4 | Pull 30 random closed work orders. In how many can a person who was not there tell what failed, what was done and why? | “The techs know what they meant.” | Anything that reads the history: search, summarisation, recurrence detection, knowledge capture for retirements. |
| 5 | Are parts consumed recorded against the work order, or issued from stores as a separate transaction that is never linked back? | “Stores handles that in the other system.” | True repair cost, parts forecasting, and knowing whether the part on the shelf is the part on the asset. |
| 6 | Is there a downtime clock at all, and if so, what starts it and what stops it? | “We record when the work order was created and closed.” | Every duration metric. Note that time to repair and total downtime are different quantities and must not be swapped. |
| 7 | What is the split between planned and corrective work orders, and is the split measured in count, hours or cost? | “Mostly PM, I think.” | Your baseline. Without it, no change can be shown to be a change. |
| 8 | How many duplicate asset records exist for the same physical equipment, and how would you detect one? | “There is no way to check that.” | Every per-asset metric, silently. History splits across records and each half looks healthier than the whole. |
| 9 | How many work orders in the last 24 months are attached to no asset, or to an asset record that no longer exists? | “Some, probably. Nobody has looked.” | The denominator of every analysis. Orphans are often the emergency work you most want to study. |
| 10 | Who is accountable for CMMS data quality by name, how much of their week is it, and what do they report on? | “It is everyone’s responsibility.” | Any sustained improvement. Unowned data quality tends to revert to its prior state. |
How should you read the answers?
Question 2 is the gate. If the hierarchy has one usable level, a modelling project on your history is not available at any budget, and the honest sequence starts with structure.
Questions 1, 8 and 9 tell you whether your denominators are real. If you cannot say how many assets you have, how many are duplicated and how many work orders belong to nothing, every percentage quoted back to you is an estimate wearing a decimal point.
Questions 3 and 4 tell you whether the history is machine readable. Weak answers rule out learning from the past. They do not, on their own, rule out coordinating the present.
Questions 5, 6 and 7 tell you whether you can measure a change. If there is no clock, decide which clock you mean before any pilot starts and hold it fixed. Time to repair and total downtime are different quantities, and improving one does not entitle anyone to claim the other.
Question 10 predicts whether any of it holds. A name and a number of hours is recoverable. A shrug means the same conversation later.
Who wrote this
EQUA AI builds EQUA AIMMS, and it builds the left-hand column of that first table: coordination forward from a fault.
The records come out structured because the work produced them, not because somebody was asked to tidy up afterwards. The asset is named because a technician had to be sent to it. The part is recorded because it had to be found. The clock exists because the job had a start and an end. Nobody has to be persuaded to do any of it, which is why it is the cheapest clean data a maintenance organisation will ever get.
Where to go next
- AI CMMS, EAM, APM and Predictive Maintenance separates the software categories these two projects get sold under, and says where each one stops.
- Maintenance Systems Record Work, They Do Not Coordinate the Fix covers the coordination gap this article’s first table describes.
- How to Measure a Maintenance AI Pilot covers picking the clock and the denominator before results exist.
Sources
- Brundage, M.P., Sexton, T., Hodkiewicz, M., Dima, A., Lukens, S. Technical Language Processing: Unlocking Maintenance Knowledge. Manufacturing Letters, volume 27, pages 42 to 46, print issue January 2021, available online 11 December 2020. DOI 10.1016/j.mfglet.2020.11.001. Author manuscript available free from NIST: tsapps.nist.gov. Source of the quoted descriptions of work order data, of the personnel trust consequence, and of annotation being necessary but potentially “prohibitively cost and time expensive”.
- Conte, A., Bolland, C., Phan, L., Brundage, M., Sexton, T. The Impact of Data Quality on Maintenance Work Order Analysis: A Case Study in Historical HVAC Maintenance Work Orders. 6th European Conference of the Prognostics and Health Management Society, Turin, 28 June to 2 July 2021. NIST publication record: nist.gov. Source of the finding that low data quality reduces analysis accuracy “often in hidden ways” and that the authors estimate common human errors in text fields to be highly non-random and to cause large KPI errors even with large sample sizes.
- Walgern, J., Beckh, K., Hannes, N., Horn, M., Lutz, M.A., Fischer, K., Kolios, A. Impact of using text classifiers for standardising maintenance data of wind turbines on reliability calculations. IET Renewable Power Generation, 2024, volume 18, issue 15, pages 3463 to 3479. DOI 10.1049/rpg2.13151. Open access. Source of the two-organisation comparison (multiples from 0.71 for category MSC, generator switching system, to a 3.5-fold higher failure rate for category XGM, fire extinguishing system, with MDK at 1.79 and XMM at 1.77), of the up-to-level-2 classifier having half of its predictions correct for the entire dataset, and of the fallback category “G” producing failure rates seven to nine times higher than the manually labelled data. Population: maintenance reports from one wind farm operator’s fleet, labelled using RDS-PP and ZEUS.
- Stewart, M., Hodkiewicz, M., Li, S. Large Language Models for Failure Mode Classification: An Investigation. arXiv preprint 2309.08181v1, 15 September 2023. arxiv.org/abs/2309.08181. Source of F1 0.46 for out-of-the-box GPT-3.5, 0.60 for a conventional text classification model trained on the annotated data, and 0.80 for GPT-3.5 fine-tuned on the same data. Task: labelling an observation with a failure mode code from a restricted code list.
- Patel, D., Lin, S., Rayfield, J., Zhou, N., Shyalika, C., Yarrabothula, S.R., Vaculin, R., Martinez, N., O’Donncha, F., Kalagnanam, J. AssetOpsBench: Benchmarking AI Agents for Task Automation in Industrial Asset Operations and Maintenance. arXiv preprint 2506.03828, submitted 4 June 2025. arxiv.org/abs/2506.03828. Source of the “curated dataset of 140+ human-authored natural-language queries grounded in real industrial scenarios”. Code at github.com/IBM/AssetOpsBench.
- ISO. ISO 14224:2016, Petroleum, petrochemical and natural gas industries: Collection and exchange of reliability and maintenance data for equipment, third edition, published 15 September 2016. Cited by number, public title and public scope only. Paid standard; its contents are not reproduced or paraphrased in this article. iso.org/standard/64076.html.
- US Environmental Protection Agency, Office of Water. Asset Management: A Best Practices Guide, EPA 816-F-08-014, April 2008. Source of the five core questions framework, of “preparing an asset inventory and system map” as the first listed best practice under core question 1, and of the guidance to use estimates where information is hard to find because “over time, as assets are rehabilitated, repaired or replaced, your inventory will become more accurate”. epa.gov.
- Black and Veatch. 2026 Water Report. News release dated 9 June 2026; survey of more than 600 US water sector stakeholders. The report itself states: “While 59% have digital strategies and 70% collect enough data, only 19% leverage it effectively (FIG. 4).” The news release renders the same finding as “Seven in 10 (70%) say they collect sufficient data, but only 19% say they leverage it effectively”. bv.com news release.