Grading the Autonomy Claim: The Questions to Ask a Vendor Whose AI Acts Without You
A buyer-side interrogation instrument for maintenance AI. The questions, the evidence a good answer produces, and the red flag a bad answer leaves. Plus what the market actually publishes, and EQUA's answers to its own questionnaire.
Working on a live operating problem? Book Your 20-Minute Assessment
An autonomy claim is testable, and almost nobody tests it. Ask the vendor to name every action the system takes without a human, name the system it writes into, and produce one time-stamped record of it happening. IBM Research announced AssetOpsBench, an open-source industrial benchmark, on 15 July 2025, and reported that the best-performing model in that write-up, GPT-4, completed 65% of the benchmark’s 141 asset-operations problems under the agents-as-tools approach. Autonomy is a configuration, not a property.
Disclosure, up front. EQUA AI builds a product that would be subject to every question below. So we answer the whole instrument ourselves, in the same terms, further down this page. Where a recommended answer matches EQUA’s own architecture, it is marked as the publisher’s position.
Key takeaways
- “Autonomous” is not one thing. The useful split is between an action the system takes and logs, and an action it drafts for a person to release. Every vendor answer should land on one of those two, for every action type.
- The write list is the answer. A system’s real autonomy is the list of systems it can write to and the fields it can change in each. Ask for that list as a document, not as a sentence.
- Publisher’s position: for a maintenance coordination product, the correct answer on control-system writes is no. No setpoint, no restart, no interlock, no actuator command, at any permission level. This is EQUA’s own architecture, and it is the direction NIST’s public OT security guidance points.
- Approval gates and spend limits have to be enforced in code, not in policy. Of the 770 utility respondents to AWWA’s 2026 State of the Water Industry question on overtime policies and procedures, 32.7% reported no formal policies or procedures and 13.0% reported comprehensive overtime management. That question is about overtime specifically, not about delegation of authority in general. A gate that inherits an unwritten policy is not a gate.
- What the market publishes is thinner than what it says. UpKeep’s Supernova page describes SCADA and historian connectors and an approval inbox, and is labelled early access. Samsara’s June 2026 maintenance example is a warranty claim drafted when a work order closes, not dispatched field work.
- Ask for one full trace including the exceptions. A vendor who will show a clean trace and not a messy one is showing a demo, not a system.
Why does the word autonomous need an interrogation at all?
Because it has no fixed referent. In current maintenance software marketing it covers at least four things: a model that reads sensor data and raises a flag; a model that writes a draft a person releases; a model that writes to a business system under rules and logs what it did; and a model that dispatches work with no person in the sequence. Those are not degrees of one thing. They have different failure modes, different audit requirements, and different answers to who is responsible when the work was wrong.
The benchmark evidence supports the caution. AssetOpsBench, announced by IBM Research on 15 July 2025, gave agents 141 problems that required calling tools and coordinating with one another. The best result reported in that write-up was 65% task completion by GPT-4. Meta’s Llama 4 Maverick reached 59%, Meta’s older Llama 3.3 70B reached 40%, and IBM’s own Granite 3.3 8B, the smallest model evaluated, reached 35%. Any vendor whose answer to “how often is it wrong” is “it isn’t” is claiming something the field’s own published benchmark does not support.
What follows is an instrument. Send it before the demo, ask for written answers, and hold the demo against them.
What is autonomous, and what is only drafted?
| Question | Evidence a good answer produces | Red flag a bad answer leaves |
|---|---|---|
| List every action type the system can take. For each, is it taken and logged, or drafted for release? | A table of action types with a mode against each, matching what the software enforces. | A single adjective applied to the whole product. “The agent handles work orders end to end.” |
| Which of those actions are on by default in a new tenant? | A stated default posture, usually draft-only, and the steps required to change it. | “It depends on your configuration”, with no default named. |
| Show the last 20 actions of one type from a real tenant, redacted. | 20 records with timestamps, actor, inputs, outputs and outcome, including rejected ones. | A single curated example, or a screenshot. |
A product whose default is autonomous action has told you where its designers put the burden of proof.
What is the system authorised to write, and to which systems?
| Question | Evidence a good answer produces | Red flag a bad answer leaves |
|---|---|---|
| Name every system the product writes to, and every field it can change in each. | A written integration matrix: system, object, fields, direction, permission required. | “It integrates with your CMMS.” Direction unstated. |
| Can it write to a PLC, DCS or SCADA system: a setpoint, a restart, an interlock or an actuator command, at any permission level? | An unqualified no, with the architectural reason, and confirmation that no permission level exposes it. | “Not currently.” “Only with approval.” “That is on the roadmap.” Each of these means the write path exists or is planned. |
| Which system remains the system of record for work? | The customer’s CMMS or EAM, named, with the vendor’s product described as writing into it. | A product that becomes the record, migrating your work history into the vendor’s schema. |
The control-system question is the one to refuse to move past. Maintenance coordination and process control are separate jurisdictions with separate consequences. NIST SP 800-82r3, the United States government’s public guide to operational technology security, published September 2023, lists restricting logical access to the OT network, network activity and systems among the major security objectives for an OT implementation, and names unidirectional gateways among the ways it may be done. That is the opposite direction of travel from a coordination product acquiring write access to a controller.
Publisher’s position: a vendor offering control writes as a premium capability has misread which risk you are buying software to reduce. The customer’s CMMS or EAM stays the system of record. The coordination product does not become it.
How is autonomy configured, and who can change it?
| Question | Evidence a good answer produces | Red flag a bad answer leaves |
|---|---|---|
| Where do autonomy settings live, and what is the unit of configuration: the tenant, the role, the asset class, the action type? | A configuration surface the customer administers, with granularity named. | Autonomy set by the vendor during onboarding, changed by a support ticket. |
| Who can raise an autonomy level, and does that change itself produce a record? | Role-restricted change, logged with actor and timestamp, visible to the customer. | Any administrator can raise it silently. |
| How are spend limits enforced: in code, or in the runbook? | A hard limit inside the action path, tested by attempting to exceed it during the trial. | A documented policy and a promise. |
| What happens when a required approver is unavailable? | A defined behaviour: wait, escalate to a named alternate, expire the request, or fall back to draft. Never silently proceed. | “It escalates.” No definition of to whom, or after how long. |
Test the spend limit rather than reading about it. Set a limit of one hundred dollars during the trial, then attempt an action that costs more. A gate in code fails closed and records the attempt. A gate in a policy document produces a purchase order.
There is a sector reason to be exacting. AWWA’s 2026 State of the Water Industry, fielded 21 September to 31 October 2025 and published 30 April 2026, added a module on overtime. Of the 770 utility respondents to its question on overtime policies and procedures, 46.5% reported basic approval procedures, 32.7% reported no formal policies or procedures, and 13.0% reported comprehensive overtime management intended to monitor, manage and reduce overtime cost. These are AWWA’s own published findings, not a hidden result, and they are narrower than they look: the question asks about overtime policy at that utility, not about spend authority in general, and the base of 770 is a screened subgroup rather than a low response rate, drawn from the 1,273 utility respondents the report counts among its 2,171 participants after respondents skipped questions or were not shown them. Read for exactly what it covers, it still earns one question of your own. If a vendor’s approval architecture is “we enforce your policy”, the written policy it inherits may not exist.
What happens when the system is wrong, and how would you know?
| Question | Evidence a good answer produces | Red flag a bad answer leaves |
|---|---|---|
| What is the false-positive behaviour: what happens when it raises something that is not real? | A described path: the finding is closed with a reason, the reason is recorded, and the record is available for review. | “The model is very accurate.” No behaviour described. |
| What is the false-negative behaviour: how do you learn about the thing it did not raise? | An explicit answer, usually that missed events are found by the existing process and fed back deliberately. Absence is hard to detect and the vendor should say so. | A claim that misses are caught automatically. |
| How is a wrong action reversed, and is the reversal itself recorded? | A rollback path per action type, with the original action and the reversal both retained. | Delete and re-enter. |
Ask the false-negative question twice. Most vendors have not thought about it, and the honest answer, that misses surface through the process that would have caught them anyway, is the one that tells you a vendor is being straight.
What record does each action leave, and who can read it?
| Question | Evidence a good answer produces | Red flag a bad answer leaves |
|---|---|---|
| Does every permitted action produce a time-stamped record? Which actions do not? | Yes, with a named exception list if one exists. | “Everything is logged”, with no schema. |
| What is in the record: actor, inputs, model version, decision, outputs, outcome? | A schema you can read, exported from a real tenant. | A user-facing activity feed presented as an audit trail. |
| Who can read it: the customer, without asking the vendor? | Direct customer access, plus export. | Logs available on request through support. |
| Can records be edited or deleted, and by whom? | Append-only, or edits recorded as new entries. | Administrative delete with no trace. |
What happens when an input is missing or an integration fails?
| Question | Evidence a good answer produces | Red flag a bad answer leaves |
|---|---|---|
| A required input is missing: what does the system do? | It stops, records what was missing, and asks. Degradation is explicit. | It infers the missing value and proceeds. |
| An integration is down: what happens to actions queued against it? | A queue with a defined depth and expiry, alerting on both, and no silent replay of stale actions on recovery. | Retry until it works, with the definition of “works” unspecified. |
| What is required on day one, at minimum, for the system to be useful? | A specific list: asset register, work history, user roles, and whatever read-only sources are in scope. | “We work with whatever you have.” |
| What does the system do with bad data: duplicate assets, blank fields, free text where a code belongs? | Data quality surfaced as a visible finding, not silently corrected, with any correction attributable to a person. | Silent normalisation, which is how a bad asset register becomes a confident wrong answer. |
| Will you show a full trace of one real workflow, including the exceptions? | A complete trace with the failures in it. | A trace with the failures removed, or a refusal. |
What happens if you leave?
| Question | Evidence a good answer produces | Red flag a bad answer leaves |
|---|---|---|
| Can the customer export everything the system holds, in a documented format, without vendor assistance? | Self-service export, tested during the trial, format documented. | Export as a professional-services engagement. |
| Does anything of operational value exist only inside the vendor’s product? | Nothing. The CMMS holds the work record; the vendor holds the coordination and the receipts, both exportable. | Work history, asset hierarchy or approval history that lives only with the vendor. |
| What happens to the data after termination, and how is deletion evidenced? | A stated retention period, a deletion process, and a certificate or log entry. | Silence, or a clause permitting indefinite retention for product improvement. |
What does the current market actually publish?
Three observations, drawn only from vendors’ own public material, the only thing a buyer can hold anybody to. Then one note on method.
UpKeep’s Supernova product page describes pre-built connectors for CMMS, ERP, SCADA and data historians, and an approval mechanism it words as “Agents execute autonomously. When something needs your sign-off, it appears in your inbox. Full context, one-click approve or reject, complete audit trail.” That same page labels Supernova early access. What the public page does not enumerate is the configuration surface: which actions require sign-off by default, who can change that, and how the audit trail is structured. Reasonable questions to put to a product that is not yet generally available.
Samsara announced new agent capabilities on 24 June 2026, and its press release of that date describes an Agent Studio as the control centre where customers set up and manage those workflows. The maintenance example appears in the company’s own blog post of the same date, and it is a warranty claim: the post says Samsara generates manufacturer-specific compliance checklists and drafts claims the moment a work order closes. That is document generation downstream of a completed repair, not a fault code turned into dispatched field work, and it should not be read as one.
IBM has moved Maximo Application Suite toward agent workflows. Its July 2025 write-up describes a Maximo Assistant already integrated, a Condition Insights agent then in pilot at an IBM facility, and an optimizer the post said would be added that month, and IBM open sourced AssetOpsBench, describing it as a framework for developers to build, evaluate and improve their own asset-management agents on the same 141 problems. Publishing a benchmark on which your own 8-billion-parameter model completes 35% of the tasks is a different posture from publishing a claim.
The fourth item is a method note rather than an observation about a vendor. Marketing surfaces and product documentation can disagree, and this piece deliberately makes no claim that any particular vendor’s pages do, because that assertion would require citing both pages precisely and a misattribution would be worse than the omission. If you find such a contradiction yourself, do not decide which page is true. Attach both to the contract as exhibits and require the vendor to reconcile them in writing. A vendor that will has told you something real. A vendor that will not has told you something too.
The rungs an autonomy claim has to be specific about
Scroll sideways to see the whole drawing.
Figure 1. The rungs an autonomy claim has to be specific about. A ladder of action classes a maintenance AI vendor should be asked to place its product against. Reading records and telemetry is held by the system. Drafting a work order for review is held by the system with the decision held by a planner. Reserving or ordering a part is held by the customer through a configured spend limit. Closing a work order is held by whoever the customer names, and the question to ask is whether that can be set to nobody. Issuing a command to a control system sits outside the ladder and is marked as never held. A note states that a vendor who cannot say which rung each of its agents sits on has not built the distinction into the product.
EQUA’s answers to its own questionnaire
We would be subject to every question above, so here are our answers, in the same terms. Hold us to them.
What is autonomous and what is drafted. Customer-defined. EQUA AIMMS ships no fixed autonomy level. Autonomy, roles, permissions, spend limits and approval gates are configured by the customer, per action type, and can be set to draft-only across the board.
What it writes to. Customer business systems: CMMS, EAM and ERP, inventory and procurement, under the permissions and approval rules the customer sets. The CMMS stays the system of record. AIMMS writes into it and does not replace it.
Control-system writes. No. AIMMS has no write path to a PLC, DCS or SCADA system: no setpoint, no restart, no interlock, no actuator command, at any permission level. Not a permission switched off. An absent path. Detection is a read. Control is a write. The boundary is on the write.
What detection means. AIMMS connects read-only to SCADA, historians and IoT sensor sources. It detects anomalies and deviations, raises the fault itself, alerts the people responsible, and surfaces slower drifts as recommended scheduled maintenance. Vibration analysis, infrared thermography, oil and wear debris analysis and motor current signature analysis are a separate discipline with their own instrumentation and specialists, and AIMMS does not stand in for them.
The record. Every permitted action produces a time-stamped receipt.
False positives and false negatives. A finding that turns out not to be real is closed with a recorded reason. Missed events surface through the process that would have caught them anyway. We have published no measured accuracy rate on any population, and will not state one until it carries its period, population and denominator.
Deployment history. No named customer, no published case study, no product screenshots. What we put in front of a buyer is the architecture, the boundary and the record the system keeps.
What is not on the site. Retention periods, export formats and the audit record schema are contract material. Ask us for them in writing, the way this guide tells you to ask everybody.
Who wrote this
EQUA AI builds EQUA AIMMS. You set what it may do: the roles, the permissions, the spend limits, the approval gates, and the points where it has to stop and ask.
Every action it takes leaves a time-stamped receipt naming what it read, what it did and who authorised it. That is what turns an autonomy claim into something a buyer can check on a Tuesday rather than something they have to take on trust at signature.
Sources
- IBM Research, “Meet AssetOpsBench, IBM’s first industry 4.0 benchmark”, 15 July 2025. https://research.ibm.com/blog/asset-ops-benchmark
- IBM Research, “IBM is adding AI agents to Maximo Application Suite”, 2 July 2025. https://research.ibm.com/blog/Maximo-automating-asset-management
- IBM, AssetOpsBench source repository. https://github.com/IBM/AssetOpsBench
- Patel, Lin, Rayfield and others, “AssetOpsBench: Benchmarking AI Agents for Task Automation in Industrial Asset Operations and Maintenance”, arXiv:2506.03828, submitted 4 June 2025. https://arxiv.org/abs/2506.03828
- UpKeep, Supernova product page, accessed 7 September 2026. https://upkeep.com/product/supernova/
- Samsara, press release on new agent capabilities for operational tasks, 24 June 2026. https://www.samsara.com/company/news/press-releases/samsara-launches-new-agentic-capabilities-to-automate-tedious-operational-tasks
- Samsara, “Samsara Beyond 2026: new AI for physical operations”, company blog, 24 June 2026, the source of the warranty claim description. https://www.samsara.com/blog/announcing-new-innovations-to-reduce-risk-increase-efficiency-and-digitally-transform-operations
- American Water Works Association, “2026 State of the Water Industry Report”, survey fielded 21 September to 31 October 2025, published 30 April 2026, total n=2,171. Figure 16 on overtime policies and procedures, n=770. https://www.awwa.org/state-of-the-water-industry/
- National Institute of Standards and Technology, Special Publication 800-82 Revision 3, “Guide to Operational Technology (OT) Security”, published September 2023. https://csrc.nist.gov/pubs/sp/800/82/r3/final