6. Evaluation

6. Evaluation

The evaluation asks one question: can the specification be applied as written, and where does it fail? It does not ask whether the framework produces better systems than other methods; that has not been tested. Three checks were made: desk checks by the specification's author, an independent re-run, and a test of the optional knowledge graph.

6.1 Desk checks

The AI model that drafted the specification applied it to Harbourline end to end and to Citigroup as analysis only, filling every template. Both tests reached their intended endings: Harbourline stopped at G3 because it had no recorded runs, and Citigroup at G1 with every finding referred. The desk checks found eight failures in the specification, all fixed before sign-off.

The desk checks also showed what the framework found that the Harbourline case had left out: outcomes the case never named, the missing decision owner, four inscription actors folded into systems, amount as a persuasive field, loss modes, the rule gap on policy wording, and the fact that the network could not be certified.

6.2 Independent re-run

Because the desk checks were made by the model that drafted the specification, they were not independent. A fresh analyst, a separate model instance that had not seen the authoring conversation or the desk-check records, was given the agreed specification without its case records, the Harbourline case file, and the URL of the FCA notice. Ten passages that would have revealed the desk-check results were removed from its copy. It was asked to run both tests and to report every place the specification failed, was ambiguous, or forced an invented entry.

The re-run reached the same endings: Harbourline fails G3 on illustrative evidence, and Citigroup's right ending is analysis only. It disagreed on G1. It blocked G1 in both cases, because neither names a decision owner, a tolerance or finding owners; the desk checks had passed G1 with a hypothetical owner. The specification did not say which reading was right, and now does: a case study may pass on proposed owners, recorded as such.

It found 30 problems the desk checks had not. Six mattered most: a loss could not enter inside an actor; a qualifier that never reached the path could not be recorded as a loss; loss modes and dimensions had no definitions; "effect" was undefined when losses interact; the channel values fitted only MCP servers; and a decision could not have two decision points. It also showed that the S2 fix had reached only one of the three places that state the safety rule. The author, a person, agreed twelve fixes, which cover 22 of the 30 problems (Appendix I). Eight remain undecided, among them whether each tier needs a default tolerance, without which G3's "within tolerance" cannot be decided.

The re-run partly tests agreement between analysts. On Harbourline, for the eight losses both analysts found, they agreed on the dimension for 8, the mode for 7 and the hop for 6. Both hop differences came from the first problem above: the specification had no way to place a loss inside an actor. The second analyst also found a ninth loss: the approver's identity never reached the model. On Citigroup the analysts diverged more. The second network had 24 actors and 13 hops, against an extract of 13 actors and 7 hops in the desk check, and 17 findings against 8. A case file with written rules constrained the analysis; a regulator's narrative did not. This is one comparison, not a measurement of agreement.

The re-run also found errors in the Harbourline case file. Two illustrative traces state figures that their own tool calls did not return. The case scores two traces as passing an assertion their wording fails. And a disturbance is "caught" by re-running a fixture that the rule change does not affect. The case file, not the specification, needs those corrections.

6.3 The knowledge graph

The ontology was loaded into Kùzu 0.11.3, an embedded graph database that reads Cypher, with Harbourline's decision, path, rules, tools, losses, findings, controls and acceptance test. Five queries were run: what reopens when a rule changes, losses with no disposition (the design gate), closed losses with no guarding assertion (the release gate), hidden mediators on a path, and tools shared by more than one decision. Queries 1 to 3 were also run on cases built to make them fail. All returned what was expected, though the reopening query also returned one loss that does not depend on the changed rule, because it works tool by tool. That led to an optional relation from a control to the rule it implements. The queries have not been run on Neo4j, a second graph database.

6.4 What the evaluation does not show

The evaluation shows that the specification can be applied to these cases and records where it failed. It does not show that teams using the framework build systems that give more correct answers, because no comparison with other methods was made. The Harbourline runs are illustrative, so the test and upkeep half of the framework has been exercised on paper only. Citigroup was analysed after its outcome was known, which is the after-the-event risk ANT's critics name. Both analysts were instances of the same model family. A real test of the analysis half needs a case analysed before its outcome, by people.

Five further limits apply. The authors wrote the Harbourline case, with its failures built in. The specification was revised on the same two cases it was tested on, and has not been applied again since the twelve fixes. The dispute over G1 was settled in the direction that lets these cases pass, on proposed owners. The independent analyst's copy still contained the ninth loss mode and the case's own list of failures, so it could use them but not rediscover them, and the comparison of the two runs was written by the session that drafted the specification. Finally, the desk checks, the independent re-run, the literature search and a review of this paper were all run by instances of one AI model family (Claude, made by Anthropic); the author, a person, agreed every decision and fix.