Foundations: Judgement & Coherence
On Judgement, Coherence, and Measurement
The underlying question
Any system that evaluates AI behaviour must resolve a prior question:
By what criteria is behaviour judged?
This question is not peripheral.
It defines the meaning of every metric, score, and downstream interpretation.
There is no neutral evaluation
All evaluation systems depend on selected criteria:
- what is measured
- how it is measured
- how outcomes are interpreted
These selections encode assumptions about:
- usefulness
- safety
- truthfulness
- acceptability
In most systems, these assumptions remain implicit.
Position
The RI Safety Layer does not attempt to eliminate human judgement.
Its current architecture makes evaluative criteria, derived interpretation, publication governance, and bounded human action explicit while keeping their authority distinct.
Structural approach
Evaluation criteria and their application are intended to be:
- declared — criteria are defined explicitly
- structured — evaluation procedures are specified
- versioned — changes to criteria can be tracked over time
- traceable — recorded outputs can be related back to the evidence, method, and rules that produced them
This allows a result to be examined in terms of:
- the behaviour recorded
- the measurement process applied
- the criteria used
- any downstream interpretation
- any separate publication-governance or human-action record
Separation of measurement, interpretation, governance and human action
A core architectural principle is that these functions do not collapse into a single “judgement” layer:
- measurement — bounded evaluation outputs derived from recorded evidence
- interpretation — advisory meaning or comparison derived from measurement and context
- publication governance — authority over whether eligible results count in published metrics or dashboards
- human stewardship — bounded records of human review or action
Measurement and canonical evidence are not rewritten merely because interpretation changes.
Interpretation does not become publication authority merely by being recorded.
A human action record does not itself become canonical evidence, an already-computed metric rewrite, or a governance decision.
Definition of coherence
Within this system, coherence is defined operationally.
Coherence is the degree of alignment between observed behaviour and a declared set of evaluation criteria.
These criteria are:
- explicitly defined
- empirically testable within their stated frame
- context-dependent
The system does not assume a single universal definition of coherence.
Different domains may define coherence differently, provided those definitions are made explicit and applied consistently.
Foundational basis
The evaluation criteria used within the system are informed by an underlying research programme focused on:
- behavioural consistency
- response stability under variation
- alignment between stated intent and produced output
- structural properties of interaction patterns
This work combines:
- empirical evaluation design
- behavioural analysis across repeated trials
- internally developed theoretical models of coherence and system alignment
These models are not treated as fixed doctrine.
They are subject to testing, revision, and evidential limitation.
Where appropriate, elements of this framework are formalised and published in technical form.
What the current architecture supports
The current public claim is bounded.
The architecture supports:
- explicit criteria and inspectable evaluation methods
- deterministic re-derivation from fixed recorded evidence under the same evaluation configuration
- traceability between available evidence, derived outputs, interpretation, and governance records
- separation between measurement, interpretation, publication governance, and bounded human action
It does not claim deterministic live-model generation, universal construct validity, independent external validation, or institution-specific readiness.
Implication
This shifts evaluation from assertion toward structured inspection.
Instead of asserting:
- “This system is safe”
- “This output is correct”
the system can provide or route an authorised reviewer toward:
- the recorded behaviour
- the measurement process
- the criteria applied
- the derived interpretation, where present
- the separate governance or stewardship record, where present
Where the relevant evidence is available and access is authorised, a reviewer can re-derive deterministic evaluation outputs, examine assumptions, and localise disagreement.
That inspectability is not the same as independent external validation, and public representation is not itself canonical evidence.
Result
Evaluation can become:
- more inspectable rather than implicit
- re-derivable from fixed evidence rather than dependent on an unrecorded process
- contestable at the level of evidence, method, criteria, interpretation, or governance
The aim is to reduce reliance on unexamined trust by making relevant boundaries and provenance recoverable.