RI Safety Layer

Appendix A

Judgement, Coherence, and Deterministic Evaluation

A Formal Note on Epistemic Structure in the RI Safety Layer (v1.0)

Current-state clarification — October 2026

This v1.0 appendix uses “judgement” as a broad historical umbrella for functions that the current implementation now separates more explicitly.

For current interpretation of this note, keep distinct:

  • measurement — bounded outputs derived from recorded evidence
  • derived interpretation — advisory meaning or comparison that does not become canonical evidence
  • publication governance — separate authority over whether eligible results count in published metrics or dashboards
  • human stewardship — bounded records of review or action that do not rewrite evidence or automatically acquire governance authority

References below to reproducibility mean deterministic re-derivation from fixed recorded evidence under the same evaluation configuration. They do not assert deterministic live-model generation or complete end-to-end repeat-run evidentiary closure.

References to inspection, audit, regulatory use, or independent observers describe architectural support for review. They do not establish independent external validation, institutional or operational readiness, certification, assurance sufficiency, or jurisdiction-specific compliance sufficiency. Access to canonical evidence may be controlled.

Read the current evidence and limitation boundaries.

A.1 Purpose of this appendix

This appendix provides a formal clarification of how the RI Safety Layer defines and implements:

  • judgement
  • coherence
  • behavioural evaluation

It addresses a central question in AI evaluation:

How are evaluation criteria defined, and why should they be considered meaningful?

The aim is not to assert a universal definition of “correct” behaviour, but to describe:

  • how criteria are constructed
  • how they are applied
  • how their influence on outcomes can be inspected, reproduced, and challenged

A.2 The epistemic problem in AI evaluation

All AI evaluation systems depend on prior assumptions.

These assumptions determine:

  • what behaviours are considered desirable
  • what constitutes failure
  • how trade-offs are resolved

In most existing evaluation frameworks, these assumptions are:

  • embedded in benchmark design
  • implicit in scoring systems
  • difficult to isolate from results

This creates three core limitations:

  • Opacity — the basis of judgement is not fully visible
  • Non-reproducibility of interpretation — results cannot be cleanly separated from evaluative assumptions
  • Limited contestability — disagreement is difficult to localise (data vs method vs judgement)

A.3 System position

The RI Safety Layer treats judgement as an explicit and separable component of evaluation.

It does not attempt to:

  • remove human-defined criteria
  • claim objective or universal correctness

Instead, it enforces:

explicit, structured, and inspectable judgement

The system does not eliminate subjectivity.

It constrains and exposes it.

A.3.1 System Architecture Overview

The system can be understood structurally as a layered architecture, shown below.

System Stack Diagram

Layered stack from foundational theory through coherence model, behavioural layer, measurement layer, governance layer, and system layer.
Figure A0. System architecture of the RI Safety Layer. The system is structured as a layered stack progressing from foundational theory through coherence modelling, behavioural definition, deterministic measurement, and governance, to external system interfaces. Each layer operates with defined responsibilities and constraints, preventing the conflation of observation, evaluation, and judgement.

A.4 Architectural separation

The system is built on a strict separation between three layers:

A.4.1 Behaviour (Observed Interaction)

  • raw model outputs
  • full conversational traces
  • input–output sequences

This layer is:

  • recorded in full
  • immutable once captured

A.4.2 Measurement (Deterministic Evaluation)

  • application of defined evaluation procedures
  • generation of structured metrics
  • production of summaries that can be re-derived from fixed recorded evidence under the same evaluation configuration

This layer is:

  • deterministic
  • repeatable under identical conditions
  • independent of downstream judgement

A.4.3 Judgement (Interpretive Layer)

  • definition of evaluation criteria
  • thresholds, classifications, and containment rules
  • decisions regarding inclusion in aggregate metrics

This layer is:

  • explicitly defined
  • version-controlled
  • auditable

A.4.4 Key property

Measurement does not depend on judgement.

Judgement operates on the outputs of measurement.

This prevents:

  • retrospective reinterpretation of behaviour
  • hidden modification of evaluation outcomes

Behaviour / Measurement / Judgement Separation

Behaviour flows to measurement, and measurement outputs flow to judgement.
Figure A1. Separation of behaviour, measurement, and judgement. Behaviour is recorded as immutable evidence. Measurement produces deterministic outputs from that evidence. Judgement operates only on measurement outputs and does not alter them.

A.5 Determinism in behavioural measurement

Within the RI Safety Layer, determinism refers to:

the property that identical inputs, evaluation procedures, and system configurations produce identical outputs

This applies to:

  • metric computation
  • summary generation
  • evidence bundle construction

Determinism ensures:

  • reproducibility of evaluation outputs from fixed recorded evidence under the same evaluation configuration
  • stability of evaluation outputs
  • independence from runtime variability

This is distinct from model determinism.

The system does not require the underlying model to be deterministic.

Instead, it ensures that:

given a recorded interaction, its evaluation is deterministic

Deterministic Evaluation Path

Recorded interaction flows to deterministic measurement, producing stable outputs.
Figure A2. Deterministic evaluation. Given a fixed recorded interaction and evaluation configuration, the system produces identical outputs across runs. Determinism applies to the evaluation layer, not to the underlying model.

A.6 Operational definition of coherence

Coherence is defined operationally as:

the degree of alignment between observed behaviour and a declared set of evaluation criteria

Formally, for a given interaction I and criteria set C:

Coherence(I, C) = ƒ(Measurement(I), C)

Where:

  • Measurement(I) produces structured behavioural descriptors
  • C defines evaluative expectations
  • ƒ applies criteria to measured behaviour

Coherence evaluated within a defined band

Coherence is evaluated along a bounded region from incoherent to coherent, with an acceptable band defined by criteria set C.
Figure A4. For a given criteria set C, behaviour is evaluated within a bounded region of acceptable alignment. The position, width, and threshold of this band are determined by explicit criteria and may vary across contexts.

A.6.1 Key properties

  • coherence is criteria-relative
  • coherence is context-dependent
  • coherence is computable and re-derivable from fixed recorded evidence within a stated evaluative frame
  • coherence is open to inspection and challenge
  • coherence is not claimed as universal truth

A.6.2 Upper-bound interpretation

There is no absolute “highest coherence” independent of context.

Within this system, the upper bound of coherence is defined as:

maximal alignment with the currently declared criteria set C

Changing the criteria set may change the location or meaning of that upper bound.

A.6.3 Practical meaning

The system does not claim to discover coherence as a universal property.

It provides a transparent mechanism for defining, applying, and testing coherence within a specified evaluative frame.

This allows disagreement to focus on:

  • the measured behaviour
  • the criteria chosen
  • the procedure ƒ used to map behaviour to alignment outcomes

rather than on hidden or implicit judgement.

A.7 Construction of evaluation criteria

Evaluation criteria within the system are constructed through a combination of:

A.7.1 Empirical evaluation design

  • repeated testing across varied prompts
  • identification of stable behavioural patterns
  • measurement of response consistency

A.7.2 Behavioural analysis

  • examination of model outputs under perturbation
  • detection of instability, contradiction, or drift
  • analysis of alignment between stated intent and response

A.7.3 Theoretical modelling

Internally developed models of coherence and system alignment inform:

  • the structure of criteria
  • the selection of behavioural dimensions
  • the interpretation of observed patterns

These models aim to describe:

  • consistency under variation
  • structural integrity of responses
  • relationship between prompt conditions and outputs

A.7.4 Iterative refinement

Criteria are not fixed.

They are:

  • updated as new behaviours are observed
  • stress-tested against edge cases
  • evaluated for unintended consequences

All changes are:

  • versioned
  • documented
  • applied prospectively (not retroactively)

A.7.5 Role of the Coherence Model

The construction of evaluation criteria is informed by an underlying model of behavioural coherence.

This model does not function as an absolute authority.

It provides a structured, explicitly defined interpretive lens through which behavioural patterns are understood.

Specifically, it:

  • identifies which behavioural dimensions are meaningful for evaluation
  • informs the construction of criteria applied to those dimensions
  • defines the contextual band within which behaviour is assessed

Evaluation therefore does not occur against an absolute standard.

It occurs relative to:

  • explicitly defined criteria
  • informed by a model
  • applied consistently within a given context

The coherence model constrains interpretation — it does not replace it.

Coherence Model as Interpretive Lens

Observed behaviour passes through the coherence model, interpretation, criteria application, and measured outcome, with a test and challenge loop.
Figure A3. The coherence model as an interpretive lens. Observed behaviour is not evaluated directly against an absolute standard. It is interpreted through a structured model, which informs criteria selection and evaluation. Outputs are then tested through reproducibility and open inspection.

The validity of this approach is not asserted.

It is tested through:

  • deterministic re-derivation from fixed recorded evidence
  • stability of measurement
  • and the ability of authorised reviewers to inspect and challenge outcomes where evidence access permits

A.8 Governance and containment

In this v1.0 note, “judgement” included publication-governance rules. In the current architecture, publication governance is treated as a separate authority that determines:

  • whether a session contributes to aggregate metrics
  • whether results are surfaced in dashboards
  • whether outputs are flagged, held, or excluded

These decisions are:

  • derived from explicit criteria
  • recorded as signed governance outcomes
  • traceable to both measurement and criteria

A.9 Contestability and inspection

A central design objective is to enable structured disagreement.

Where evidence access is authorised, reviewers can:

  • inspect the raw interaction
  • reproduce the measurement outputs from fixed recorded evidence
  • examine the applied criteria
  • evaluate the resulting interpretation or governance decision

Disagreement can therefore be localised to:

  • the behaviour itself
  • the measurement process
  • the criteria definition

This supports:

  • clarity of critique
  • structured refinement
  • bounded technical review

A.10 Limitations

The system does not:

  • establish a universal definition of safe or correct behaviour
  • eliminate the need for human judgement
  • guarantee that chosen criteria are optimal

Instead, it provides:

  • a framework in which judgement is explicit
  • a process in which evaluation is reproducible from fixed recorded evidence under fixed conditions
  • a structure in which assumptions can be tested and revised

A.11 Implications for AI safety evaluation

By separating behaviour, measurement, interpretation, and publication governance, the current architecture supports:

  • deterministic re-derivation of bounded evaluation outputs from fixed evidence
  • governance of metrics prior to publication
  • traceable linkage between assumptions and outputs
  • bounded technical review where evidence access is authorised

These properties do not themselves establish regulatory acceptance, third-party audit sufficiency, independent validation, or institutional readiness.

A.12 Summary

The RI Safety Layer introduces a structured approach to AI evaluation in which:

  • behaviour is recorded
  • measurement can be deterministically re-derived from fixed recorded evidence
  • derived interpretation remains distinct from canonical evidence
  • publication governance remains a separate authority

Coherence is not assumed.

It is defined, applied, and open to challenge within a stated evaluative frame.

Inspectable process, bounded reproducibility, explicit assumptions, and visible limitations replace stronger claims of universal verification or readiness.