# Version 2 internal synthetic reference evaluation

Run from any directory with Python 3.11+:

```sh
python3 app/static/assets/v2/eval/run.py
```

This writes `results.json` (192 individual retrieval cases, 144 permission-denied cases and their summaries) and `chart.svg` in this folder. No services, personal records, API keys, external packages or network calls are used. The code contains the entire deterministic input generator.

## What the numbers mean

There are 48 invented projects, with an owner on each project, and one linked document, meeting and thread per project. The reference lookup answers "who owns this record?" by finding the associated project and returning its owner. It verifies the source permission before looking up a record. For realism *within this deliberately simple fixture*, links are withheld for document IDs divisible by 8 (6 cases), meeting IDs divisible by 6 (8 cases), and thread IDs divisible by 4 (12 cases). Each of the 192 task queries counts as correct only if it returns the known owner; missing links count as failures. The separate 144 checks deny the queried source and count any returned owner as an exposure.

The observed retrieval successes are 48/48 direct project queries, 42/48 document queries, 40/48 meeting queries and 36/48 thread queries. Denied-source exposures: 0/144. Raw successes **and** failed cases are retained in `results.json`.

## Scope and limits

This tests a **small, stand-alone reference function**, not production Zortex Version 2. It does not evaluate real connectors, data freshness, document parsing, entity matching, relevance ranking, prompt construction, AI answers, latency, or a security boundary in the application. Deliberately missing edges make the failure counts partly a property of the fixture construction; the task set is neither a representative population nor an external industry benchmark. No named competitors were measured. No product claims should be inferred from these percentages.

The launch copy calls these *internal synthetic reference-evaluation results* rather than product benchmark scores. The chart is drawn from the same `task_summary` records as the JSON output, by `run.py`.