# Verity open benchmark — replication kit: toolmark-v1

Screwdriver toolmarks (tmaRks, consecutively manufactured; tool edges)

* split_hash: `4ead8b698bee3ce2224ed502f2bb6169118617c883b47e2e46df3056f11c9b20`
* pairs: 167332 (3530 same-source / 163802 different-source)
* sources: 56, frozen folds: 10
* headline metric: calibration loss (Cllr − Cllr_min, mean over source-disjoint folds)

## Contents

* `pairs.csv.gz`     — the frozen pairs: `pair_id,hash_a,hash_b,label,source_a,source_b,folds`.
  `hash_a`/`hash_b` are SHA-256 content hashes of the marks (a multi-scan mark hashes its
  sorted scan hashes); `folds` lists the fold indices in which the pair is a *test* pair.
* `marks.csv.gz`     — the mark-hash → scan mapping:
  `mark_hash,source,label,n_scans,scan_hashes` (`scan_hashes` is ";"-joined).
  Composite (multi-scan) `hash_a`/`hash_b` values resolve to their individual
  scan content hashes here.
* `folds.json`       — each frozen fold's held-out source set.
* `provenance.json`  — protocol, scorer config hash, datasets, and the `split_hash`.
* `scoring.py`       — the scorer (numpy-only), byte-identical to the leaderboard's.
* `evaluate.py`      — score a submission offline: `python evaluate.py my_submission.csv`.

## The submission contract

One likelihood ratio per pair. The LR for a pair must be produced without using the benchmark labels of any pair involving either of its sources (leave-the-pair's-sources-out) — the same source-disjoint discipline the frozen folds score.

A submission CSV has a `pair_id,lr` header and exactly one finite, strictly
positive likelihood ratio per frozen pair. Offline `evaluate.py` output equals
the leaderboard scoring; submit via:

    POST https://data.verity.codes/benchmark/splits/toolmark-v1/submissions
    {"submitter": "you", "method": "your-method", "url": "https://…", "csv": "<pair_id,lr…>"}

The underlying scans are public: resolve any `hash_a`/`hash_b` against the
Verity catalog (https://verity.codes/catalog, content-addressed) or recompute
the hashes from the source datasets listed in `provenance.json`.
