We are currently building something exciting and will be launching soon. Registration is temporarily closed. Launching soon — registration temporarily closed.

Local evaluation toolkit

Benchmarks measure our evaluation sets, not your traffic. The honest way to decide whether Sentiqra fits your use case is to measure it on your labeled examples — and that data is usually too sensitive to send anywhere. The toolkit below runs the evaluation on your machine: your labeled dataset is read locally and is never transmitted. The only network calls are the same /{domain}/analyze requests your production traffic already makes.

1. Prepare a labeled dataset (JSONL)

One JSON object per line. Required fields: text, domain, label; optional: language (default en), category (free-form, used for slicing).

{"text": "i will kill you tonight", "language": "en", "domain": "trust-safety", "label": "death_threat", "category": "positive"}
{"text": "the film's villain says \"i will kill you\" before escaping", "language": "en", "domain": "trust-safety", "label": "negative", "category": "quotation"}
{"text": "their policies are destroying this country", "language": "en", "domain": "trust-safety", "label": "boundary", "category": "boundary"}

Label semantics — the toolkit computes exactly what the published held-out report does:

  • "negative" — your expectation is no signal. Any emitted signal counts toward the false-positive measure.
  • "<label code>" — a positive item; a hit means that code appears among the emitted signals.
  • "boundary" — genuinely ambiguous; flag rate is reported but never judged.

Use the label codes shown in each analyzer's taxonomy documentation.

2. Run the evaluation locally

curl -O https://sentiqra.tech/eval-toolkit/sentiqra-eval.py
export SENTIQRA_API_KEY=sk_live_...
python3 sentiqra-eval.py --dataset eval.jsonl --out report.json

Python 3.8+, standard library only — read the script before running it (it's short). --delay spaces out API calls (default 1.05s) to stay inside your plan's rate limits. Calls that fail are counted as misses and surfaced — never dropped silently.

3. What the report measures

False-positive measure

Fraction of your negative items where any signal fired — per domain, category, and language bucket. Denominator is always the full evaluated cell.

Abstention & coverage

Share of items returning zero signals. If you only care about quality when a result fires, read it beside coverage — never alone.

Per-label precision / recall

Item-level: an emitted code counts toward that label's FP whenever the item doesn't expect it. Cells under the minimum support are marked insufficient rather than hidden.

Policy errors at operating points

For each stated example score cutoff: coverage, false-action rate on negatives, miss rate on positives. Example points — not recommendations.

All rates carry Wilson 95% intervals. An interval spanning "0–40%" on 5 items tells you the truth a bare percentage would hide.

Sharing results without sharing data

--submit POSTs an allowlisted aggregate to /v1/eval-report — counts, rates, and the public configuration aliases you observed. The schema cannot carry case text or per-item rows; the server rejects any field outside the allowlist. Inspect the exact payload first with --submit-dry-run. Aggregate reports can support a quality investigation; they are evidence for review, not proof of a regression on their own.

Choosing an operating point

There is no universally valid threshold — the right cutoff is a property of your cost function, not of our engine. A moderation queue that tolerates review load should sit at a different point than an auto-enforcement path. Use the policy-error table to find where your FP and miss rates cross a level you can live with, then validate on a second sample before committing it to production.

Our own held-out results — including the languages and categories where we are weakest — are published in the Evidence Center.

4. Comparing configurations before you adopt one

If you pin a configuration (detection_config), a new release does not move you — the pin replays the pinned configuration or fails, and the current configuration is never substituted silently. To decide whether an upgrade changes your policy behavior, run the same evaluation set against both pins with sentiqra-compare.py: 

curl -O https://sentiqra.tech/eval-toolkit/sentiqra-compare.py
export SENTIQRA_API_KEY=sk_live_...
python3 sentiqra-compare.py --dataset eval.jsonl \
    --old <your-current-pin> --new <candidate-pin> \
    --policy policy.json --out diff.json

The tool reports added and removed signal codes, score movements, and — with a policy map ({"actions": {"code": "block", …}, "default": "allow"}) — every row where the downstream action changes. It exits non-zero when behavior differs, so it can gate a pin move in your CI. Everything runs locally: your dataset and policy file are never uploaded, and the report contains row references, not your text.

Once you've picked thresholds, turn them into a versioned policy file with Policy Decisions & Appeals — a deterministic evaluator that maps responses plus your context facts to actions, matched rules, and review requirements, with an append-only appeals record.

5. Validating a staged pipeline before you route on it

A conditional workflow (primary screen → specialized analyzers only when warranted) saves analyzer runs — but the first stage decides which cases the second stage never sees. Individually strong analyzers do not automatically make a strong pipeline. If a gate passes 95% of relevant cases and the specialist detects 95% of what reaches it, end-to-end recall is 0.95 × 0.95 = 0.9025 — measured as conditional rates, no independence assumption needed.

sentiqra-workflow.py replays a versioned sentiqra/workflow/v1 manifest over the recorded responses of an approved baseline — every analyzer run directly on every case — and reports each gate's exclusion of specialist-relevant cases, conditional and end-to-end recall, additional misses, runs saved, and every case where the final policy decision differs from baseline: 

curl -O https://sentiqra.tech/eval-toolkit/sentiqra-workflow.py
curl -O https://sentiqra.tech/eval-toolkit/sentiqra-workflow-record.py
curl -O https://sentiqra.tech/eval-toolkit/sample-workflow.json

# record the approved baseline — every stage's analyzer runs directly
# on every case under its pinned detection_config (billed normally)
python3 sentiqra-workflow-record.py --workflow workflow.json \
    --texts texts.jsonl --out cases.jsonl

# measure the routed graph against it — local replay, no network
python3 sentiqra-workflow.py evaluate --workflow workflow.json \
    --dataset cases.jsonl --out report.json
  • The manifest freezes stage set, gate rules, and detection_config pins per version — bump it on any change.
  • Each stage records completed / failed / overloaded / skipped. A skip is never a negative finding; a gate that can't evaluate (upstream failed or abstained) lands in the report's uncertain bucket, not in exclusions.
  • Domain signals stay separate. The manifest has no score-fusion operator — a condition reads one stage's signals; multiplying or averaging scores into a composite "probability" is rejected at validation.
  • replay_fingerprint covers routing decisions + policy outcomes: reprocessing the same manifest on the same data reproduces both exactly.

6. Translating rates into alert volume and review workload

"95% detection" does not tell your operations team how many incorrect alerts to expect. Alert precision depends on the prevalence of the target condition in your traffic: at 100,000 items/day, 0.1% prevalence, 95% recall, and 1% FPR (hypothetical values — not Sentiqra measurements), the queue sees 95 correct and 999 incorrect alerts — ~8.7% precision, ~54.7 staff-hours/day at 3 minutes per review. sentiqra-prevalence.py applies your measured rates to your own volume, prevalence, and review-time assumptions: 

curl -O https://sentiqra.tech/eval-toolkit/sentiqra-prevalence.py
python3 sentiqra-prevalence.py --eval-report report.json \
    --traffic 100000 --prevalence 0.1% \
    --prevalence-scenarios "0.05%,0.5%" \
    --review-minutes 3 --reviewer-hourly 45 --out workload.json
  • Inputs are split into measured (recall and FPR from your labeled evaluation, carried with support counts and Wilson 95% intervals — propagated through every output) and assumed (traffic, prevalence, review time — yours to vary).
  • Report sentiqra/prevalence-report/v1 contains expected correct alerts, missed cases, incorrect alerts, alert precision, review hours (and cost, if you supply a rate) per prevalence scenario.
  • A rate without a support count is marked unverifiable and produces no interval. Signal scores are match strength under a pinned configuration — never substitute them for probabilities here.

7. Running a controlled pilot before you scale

A successful API integration does not establish that the operation improved — "API connected", "many signals returned", and "reviewers liked the demo" are integration facts, not outcomes. A pilot must compare an agreed workflow against an appropriate baseline arm: a bare before/after comparison can confound the change with unrelated time-dependent effects (seasonality, queue growth, staffing changes).

sentiqra-pilot.py makes the pilot contract explicit — the criteria are written before execution and the evaluation is frozen under a configuration digest: 

curl -O https://sentiqra.tech/eval-toolkit/sentiqra-pilot.py
python3 sentiqra-pilot.py init --out pilot-plan.json
# fill every REQUIRED field — before the pilot starts
python3 sentiqra-pilot.py evaluate \
    --plan pilot-plan.json --measurements measurements.json \
    --out scorecard.json
  • The plan (sentiqra/pilot-plan/v1) requires: a single primary outcome with a target, a parallel or otherwise controlled baseline arm, error/review-capacity/cost limits, at least one stop condition, rollout stages starting in shadow (compare decisions without acting), a declared case unit (conversations stay whole across arms — fragments never split into different groups), frozen config/policy digests, and named ownership.
  • The scorecard (sentiqra/pilot-scorecard/v1) emits an explicit go / no-go / incomplete: a triggered stop condition means no-go even when metrics pass; a missing required metric is incomplete, never passed.
  • Configuration drift is a new evaluation: measurements under a different config digest, policy version, or evaluation ID cannot satisfy the frozen plan — tuning mid-pilot starts a new evaluation rather than retroactively passing the old one.
  • Advancing past a requires_approval stage needs a recorded approval — customer sign-off, with safeguards, before bounded rollout.
  • Unresolved limitations and ownership travel with the decision; a "go" is bounded to the measured stage and frozen configuration, not a blanket production approval.