Developer tooling

Integration testing & failure sandbox

A successful demo is not an integration test. Most production incidents come from the branches quickstarts skip: partial results, overloaded analyzers, refused pins, conflicting retries. The sandbox lets you exercise every documented failure branch locally — no account, no inference, no billing.

What ships

sentiqra-sandbox.py — a local HTTP mock of POST /v1/analyze and POST /verify with the real request-validation, idempotency, and retry-child semantics. Stdlib Python, nothing to install.
sandbox-fixtures.json — the scenario matrix, version-pinned to the API contract registry (registry_version). Our CI fails if the fixtures drift from the contract.
sentiqra-integration-check.py — a harness that drives your adapter through every scenario and grades its decisions and its wire behavior (request log), so unsafe retry loops can't hide behind a passing-looking output.
sentiqra-reference-client.py — a minimal correct adapter. It passes the harness; run --selftest to see the expected result.
sample-policy.json — a versioned sentiqra/policy/v1 policy for exercising downstream decisions, usable with sentiqra-policy.py.
sentiqra-prepare.py — the versioned customer-side input-preparation contract for document and conversation workflows: deterministic sentence-atomic chunking, explicit NFC normalization, speaker/quotation preservation, and a customer-held manifest binding requests and receipts to source offsets. Run --selftest to verify the boundary guarantees.
sentiqra-workflow.py + sentiqra-workflow-record.py + sample-workflow.json — end-to-end validation of staged pipelines: the recorder runs every stage's analyzer directly on every case under its pinned configuration (the approved baseline, billed normally); the evaluator replays the frozen sentiqra/workflow/v1 routing graph over those recorded responses and reports gate exclusions of specialist-relevant cases, conditional/end-to-end recall, additional misses, runs saved, and policy-decision diffs. A deliberately skipped analyzer is never a negative finding.
sentiqra-agent-gateway.py — reference tool-execution gateway for LLM agents (action-boundary enforcement): authorizes actor+tenant+arguments against a versioned sentiqra/agent-policy/v1, invokes the declared required checks itself (the agent cannot skip them), and mints a signed single-use approval bound to tenant, actor, tool, argument digest, policy version, and validity interval. Semantic analysis is evidence, never authorization — a benign result grants nothing.
sentiqra-prevalence.py — the prevalence and review-workload calculator: feeds a sentiqra-eval.py report's measured recall + false-positive rates (with Wilson intervals) through YOUR traffic, prevalence, and review-time assumptions to produce expected correct/incorrect alerts, missed cases, alert precision, and staff-hours per day — across alternative prevalence scenarios. Keeps measured inputs separate from assumptions and never treats signal scores as probabilities.
sentiqra-audit.py — production quality audits that cannot learn only from flagged cases (methodology): plan draws a deterministic stratified sample across flagged, unflagged, abstained, and failed cases — recording selection probabilities so deliberate oversampling of rare or important cases stays correctable — with a --blind mode that hides model predictions until the reviewer has judged. report emits sentiqra/audit-report/v1: sampling design, missing/pending labels disclosed (never counted as negatives), delayed outcomes tracked separately from immediate labels, and full-population accuracy emitted only when every stratum contributed labels — flagged-only labels yield flagged-conditional precision, not production accuracy.
sentiqra-pilot.py — the controlled-pilot contract for scaling a new use case (methodology): init writes a sentiqra/pilot-plan/v1 that must declare the primary outcome, comparison baseline (before/after-only designs are rejected as confounded), error/review-capacity/cost limits, stop conditions, rollout stages starting in shadow, the case unit kept whole across arms, and the frozen evaluation configuration — all before execution. evaluate emits a sentiqra/pilot-scorecard/v1 go/no-go decision: config drift is a new evaluation, a triggered stop condition overrides passing metrics, and unresolved limitations and ownership travel with the decision.

curl -O https://sentiqra.tech/eval-toolkit/sentiqra-sandbox.py
curl -O https://sentiqra.tech/eval-toolkit/sentiqra-integration-check.py
curl -O https://sentiqra.tech/eval-toolkit/sentiqra-reference-client.py
curl -O https://sentiqra.tech/eval-toolkit/sandbox-fixtures.json
curl -O https://sentiqra.tech/eval-toolkit/sample-policy.json

python3 sentiqra-integration-check.py --selftest   # reference client: all scenarios pass
python3 sentiqra-integration-check.py --client "python3 my_adapter.py"

Failure branches you can exercise

Select a scenario per request with X-Sentiqra-Scenario (or ?scenario=, or the sandbox's --scenario default). Validation, idempotency, and pinning behave contractually in every scenario — send a malformed body or reuse a key with a different payload and you get the real error semantics, not a canned response.

Scenario	What it exercises
success	Happy path — still requires per-analyzer status inspection.
no_signals / abstained	Completed runs that detected nothing or declined. Not proof of safety — must route, not clear.
insufficient_content	Per-analyzer `not_applicable` — nothing ran, nothing billed, not a clean bill.
partial	One analyzer fails (`failed`, unbilled). Correct handling: bounded `X-Retry-Failed` child with the same Idempotency-Key, or explicit review — never a verdict from the signals alone.
overloaded / service_overloaded	Admission denial at analyzer level (200, partial) and request level (503). Bounded keyed retries honoring `Retry-After`.
timeout	Server stalls past your client deadline. Retries while the first attempt is in flight get `409 idempotency_in_progress`.
quota_exhausted / rate_limited	429s that are terminal-until-reset vs. retryable-with-backoff. Retrying quota is flagged.
idempotency_conflict	409 on key reuse with a different body. Also exercised organically: same key + same body replays free.
configuration_*	Pin refused (`unavailable`/`suspended`/`retired` via `detection_config`). The current configuration is never silently substituted — your adapter must surface that.
unknown_state	Deliberate contract violation: an analyzer status outside the documented enum. Fail closed, flag it, never assume `completed`.
invalid_receipt	Tampered `verification.signature` — `POST /verify` reports `signature_valid: false`.
malformed_request / internal_error	Terminal 400 vs. bounded-retryable 500.

What the harness checks

Your adapter is invoked per scenario and must write a decision JSON (contract below). The harness grades two sources: the decision, and the sandbox's request log — so a retry storm or missing idempotency key fails even if the final output looks right.

Per-analyzer status inspection — every requested analyzer's status is echoed in the decision; partial results can't pass as full clearances.
Configuration ID preservation — verification.configuration is carried verbatim; a refused pin is never substituted with a different configuration.
Unknown-state handling — undocumented enum values are flagged and treated as not-completed.
Receipt verification — where receipts matter, /verify is consulted and a tampered signature stops the flow.
Retry discipline — total requests stay bounded (cap 5), retries reuse one Idempotency-Key, and terminal errors are not retried.

Adapter contract

your-adapter --base-url URL --key KEY --scenario NAME --text TEXT \
             [--language en] [--analyzers a,b] [--detection-config sqcfg_X] \
             --out decision.json [--timeout SECONDS]

# decision.json — what your integration concluded
{"action": "processed|review|defer|error", "reason": "...",
 "analyzers": {"<slug>": {"status": "...", "assessment": "..."}},
 "unrecognized": ["<undocumented states encountered>"],
 "config_id": "<verification.configuration verbatim>",
 "receipt": {"checked": true, "verified": true},
 "requests_made": 1}

Forward X-Sentiqra-Scenario to the sandbox so the harness's request log attributes your requests correctly.

Honest boundaries

Every signal and receipt the sandbox emits is a labeled fixture (fixture_* codes, sqcfg_TESTDATA aliases). Nothing is a detector prediction; nothing reaches inference or billing. Don't quote fixture output as capability evidence.
The sandbox signs fixture receipts with HMAC (contract v2 semantics) and verifies them via POST /verify. Production receipts carry Ed25519 signatures verifiable offline — test that path against the real /v1/verify/public-key with the standalone verifier.
Sandbox fingerprinting uses a deterministic canonicalization sufficient for simulation; production uses RFC 8785 JCS. Conformance-test receipts against production, not the sandbox.
The harness grades observable behavior (decision + wire log). It can't prove your UI renders states honestly — pair it with your own review.