Long-horizon finance under hidden constraints
How frontier models perform when a correct spreadsheet is not enough and the recommendation must also obey source agreements and policy rules.
Request preview ↗
Research and benchmarks
Measure where the frontier still breaks.
Sofitra studies model performance inside real professional workflows: long horizons, incomplete evidence, stateful tools, hidden constraints, recoverable errors and outcomes that can be audited.
Research thesis
A benchmark is useful only while it exposes a capability gap.
We combine hidden task distributions with action-level replay and component rewards, so researchers can see not only whether a model failed, but where, how and why.
Current programme
Sofitra Frontier Finance Index.
A continuously refreshed view of model performance across source-grounded finance work. Public releases will use sanitised samples; the full task distribution and hidden graders remain private.
Index architecture
One score is not enough to explain professional capability.
The index separates outcome quality, source integrity, spreadsheet integrity, process safety, constraint compliance, recovery behaviour and professional judgment.
01Source grounding
02Accounting integrity
03Tool execution
04Error recovery
05Judgment quality
Research releases
What we are measuring.
The first public reports are in preparation. Approved findings, model names and scores will be published here as they are cleared for release.
When models repair the symptom, not the cause
A taxonomy of spreadsheet-repair failures: hard-coded patches, broken lineage, hidden circularity and case-specific overfitting.
Join the research list ↗Designing verifiers that resist plausible shortcuts
Layered outcome, process and evidence checks for tasks where superficial completion can otherwise earn misleading reward.
Request a briefing ↗Failure taxonomy
Replay turns a score into a diagnosis.
We classify failures at the point they occur in the workflow and connect them to the reward components they affect.
Understanding failures
The model reads the wrong source, misses a note, confuses periods or invents support.
- Retrieval miss
- Source authority error
- Temporal mismatch
- Unsupported claim
Action failures
The model chooses the wrong tool, corrupts state, skips a dependency or cannot recover.
- Tool-selection error
- Unsafe state change
- Sequence failure
- Recovery failure
Judgment failures
The work calculates correctly but misses materiality, constraints, escalation or decision logic.
- Constraint neglect
- Materiality error
- Risk omission
- Recommendation mismatch
Benchmark methodology
Designed to remain useful as models improve.
The public-facing benchmark is a small window into a larger private environment and refresh system.
Private tasks and graders
Holdouts, edge cases and critical reward logic remain undisclosed to reduce leakage and targeted overfitting.
Model and human calibration
Difficulty is measured against selected model baselines and expert completion, not intuition alone.
Explainable scoring
Outcome, grounding, process, constraint and judgment components are reported separately.
Failure-led evolution
As task families saturate, observed shortcuts and new operating cases generate the next distribution.
Responsible publication
Reveal capability gaps without revealing private data.
Public artifacts are sanitised, rights-cleared and deliberately separated from the full source corpus and hidden evaluation infrastructure.
Privacy-safe examples
Public cases use transformed entities, controlled excerpts or purpose-built derivative samples.
Hidden evaluation estate
Private task distributions, source universes and critical graders are not released.
Methodology clarity
Scope, model configuration, scoring method and known limitations are documented.
Reproducible releases
Environment, task and verifier versions are tracked so results can be interpreted correctly.
Benchmark access
See how a model fails before deciding how to train it.
Request a narrow benchmark, a model comparison or a research briefing on environment and verifier design.