PermuteLab
Rigorous Infrastructure for Teams Shipping AI Products
About PermuteLab
PermuteLab is built to help engineers who got tired of guessing. We’re transforming how teams develop and deploy AI products by bringing statistical rigor and real instrumentation to AI development.
Named after permutation testing a class of statistical methods that makes no distributional assumptions we embody a philosophy of testing, not guessing.
Don’t approximate, don’t lean on conventions that quietly break in production. Run the test, get the answer, ship with confidence.
We’re building a portfolio of developer tools for teams turning model outputs into shipped products.
Today’s AI teams ship prompts the way web developers shipped CSS in 2008, change something, eyeball examples, push it live, hope nothing breaks.
We’re bringing the maturation that web development, mobile, and data engineering each went through to AI.
Real instrumentation, real statistical rigor, and real correlation between what the model is doing and what the business is doing.
Everything we ship follows three non-negotiable principles: statistical correctness, minimal operational footprint, and honest assessment of what can and cannot be proven.
Our Products
LLM Experimentation Platform
A hosted or self-hosted A/B testing platform for LLM-powered features that produces statistically defensible results. Features include confidence intervals, false discovery rate control, sample ratio mismatch checks, LLM-as-judge evaluations with cost controls, and attribution from variant assignment to downstream business outcomes. Available with SDKs in Python, TypeScript, and Java.
Statistical Analysis & Instrumentation
Comprehensive instrumentation tools that bring real statistical rigor to AI product development. Measure what your models are actually doing, correlate model behavior with business outcomes, and make data-driven decisions with confidence intervals and rigorous statistical methods.
Production Monitoring & Validation
Lightweight monitoring infrastructure designed for application engineers to run in production without requiring a dedicated data science team. Validate model outputs, detect issues before they impact users, and maintain statistical correctness across your AI-powered features.