Speed reveals truth; patience reveals value.

The narrative that AI models are black boxes is no longer a fringe concern. Last week, Vals AI closed a $40 million Series A round led by a16z at a $400 million valuation, positioning itself as the first independent, third-party evaluation layer for large language models (LLMs). The pitch is seductive: instead of relying on static benchmarks like GSM8K or HumanEval—which are increasingly contaminated by training data—Vals claims to assess models on real-world tasks extracted from GitHub pull requests. The hook? OpenAI, Anthropic, Google, Meta, and xAI have all allegedly cited Vals’s results in their model cards.

Context: Why Now? The AI evaluation market has been a silent war. For years, model performance has been measured by academic benchmarks that are gamed, leaked, or simply irrelevant to production environments. The industry’s dirty secret: every major model vendor optimizes for these benchmarks, creating a “Goodhart’s Law” feedback loop where the metric becomes the target. Enterprise buyers, meanwhile, struggle to compare models on their own codebases or workflows. Vals enters this void with a seemingly simple solution: extract hidden tests from historical pull requests across any GitHub repository, run them against the model, and grade the output automatically. The “private, personalized” evaluation is the core product—and the entry point for B2B sales.

But the timing is non-trivial. The post-Dencun era in crypto, for example, has shown that infrastructure layers can become saturated quickly. Similarly, the AI evaluation layer is still nascent, but capital is flooding in. a16z’s bet on Vals is not a bet on current revenue (which remains undisclosed) but on the belief that third-party evaluation will become a mandatory component of AI procurement, much like SOC 2 audits are for cloud services. The valuation of $400 million for a company with no public ARR and a vague “8x revenue growth” claim (likely a misstatement of “8x vs. 2025 annual projection”) signals that VCs are buying the category, not the company.

Core: Key Facts and Immediate Impact

Let’s dissect the tech. Vals’s innovation is not in model architecture but in evaluation infrastructure. They pull actual developer tasks from any public GitHub repository’s history—issues, pull requests, code reviews—and convert them into hidden tests. The model is asked to complete the task, and the output is compared against the ground truth (the actual merged code). This is a direct productization of the SWE-bench paradigm, but with a twist: it’s cross-domain, covering finance, law, and medicine. The immediate impact is twofold:

1. Enterprise adoption: Companies can now evaluate models against their own private codebases, lowering the switching cost between providers. This shifts decision-making from “which model is best on the leaderboard” to “which model is best for my specific repo.”
2. Model accountability: If major labs cite Vals’s results, it creates a de facto standard. This could force smaller open-source models to also undergo third-party evaluation, accelerating the “audit economy.”

Based on my experience reverse-engineering 0x V2’s contract architecture in 2017, I know that speed of verification is often more important than depth of analysis. Vals’s approach is fast, but it inherits the same risks as any black-box testing: the test set itself can be contaminated, and the model’s performance on “hidden” tasks may not generalize to unseen workflows.

From a data perspective, Vals claims to have evaluated models across thousands of repositories. But the sample size and statistical significance of these evaluations are not disclosed. In my analysis of Aavegotchi’s NFT data in 2021, I learned that small sample sizes can produce misleading narratives. The same applies here: if Vals’s evaluations are based on a narrow set of repositories (e.g., only popular open-source projects), the results may not be representative of production environments.

Contrarian: The Unreported Angle

Here’s the problem no one is talking about: Vals’s independence is compromised by its business model. The same companies that are being evaluated—OpenAI, Anthropic, etc.—are also potential customers. If Vals charges model vendors for “evaluation as a service,” the incentive to produce favorable reports is real. The article does not disclose whether the model card citations are paid or unpaid. If they are paid, the whole “third-party” claim is a farce. It’s the same conflict of interest we see in the crypto space with audit firms that are paid by the same protocols they audit. The Terra/Luna collapse taught us that paid audits can be selective.

Moreover, the technical risk of reverse-engineering the hidden tests is substantial. If Vals uses public GitHub history, model vendors can train on that data. Vals claims to use private repositories for enterprise customers, but that creates a two-tier system: public benchmarks vs. private ones. The private ones are more trustworthy, but they also require deep integration with the customer’s codebase, which is a high-friction sell.

Another blind spot: cross-domain evaluation. Finance, law, and medicine require domain-specific verification. Can Vals’s automated tests accurately assess a legal contract’s compliance or a medical diagnosis’s correctness? Unlikely without human-in-the-loop validation. The article does not mention the human cost of labeling these tasks, which suggests the company may be over-promising on automation.

Finally, the article’s source is a blockchain monitoring channel called “Dongcha Beating.” This is a red flag: the information is drawn from an anonymous, non-credentialed source. In my 2024 Bitcoin ETF whitepaper breakdown, I learned that institutional-grade analysis requires verifiable sources. The lack of independent verification for Vals’s revenue claims, customer count, and model card citations means the entire narrative rests on trust. In a market that is already skeptical of AI hype, that trust is fragile.

Takeaway: What to Watch Next

The real test will be whether Vals can resist the gravitational pull of its own investors. a16z has a portfolio of AI companies; if Vals starts auditing those companies, the conflict of interest will be hard to ignore. Watch for three signals: - Disclosure of pricing and customer contracts: If Vals publishes a transparent fee structure for model vendors, it will increase credibility. - Independent third-party verification of its own evaluation methodology: Who audits the auditors? If Vals submits its own tests to a public audit, it would set a standard. - Adoption in regulated industries: If financial regulators or healthcare bodies cite Vals’s evaluations as evidence of model safety, the market will legitimize. Until then, this is a high-stakes gamble on the “audit economy” that may or may not materialize.

Speed reveals truth; patience reveals value. The truth of Vals’s claims will emerge not in press releases, but in the next model card update that either includes or excludes their logo. I’ll be watching the git history.