Measuring Smoke: The Tokenomics Foundation and AI's Impossible Unit Problem
The press release arrived with the defensive posture of a man holding two passports. "Tokenomics Foundation," it announced—a new standards body dedicated to the measurement and standardization of AI token economics. Then, in language that repeated itself three times across a single page: "This is not affiliated with cryptocurrency. This has nothing to do with crypto. No relation to the blockchain ecosystem whatsoever."
Twice would be suspicious. Three times is an admission.
I have been auditing token systems for a decade. I started with ICO whitepapers in 2017, moved through DeFi liquidity stress testing in 2020, watched the NFT floor-price fiction collapse in 2022, and built macroeconomic models for the Abu Dhabi Central Bank's digital currency pilot. In all that time, one rule has never failed: when a project spends more words denying a connection than explaining its substance, the substance is thin.
There is no website. No founding membership list. No draft standard. No technical whitepaper. No reference implementation. What exists is a name, a press release, and a conviction that the AI industry's token accounting chaos is a problem worth solving.
The conviction is correct. The execution is another matter.
Understand the problem first. It is real, it is structural, and it is getting worse.
The market prices AI inference in units called tokens. OpenAI charges per token. Anthropic charges per token. Google charges per token. Meta's open models are deployed through providers who charge per token. The entire AI industry—from API vendors to GPU cloud providers—has converged on this unit as its atomic accounting standard. The flaw is that a token is not an atomic unit. It is not comparable across systems.
The tokenizers used by major model vendors sit on different algorithmic families. Byte-pair encoding variants. SentencePiece. Byte-level tokenization. Each produces different segmentations of the same input text. An English sentence might tokenize into 25 tokens in one vendor's system and 31 in another's. The divergence is worse for multilingual text, where tokenizer quality varies wildly across languages. For the same semantic content, a Japanese user might consume 2.3 times the tokens of an English user, depending on the model's tokenizer vocabulary.
Then there is the multimodal problem. Vision models convert images into patches; sending an image to some APIs can consume hundreds of tokens' worth of compute. Audio models convert speech into frames. Video models convert content into sequences measured entirely differently. Each vendor defines these conversions through proprietary heuristics. There is no common denominator.
I quantify this because the gap actually matters. In analysis I performed for institutional clients tracking AI infrastructure exposure, the token variance between two leading API providers for identical multilingual workloads ranged from 18 to 40 percent. That is not noise. That is a pricing opacity engine. What this means in practice: enterprises running identical AI workloads on different providers cannot compare costs, cannot predict budgets, and cannot perform basic unit economics. "Cost per million tokens" is the industry's headline metric. It is also one of its least meaningful ones.
Now consider what Tokenomics Foundation claims it will do: standardize the measurement. On its face, an honorable mission. In practice, a minefield.
The core question is not whether token measurement can be standardized. Everything can be standardized if you have enough power, cruelty, and time. The question is whether this organization can do it—and more importantly, whether the people who benefit from the current chaos will allow it to survive.
Let me take the technical problem seriously, because the technical problem is where every standard either gains credibility or loses it. If you want to understand why AI token measurement is a mess, you have to start with the tokenizers themselves, because the tokenizer is the unit-defining artifact. In metrology, the discipline of measurement science, you always start with the artifact. You ask: what is the physical realization of the unit? For the meter, it was a platinum-iridium bar, then a frequency of light. For the second, it is the vibration of cesium-133 atoms. For the AI token, there is no artifact. There is only each vendor's algorithmic choice.
Byte-pair encoding, the dominant family, works by iteratively merging the most frequent pairs of byte sequences in a training corpus. The vocabulary is emergent, corpus-dependent. Train it on English and you get a tokenizer that treats "the" as one token but fragments "Zurich" into Z-u-r-i-c-h because the word is rare in training data. Train it on code and you get efficient tokenization of syntax but inefficient tokenization of natural language. Train it on a mixture, as most modern tokenizers are, and you get a set of compromises that encode the biases of the training distribution into the unit of account.
SentencePiece operates differently. It treats the input as a raw character stream and applies either BPE or unigram language modeling directly to the Unicode codepoints. This handles multilingual text better, but at a cost: it produces more tokens per word for languages that use the Latin alphabet, because every character boundary is a candidate split. The result is that a standard English sentence costs more tokens under SentencePiece than under a well-tuned BPE tokenizer.
Byte-level tokenizers sidestep the vocabulary problem entirely. They convert everything to bytes—256 possible values—then apply BPE on the byte stream. This is the most lossless approach. Every possible input can be represented. But it is also the most token-inefficient for alphabet-heavy languages. The English word "tokenization" might be a single token in a BPE model with a rich vocabulary, three tokens in a SentencePiece model, and five or six byte-level tokens.
Here is what this means for standardization. If you want a canonical token count, you are actually choosing one of three philosophies. Do you want to count semantic units, which requires a tokenizer trained to encode meaning efficiently? Do you want to count character-level information content, which means the byte-level tokenizer becomes your reference? Or do you want to count compute consumption, in which case the token count must be tied to the specific matmul operations on a specific GPU architecture? These are not the same. They give different numbers for the same input. And no amount of clever conversion will reconcile them perfectly, because they are measuring different dimensions of the same object.
This is the metrology problem at its sharpest. A standard must choose a dimension. The dimension defines the artifact. The artifact defines the unit. The unit defines the price. Whoever controls that choice controls the money flows underneath.
Take the multilingual tax as a concrete example. In my audit work with enterprises operating across Southeast Asia, I have seen firsthand what happens when a company compares token costs for the same product across English and Vietnamese. The token multiplier for Vietnamese text—the ratio of Vietnamese tokens to English tokens for the same semantic content—ranges from 1.8 to 2.6 depending on the tokenizer. This is not an efficiency difference. It is a tax. A Vietnamese-speaking user is paying 2.6 times as much per unit of meaning as an English-speaking user. The API provider can claim they are charging a fair per-token rate. The token count does not lie, but it also does not reflect meaning. The moment a standard defines "one token" as one unit of the canonical tokenizer's output, it is locking in a certain distribution of multilingual taxation. If the canonical tokenizer is trained on English-heavy data—which is almost certainly what will happen, because that is what most available training data looks like—the standard will institutionalize the multilingual penalty. It will not solve the problem. It will make the problem permanent.
Consensus is fragile, and nowhere is that more visible than in the standards process.
Now move to the economic dimension, because this is where I have spent most of my career. The standardization paradox can be stated simply: the entities most capable of creating the standard are the ones with the largest incentive to keep it weak. The entities most harmed by the absence of a standard are the ones with the least technical leverage to create one.
If you are an API provider, your pricing structure is designed around a specific tension. Consumers must be able to understand the price. Competitors must not be able to easily compare it. That tension produces the market as it exists: a unified token vocabulary over divergent tokenizer implementations. Every provider can claim they charge X dollars per million tokens while the denominator—what those tokens are—differs by a factor of 1.3 or more depending on the workload. This is not fraud. It is engineering, applied to accounting arbitrage.
A standardized token would kill that arbitrage. Or rather, it would drastically reduce it. If an enterprise can map every provider's token count into a canonical metric, it can compare unit costs with the confidence of a commodity buyer standing in front of a grain silo. Providers know this. They also know which direction the wind blows: when comparability increases, price pressure on higher-cost providers increases. Unless their tokenizer produces fewer tokens per unit of semantic content, which is itself a strategic design variable. OpenAI has spent years optimizing its tokenizer to compress English better than its competitors. That is a commercial decision disguised as a technical one. The tokenizer is a competitive weapon.
So a standards organization enters a battlefield where the weapon is the unit of measurement. The model vendors will not surrender their tokenizers without a fight. What they might do, if the standard becomes unavoidable, is adopt a dual system: report their proprietary token count for modeling purposes and the canonical token count for billing comparison. That would preserve their operational freedom while complying with the standard's spirit. But it would also expose the gap between the two, which could generate the very comparison pressure they are trying to avoid. The strategic equilibrium is tricky.
What would a real standard require? Let me take seriously what a functional standard would look like. Based on my experience auditing standardized protocols, I would expect, at minimum, five things.
One: a reference tokenizer specification. Not a description of tokenizers, but a normative, implementable algorithm with precisely defined outputs. Probably a BPE variant with a defined vocabulary, or a defined byte-level mapping. It must be a benchmarkable, testable artifact, with behavior defined in terms of known invariants. Think of it as the SI defining the meter not as a bar in a basement, but as a specific frequency of light. A reference tokenizer would be the same kind of object: executable, verifiable, reproducible.
Two: a conversion schema. A mapping protocol between the canonical token and the proprietary token counts reported by each vendor. This is the economic heart of the standard. It would also be the part most likely to fail, because each vendor's tokenizer changes when their models change. The conversion tables become outdated the moment a new model release arrives. Which is always.
Three: a measurement methodology for non-text content. Image patches, audio frames, multimodal sequences. The standard would need to define not just units, but measurement budgets: what counts as one token's worth of image? Is it the number of patches? The compute consumed per patch? The ratio between text and vision tokens? These are not settled questions. They are open research questions wearing the costume of a standardization problem.
Four: an audit and certification framework. This is what separates a standard from a suggestion. Without certification, without an audit trail, without either mandatory or de facto enforcement, a standard is a PDF with good intentions and no teeth.
Five: a governance structure. Who controls the reference implementation? Who handles disputes about vendor compliance? Who updates the conversion schemas? Who arbitrates when a vendor's new model shifts the tokenizer baseline? Standards bodies that answer these questions well achieve interoperability. Standards bodies that don't become walled gardens or dead letters. The lack of any disclosed governance from the Foundation is the most telling gap in its public material.
Liquidity is a mirage in high heat. That phrase applies to the structural liquidity of this entire market. The token economy looks liquid on the surface—you can buy and sell tokens, you can convert between providers, you can reallocate compute spend. But the liquidity is an illusion created by the pretense that tokens are fungible. They are not. When an enterprise tries to bank their AI budget on a specific cost-per-token estimate, the mirage evaporates. The costs shift. The token counts shift. The apparent comparability of vendors is a mirage sustained by the heat of the bull market in AI infrastructure.
The FinOps reality is instructive here. The market for AI cost management is real, and it is growing as fast as any AI infrastructure sector. These tools aggregate spend across providers, attempt to normalize costs, and provide dashboards. But every one of these tools suffers from the same disease: they normalize across vendors using vendor-reported token counts. Garbage in, garbage out. They cannot correct the underlying comparability problem because the unit is unstable.
In my institutional work, I have watched CFOs at AI-heavy companies struggle with this confusion. They sit in front of dashboards showing model call counts, token counts, and costs per token. The numbers are precise to two decimal places. The precision is entirely spurious. The underlying data is not comparable. A "cost per million tokens" figure from one provider is a different economic statement than the same figure from another provider. The CFO treats them as equivalent because they share a unit label. This is the fundamental accounting error of the AI era, and it is only going to get worse as token-based business models proliferate.
The Tokenomics Foundation's best-case scenario is to become the reference layer for exactly this ecosystem. Its standard would define how tools translate vendor-specific token counts into canonical amounts. It would define how invoices must report token usage. It would, ideally, allow enterprises to verify their bills against an independent measurement of the workloads they submitted.
That is a genuinely valuable function. It is the function of a Bureau of Standards for AI metering. The demand is real. Every enterprise adopting LLM APIs, every finance team wrestling with token line items, every cloud buyer asking for a chargeback structure that means anything, is a potential constituent.
And yet the market force is not necessarily enough to overcome the inertia I described. A standard is a coordination problem. The entities best positioned to create it—the model vendors—benefit from its absence. The entities that need it most—the buyers—are at the coordination-constrained player end of the market, with weak or no influence over the providers. For the standard to matter, it needs to be either mandated by regulation (unlikely anytime soon), so widely adopted by the third-party tooling ecosystem that it becomes de facto, or integrated into major cloud marketplaces as a procurement requirement.
That third vector is why the Foundation's commercial feasibility is non-trivial. Consider an enterprise AI procurement process. Purchasing managers want comparable, verified unit economics. They will demand a clause that vendor invoices report token usage in a canonical metric. If the Foundation can be adopted as the canonical definition in five or ten enterprise procurement templates, it becomes infrastructure. If the big FinOps tool vendors build the Foundation's reference implementation into their products, it becomes infrastructure. It can become the Linux of AI metering—not because of vendor generosity, but because the buyer ecosystem coalesced around it.
The competitive landscape, meanwhile, is far from empty. The FinOps Foundation already attempts to define cloud cost accounting frameworks. OpenTelemetry has GenAI semantic conventions defining observability fields. MLCommons benchmarks models with token-throughput evaluation envelopes. There is no single body with the authority to define the AI metering layer. There is no regulator. There is no sovereign body claiming jurisdiction over the token. The field is open for the taking—and it is also open for the dying.
Standard-setting organizations have a shocking failure rate. I have watched multiple standards bodies in the insurance and finance sectors die slow deaths over the past two decades. The pattern is always the same: the founding coalition announces itself with grand ambitions; the first technical meeting reveals the fundamental disagreement that the press release was designed to obscure; the founding coalition splits into factions, each claiming the standard is theirs; the standard fragments, or survives in name but not in practice because nobody adopts the reference implementation. What distinguishes the winners is the ability to build a deployment cohort. The standard does not win because it is correct. It wins because it is installed.
In the case of AI metering, installation means the reference implementation shipped into enterprises, into FinOps products, into procurement templates. The Foundation needs adoption. Without adoption, the standard is a narrative. In this market, narrative is a fleeting asset. Bubbles don't pop; they deflate slowly. The AI token bubble—for that is what the pricing structure amounts to—will not pop when someone publishes a whitepaper. It will deflate as buyers become smarter, as contracts become tighter, as comparisons become possible. The Foundation accelerates that deflation if it produces a usable artifact. It feeds the bubble if it produces only press releases.
Now let me return to something that bothered me from the first paragraph. The name. Tokenomics.
The word was coined in the crypto ecosystem to describe the economic design of blockchain tokens. It is a portmanteau of token and economics that came into vogue during the ICO era of 2017, when I was leading forensic audits of emission schedules for 14 mostly-worthless ICOs. The term is not AI-native. It was transplanted.
The Foundation's insistence on denying the crypto connection—repeated three times in a one-page announcement—tells me something fascinating. Whoever planted the name knew full well what it evoked. They chose it anyway. Then they wrote a press release managing the expected confusion. That sequence suggests either an opportunistic naming choice made by people who know the crypto association but don't want to deal with it, or a deliberate act of cultural borrowing from a space whose financialized token logic they admire.
The technical irony is that the crypto world has already attempted this exact standardization problem. In my audit work, I saw it again and again: crypto projects claiming to have standardized token accounting, creating beautiful models of token velocity, token utility, token emission schedules. None of it held up. The biggest lesson I carried into my CBDC research at the Abu Dhabi Global Centre was that token standardization is a coordination fiction. It works only when the participants have aligned incentives to maintain it. Without alignment, it becomes a repository of elegant definitions nobody follows.
I ran the Foundation's framing against my own AI-chain convergence thesis. For the past year, I have been developing a predictive model correlating AI compute demand on decentralized networks with global energy price cycles. My hypothesis is that AI-driven data verification will become the primary utility for Layer-1 blockchains post-ETF approval. The token measurement problem is central to that thesis, because the unit of measurement determines the economics of compute exchange. If AI tokens can be standardized across the industry, the compute market becomes more efficient. If they cannot, the market remains fragmented, and the fragmentation is itself valuable to certain players who profit from opacity.
So what would actual progress look like? If the Foundation were serious, its roadmap would be as follows. Publish the reference tokenizer specification. Open source it. Define the conversion schema. Publish the uncertainty model—there will be error; the question is how much. Define the multimodal measurement protocol. Publish reference test vectors that any auditor can run. Recruit model vendors as observers, not decision-makers, so the process does not become a vendor cartel. Get the FinOps tooling vendors to integrate. Get the cloud marketplaces to cite.
That is what distinguishes a standard from a suggestion. Right now, the Foundation is at the suggestion stage: a name, a press release, an existential denial. It could be the foundation of something valuable. What it is not yet is a standard.
The obvious narrative here is that Tokenomics Foundation is a fraud—a press release with a crypto-adjacent name, no technical substance, and a desperate denial of its own etymology. The contrarian angle is that this might be less fraud than disintermediation in disguise. The Foundation, despite its vocal crypto denial, may actually be a crypto-native play. Not on blockchains. On the economic DNA of the token. The term "tokenomics" was never about the blockchain itself. It was about the economic harness built around measured units of access and compute. In the AI era, the token is a measure of compute, a unit of access, and it has become the de facto pricing instrument. Whether a chain secures it is irrelevant. The tokenization logic is the same. The Foundation may be applying the lessons of crypto-economic design—standardizing units, making markets legible, enabling arbitrage—without wanting to inherit the stigma.
The second contrarian read is the opposite. The Foundation may be a decoy. If it draws its reason for existence from the fact that AI token measurement lacks a standard, it adds no value by actually standardizing. Its value, in this framing, comes from the market belief that standardization is near. This is the classic airport security theater play: the illusion of order, some number of contracts referencing the "standard" as a token of rigor, and no actual implementation behind it. For that kind of entity, the press release is the product. The denial is the brand. The lack of technical details is not a failure; it is the point. A specific technical proposal would be ripe for criticism. An ambiguous proposal is a blank canvas onto which buyers project their hope for a solution.
I find both readings compelling. And they are not mutually exclusive. The Foundation could begin as theater and become real if it receives enough engagement. Standards, like currencies, are sustained by the belief of those who settle contracts in them. The Foundation is currently not even at the stage of having a currency. It is in the pre-stage. It has minted a promise. Whether that promise is kept, with actual definitions and test vectors, will determine whether it is a settlement layer or a spoof.
I am not writing an obituary. I am writing a calibration check. The pain the Tokenomics Foundation claims to address—incomparable token pricing, unverifiable AI bills, the chaos of cost-per-token across opaque vendor systems—is real and intensifying. I have experienced it in my own institutional work. But intent is not an artifact. A standard is a traceable definition, a consensus verified by installed implementations. Standards committees produce failed outcomes more often than successful ones, and this one has yet to demonstrate even the first ingredient: a testable artifact.
Watch for three things. A public specification draft. A founding membership list that includes FinOps tooling vendors or cloud marketplaces, not just token-adjacent names. And proof of actual implementation in enterprise procurement, or licensing agreements from the major API vendors. If those arrive, the Foundation becomes something worth studying. If they do not, it will remain exactly what it is now: a name, a press release, a denial.
The AI industry is already measured in trillions of dollars of compute and model spend. Every quarter without a metering standard increases the administrative entropy of that spend. The Foundation is either the beginning of its resolution, or a brief interruption in the long-term drift toward an unstandardized market. Either way, the odds favor what I have observed in every token system I have audited: code is law, until the chain forks. Whoever defines the token we count, defines the money we pay. So watch the reference implementation. Everything else is marketing.