SmolAgents opened 31.1% of the attack paths. CrewAI opened 11.9%. Same Claude Sonnet base model, same 32 payloads, same simulated tool environment. That is not a model-layer gap. That is a framework-layer execution gap, and it is exactly the number that should make every DeFi risk team stop pretending an AI agent is just a chat wrapper.
The SADF study — “The Orchestration Framework Is the Attack Surface,” presented at DEF CON 34’s AI Village — did not just find another prompt injection. It built a repeatable measurement for how much extra attack surface a framework adds on top of a competent model. Fixed base model. Direct API as control. Five targets: Direct API at 15.5% ACR, CrewAI at 11.9%, LangChain at 18.1%, AutoGen at 20.0%, SmolAgents at 31.1%. ACR, attack completion rate, is the share of adversarial payloads that actually achieved their goal. 5,119 evaluation rows, 32 payloads, eight failure-mode categories, all executed inside a simulated tool environment with no real credentials attached.
That design matters because DeFi has already started building on agent frameworks. Aave and Compound now see agent-operated wallets managing collateral. Liquidation bots are becoming agentic. Yield optimizers are routing through LLM-based rebalancers. Each of those layers trusts the orchestration framework to call tools in the right order. The SADF data says that trust is misplaced. In a Direct API world, an attacker must manipulate the model itself. In a framework world, an attacker has a second front: the code that parses tool inputs, stores memory, and delegates authority. That is a larger and softer target.
Most security products in crypto only scan smart contracts. They do not scan the agent’s tool-call graph. The graph is where the attack lives. The control structure also isolates a variable most security teams never audit: the router, the memory store, the tool-call parser, the delegation logic. In crypto terms, this is the difference between auditing the smart contract and auditing the wallet that signs the transaction. The contract can be perfect. The wallet can still be tricked. Based on my audit work on DeFi arbitrage bots, I have seen the same pattern again and again: the cleverest strategy is undone by the middleware that feeds it. The chart is just the echo; the code is the voice.
Most ACR numbers you see in vendor marketing are computed by a script that checks whether the attack string appears in the response. If the model says “I cannot help you with that” while quoting the attack, the script records a success. SADF’s refusal-filtered scoring fixes that. Claude Sonnet’s true ACR dropped to 15.5% after filtering; Claude Haiku’s to 22.3%. A metric that cannot distinguish a refusal from an execution is worthless. This is not a footnote. It means every agent benchmark not filtered for refusals may be inflated by an order of magnitude. I have seen the same error in crypto security reports where transaction simulation counted a reverted transaction as a successful exploit. Different field, same cost: false risk.
The failure-mode taxonomy maps directly to on-chain operations. Tool Call Hijacking is a malicious swap approval. Output Poisoning is a corrupted price feed being fed back into a decision loop. Cross-Tool Injection is one compromised oracle calling more sensitive instruments. Memory Poisoning plants false context in the agent’s state. RAG Poisoning corrupts the knowledge base that the agent trusts. Delegated Authority Abuse turns a sub-agent into a trojan. Multi-Agent Propagation spreads the blast radius across isolated strategies. Context Boundary Violation leaks one trade’s instructions into another. This taxonomy is not merely a list. It gives us the vocabulary to describe failure before it becomes a headline. A compromised governance vote might actually be RAG Poisoning if the agent reads a poisoned summary. A flash loan exploit might be Cross-Tool Injection if the tool chain passes malicious data from one operation to the next. In 2020, yield farming was the only shelter in the storm. Now the yield is being farmed by agents whose memory and tool-call boundaries are the storm.
Framework-level numbers confirm the taxonomy is doing real work. CrewAI’s discrete task isolation produced the best ACR, 11.9%, actually below the Direct API baseline. SmolAgents, with its permissive tool access and delegated sub-agents, produced the worst at 31.1%, including a unique 20% RAG Poisoning rate and a striking 64% Context Boundary Violation rate. In a SmolAgents deployment, nearly two out of every three malicious prompts broke out of their intended context window. For an autonomous trading agent, that is the difference between a hallucinated token recommendation and a liquidation cascade.
The commercial implications are straightforward. The framework layer is now a procurement variable. A 2.6x ACR gap between CrewAI and SmolAgents is the kind of number compliance teams can embed in an RFP. Security-evaluation-as-a-service becomes the obvious product: repeat SADF-style tests, package them as CI/CD checks, and convert an audit into a subscription. Real CVEs validate the trend. Azure SRE Agent (CVE-2026-62830) and Langflow (CVE-2026-9198) prove framework-level vulnerabilities are not theoretical. Code executes promises; men make excuses. Vendors will explain their scores. The market should demand the test file.
The insurance angle is even sharper. If agent frameworks become insurable, ACR data becomes pricing raw material. A DeFi protocol that runs agents could pay lower premium for audited framework configurations. That would push the entire industry toward measurable security. But none of that happens if the evaluation dataset stays behind closed doors. Open data is the missing catalyst.
My first reaction to ACR data was not “buy the safest framework.” It was “where is the dataset?” I need to rerun the evaluation with my own payloads, my own configuration, and my own models. Without the test harness, the conclusion is just a chart. I don’t trade off charts alone; I trade off audited mechanics. If SADF releases its dataset under a permissive license, the industry gains a baseline. If it does not, the commercial value is trapped inside a PDF.
Now the contrarian part. Read the ranking, then read the methods. Thirty-two payloads is not a representative sample of real adversarial operations. Rare but high-impact attack paths — a single privileged tool call that drains a vault — may be under-sampled. The eight failure modes are not equally weighted, and the study claims coverage of more architectures than it reports detailed ACR data for. The previous version of the research, which claimed ten coordinated disclosures, now sits in a SUPERSEDED folder. Transparency is good. Reusing the old numbers is not. This is still a proof of concept, not a certified standard. Treat it as a starting protocol, not an oracle.
The second blind spot is configuration. The study fixed the model but did not decompose temperature, system prompts, or tool-permission granularity. A framework with restrictive default permissions will have a different ACR than the same framework with permissive knobs turned up. A default-config test is a start, but production agents run with custom prompts and broad tool grants. The 11.9% vs 31.1% spread may be as much about defaults as about architecture. There is also the model×framework interaction effect. Would the ranking hold with GPT-5.4, DeepSeek, or Llama? The team fixed the model to isolate the framework. That is correct experimental hygiene, but in an industry where agents are composed like lego blocks, an interaction effect is not an edge case. It is the default condition.
All tests also ran in a simulated tool environment. Real permission boundaries, real tool-response timing, real multi-step workflows — those variables did not exist in the sandbox. A framework that passes the simulation may still bleed in production. On-chain eyes saw the mania before the crowd did. This dataset is the blockchain equivalent of reading the wallet, not the headline.
Here is the actionable version. If you run autonomous trading agents, put the orchestration framework on the same due-diligence list as the model weights. Demand ACR-style evaluation in simulated tool environments before mainnet deployment. Treat tool-call routers as privileged infrastructure. Watch for security patches from CrewAI, LangChain, AutoGen, and SmolAgents. When a vendor says “our model is safer,” ask for the framework’s score, not the model’s leaderboard. In a bear market, survival matters more than gains. Survival isn’t about being right; it’s about staying solvent. The next exploit may not live in the smart contract. It may already be in the tool-call router between your strategy and your capital. Is your portfolio ready for that audit?
