Diagrams — SDD-10: EVMbench

flowchart TB
    DATA[(the dataset<br/>117 vulns · 40 repos<br/>each: vuln code + fix + exploit + tests)]
    DATA --> DETECT
    DATA --> PATCH
    DATA --> EXPLOIT
    subgraph DETECT["Detect Mode — recall"]
        D1[harness receives vulnerable code]
        D1 --> D2[does it flag the vuln?]
        D2 --> D3[ground-truth match]
    end
    subgraph PATCH["Patch Mode — remediation"]
        P1[harness produces a fix]
        P1 --> P2[passes test suite + kills vuln?]
        P2 --> P3[test + regression gate]
    end
    subgraph EXPLOIT["Exploit Mode — adversarial depth"]
        E1[harness produces a tx sequence]
        E1 --> E2[foundry/hardhat runs it on forked chain]
        E2 --> E3[execution-verified: triggers vuln or not]
    end
    DETECT --> SCORE
    PATCH --> SCORE
    EXPLOIT --> SCORE
    subgraph SCORE["Three-Mode Scorecard"]
        S["X% Detect · Y% Patch · Z% Exploit<br/>optionally by vulnerability class<br/>comparable across harnesses"]
    end

    style DETECT fill:#0d1b2a,stroke:#5eead4,color:#5eead4
    style PATCH fill:#0d1b2a,stroke:#5eead4,color:#5eead4
    style EXPLOIT fill:#0d1b2a,stroke:#5eead4,color:#5eead4
    style DATA fill:#1a1015,stroke:#5eead4,color:#e4e4e8
    style SCORE fill:#1a1015,stroke:#5eead4,color:#e4e4e8

flowchart LR
    subgraph DATASET["EVMbench Dataset"]
        REPOS[40 real-world<br/>audited protocols]
        VULNS[117 vulnerabilities<br/>drawn from those repos]
        TAX[taxonomy:<br/>reentrancy · access control<br/>integer · oracle · flash-loan<br/>logic errors · ...]
        REPOS --> VULNS
        VULNS --> TAX
    end
    DATASET --> BENEFITS
    subgraph BENEFITS["Why this structure"]
        B1[real-world messiness<br/>inheritance · proxies · integrations]
        B2[repo diversity prevents<br/>over-fitting to one codebase style]
        B3[taxonomy reveals<br/>strengths/weaknesses by class]
        B4[117 = statistical signal<br/>affordable to run]
    end

    style DATASET fill:#0d1b2a,stroke:#5eead4,color:#5eead4
    style BENEFITS fill:#1a1015,stroke:#5eead4,color:#e4e4e8

flowchart TB
    subgraph PROFILE_A["Profile A: Finding-Flagger"]
        A1[90% Detect]
        A2[20% Patch]
        A3[20% Exploit]
        A1 & A2 & A3 --> A_OUT[use for triage, not remediation]
    end
    subgraph PROFILE_B["Profile B: Attack-Reconstructor"]
        B1[50% Detect]
        B2[30% Patch]
        B3[70% Exploit]
        B1 & B2 & B3 --> B_OUT[needs the vuln pointed at it; deep on confirmed targets]
    end
    subgraph PROFILE_C["Profile C: The Goal (rare)"]
        C1[high Detect]
        C2[high Patch]
        C3[high Exploit]
        C1 & C2 & C3 --> C_OUT[broad + deep + remediating]
    end

    style PROFILE_A fill:#0d1b2a,stroke:#5eead4,color:#5eead4
    style PROFILE_B fill:#0d1b2a,stroke:#5eead4,color:#5eead4
    style PROFILE_C fill:#0d1b2a,stroke:#5eead4,color:#5eead4

flowchart LR
    subgraph CURRENT["Current EVMbench Scorecard"]
        CU1[X% Detect — recall only]
        CU2[Y% Patch]
        CU3[Z% Exploit]
    end
    CURRENT --> ADD
    subgraph ADD["Axes to Add"]
        AD1[cost axis:<br/>X% Detect at what $/10K LOC?<br/>90% at $100 vs 90% at $2.31<br/>= different tools]
        AD2[precision/FP axis:<br/>run on fixed code, count flags<br/>recall without precision<br/>lets 'flag everything' score 100%]
        AD3[versioning:<br/>v1, v2, ... refresh<br/>resist gaming, track progress]
    end
    ADD --> DEPLOYMENT[deployment-decision benchmark]

    style CURRENT fill:#1a1015,stroke:#9494a0,color:#9494a0
    style ADD fill:#0d1b2a,stroke:#5eead4,color:#5eead4
    style DEPLOYMENT fill:#0d1b2a,stroke:#5eead4,color:#5eead4

# Diagrams — SDD-10: EVMbench

---

## Diagram 1 — The Three-Mode Evaluation (Detect / Patch / Exploit)

```mermaid
flowchart TB
    DATA[(the dataset<br/>117 vulns · 40 repos<br/>each: vuln code + fix + exploit + tests)]
    DATA --> DETECT
    DATA --> PATCH
    DATA --> EXPLOIT
    subgraph DETECT["Detect Mode — recall"]
        D1[harness receives vulnerable code]
        D1 --> D2[does it flag the vuln?]
        D2 --> D3[ground-truth match]
    end
    subgraph PATCH["Patch Mode — remediation"]
        P1[harness produces a fix]
        P1 --> P2[passes test suite + kills vuln?]
        P2 --> P3[test + regression gate]
    end
    subgraph EXPLOIT["Exploit Mode — adversarial depth"]
        E1[harness produces a tx sequence]
        E1 --> E2[foundry/hardhat runs it on forked chain]
        E2 --> E3[execution-verified: triggers vuln or not]
    end
    DETECT --> SCORE
    PATCH --> SCORE
    EXPLOIT --> SCORE
    subgraph SCORE["Three-Mode Scorecard"]
        S["X% Detect · Y% Patch · Z% Exploit<br/>optionally by vulnerability class<br/>comparable across harnesses"]
    end

    style DETECT fill:#0d1b2a,stroke:#5eead4,color:#5eead4
    style PATCH fill:#0d1b2a,stroke:#5eead4,color:#5eead4
    style EXPLOIT fill:#0d1b2a,stroke:#5eead4,color:#5eead4
    style DATA fill:#1a1015,stroke:#5eead4,color:#e4e4e8
    style SCORE fill:#1a1015,stroke:#5eead4,color:#e4e4e8
```

**Reading**: EVMbench's core design decision is the three-mode separation. Detect, Patch, and Exploit are three different questions about three different capabilities — and they are reported separately, never composited. A harness with 90% Detect and 20% Exploit is a finding-flagger; one with 50% Detect and 70% Exploit is an attack-reconstructor. A single composite score would hide this distinction; the three-mode scorecard forces every harness to report honestly where it is strong and where it is weak. The Exploit mode is execution-verified — the transaction sequence runs on a forked chain, so "produced an exploit" is not a self-reported claim.

---

## Diagram 2 — The Dataset Structure (Why 117 × 40)

```mermaid
flowchart LR
    subgraph DATASET["EVMbench Dataset"]
        REPOS[40 real-world<br/>audited protocols]
        VULNS[117 vulnerabilities<br/>drawn from those repos]
        TAX[taxonomy:<br/>reentrancy · access control<br/>integer · oracle · flash-loan<br/>logic errors · ...]
        REPOS --> VULNS
        VULNS --> TAX
    end
    DATASET --> BENEFITS
    subgraph BENEFITS["Why this structure"]
        B1[real-world messiness<br/>inheritance · proxies · integrations]
        B2[repo diversity prevents<br/>over-fitting to one codebase style]
        B3[taxonomy reveals<br/>strengths/weaknesses by class]
        B4[117 = statistical signal<br/>affordable to run]
    end

    style DATASET fill:#0d1b2a,stroke:#5eead4,color:#5eead4
    style BENEFITS fill:#1a1015,stroke:#5eead4,color:#e4e4e8
```

**Reading**: The dataset structure is a validity decision. The 40 real-world, audited protocols provide messiness — inheritance, proxy patterns, integrations — that synthetic CTF challenges lack, so a good score reflects real-world performance rather than over-fitting to clean patterns. The repo diversity prevents a harness from tuning to one codebase's style. The 117-vulnerability count is large enough for statistical signal and small enough to run affordably (an LLM-based harness at ~$2/10K LOC runs the whole set for tens of dollars). The taxonomy is the dimension along which a harness's profile becomes legible — "strong on reentrancy, weak on oracle manipulation" is more useful than a single recall number.

---

## Diagram 3 — Scorecard Interpretation (Three Independent Competencies)

```mermaid
flowchart TB
    subgraph PROFILE_A["Profile A: Finding-Flagger"]
        A1[90% Detect]
        A2[20% Patch]
        A3[20% Exploit]
        A1 & A2 & A3 --> A_OUT[use for triage, not remediation]
    end
    subgraph PROFILE_B["Profile B: Attack-Reconstructor"]
        B1[50% Detect]
        B2[30% Patch]
        B3[70% Exploit]
        B1 & B2 & B3 --> B_OUT[needs the vuln pointed at it; deep on confirmed targets]
    end
    subgraph PROFILE_C["Profile C: The Goal (rare)"]
        C1[high Detect]
        C2[high Patch]
        C3[high Exploit]
        C1 & C2 & C3 --> C_OUT[broad + deep + remediating]
    end

    style PROFILE_A fill:#0d1b2a,stroke:#5eead4,color:#5eead4
    style PROFILE_B fill:#0d1b2a,stroke:#5eead4,color:#5eead4
    style PROFILE_C fill:#0d1b2a,stroke:#5eead4,color:#5eead4
```

**Reading**: The three EVMbench scores are independent — a harness can be strong in one mode and weak in another, and the combination defines what the tool actually is. Profile A (high Detect, low Patch/Exploit) is a finding-flagger: good for triage, useless for remediation. Profile B (high Exploit, lower Detect) is an attack-reconstructor that needs the vulnerability pointed at it but goes deep on confirmed targets. Profile C (high across all three) is the goal and is rare. The three-mode scorecard is what lets you read which profile your harness occupies — and a composite score would have hidden all of this.

---

## Diagram 4 — The Build-On: Adding Cost and Precision Axes

```mermaid
flowchart LR
    subgraph CURRENT["Current EVMbench Scorecard"]
        CU1[X% Detect — recall only]
        CU2[Y% Patch]
        CU3[Z% Exploit]
    end
    CURRENT --> ADD
    subgraph ADD["Axes to Add"]
        AD1[cost axis:<br/>X% Detect at what $/10K LOC?<br/>90% at $100 vs 90% at $2.31<br/>= different tools]
        AD2[precision/FP axis:<br/>run on fixed code, count flags<br/>recall without precision<br/>lets 'flag everything' score 100%]
        AD3[versioning:<br/>v1, v2, ... refresh<br/>resist gaming, track progress]
    end
    ADD --> DEPLOYMENT[deployment-decision benchmark]

    style CURRENT fill:#1a1015,stroke:#9494a0,color:#9494a0
    style ADD fill:#0d1b2a,stroke:#5eead4,color:#5eead4
    style DEPLOYMENT fill:#0d1b2a,stroke:#5eead4,color:#5eead4
```

**Reading**: EVMbench's current scorecard measures correctness in three modes — which is what makes harnesses comparable. But correctness alone is not a deployment decision. Adding a cost axis (X% Detect at what price? — the axis Heimdallr reports natively) distinguishes a $100 harness from a $2.31 one at equal recall. Adding a precision axis (run on the fixed code; count the false flags) closes the recall-without-precision loophole that lets a "flag everything" harness score 100% Detect. And versioning (v1, v2, ...) resists the gaming that fixed benchmarks invite. With these three additions, EVMbench turns from a correctness benchmark into the deployment-decision tool a smart contract team actually needs.