DSL Reference

Name: poly-bench
Author: poly-bench

Complete syntax reference for poly-bench .bench files

Docs in progress. Some content or features may be missing or not covered in detail.

The poly-bench DSL is a declarative language for defining cross-language benchmarks. This reference covers the complete syntax for .bench files — every block type, field, option, and value form that the parser accepts.

File Structure

A .bench file consists of optional standard library imports, an optional file-level globalSetup block, and one or more suite blocks.

1# Optional standard library imports
2use std::charting
3use std::anvil
4
5# Optional file-level global setup
6globalSetup {
7  anvil.spawnAnvil(fork: "https://eth.llamarpc.com")
8}
9
10# One or more suites (declare suite is required)
11declare suite mySuite performance timeBased sameDataset: true {
12  # Suite configuration, setup blocks, fixtures, benchmarks
13}

Comments

Comments begin with # and extend to the end of the line. They can appear anywhere in the file.

1# This is a full-line comment
2
3declare suite example performance timeBased sameDataset: true {
4  warmup: 100  # Inline comment after a value
5
6  bench myBench {
7      go: doSomething()  # After expressions too
8  }
9}

Standard Library Imports

Import standard library modules at the top of your file before any other blocks. Multiple imports are allowed.

1use std::charting   # Chart generation
2use std::anvil      # Ethereum node spawning
3use std::constants  # Mathematical constants

See the Standard Library reference for module details.

`globalSetup` Block

The globalSetup block runs once before all benchmarks in the file. It is the correct place to spawn long-lived services like an Anvil node. It can appear at the file level (before any suite) or inside a suite block.

1use std::anvil
2
3globalSetup {
4  anvil.spawnAnvil()                           # Local node, no fork
5  anvil.spawnAnvil(fork: "https://rpc.url")    # Fork from an RPC endpoint
6}

When anvil.spawnAnvil() is called, the variable ANVIL_RPC_URL is automatically injected into TypeScript setup code as a module-level constant. Go and Rust access it via the environment variable ANVIL_RPC_URL.

`suite` Block

A suite groups related benchmarks with shared configuration, setup code, fixtures, and lifecycle hooks.

1declare suite <name> <suiteType> <runMode> sameDataset: <true|false> {
2  # ── Metadata ──────────────────────────────────────
3  description: "Human-readable description"
4
5  # ── Iteration control ─────────────────────────────
6  iterations: 100000        # Fixed iteration count (used when runMode: iterationBased)
7
8  warmup: 1000              # Warmup iterations before timing starts
9  targetTime: 3000ms        # Target wall-clock time for runMode: timeBased
10  minIterations: 10         # Floor for auto-calibrated iteration count
11  maxIterations: 1000000    # Ceiling for auto-calibrated iteration count
12
13  # ── Statistical controls ──────────────────────────
14  count: 3                  # Number of timed runs per benchmark
15  cvThreshold: 5.0          # Target coefficient of variation (%)
16  outlierDetection: true    # IQR-based outlier removal
17
18  # ── Comparison ────────────────────────────────────
19  compare: true             # Show comparison table in output
20  baseline: "go"            # Language for speedup ratios ("go", "ts", "rust", "python", "c", "csharp", "zig")
21
22  # ── Observability ─────────────────────────────────
23  concurrency: 1            # Parallel workers per benchmark
24  sink: true                # Black-box sink to prevent dead-code elimination
25
26  # ── Execution order ───────────────────────────────
27  order: sequential         # sequential | parallel | random
28  timeout: 60000ms          # Suite-level timeout
29
30  # ── Language requirements ─────────────────────────
31  requires: ["go", "ts"]    # All benchmarks must implement these languages
32
33  # ── Child blocks ──────────────────────────────────
34  globalSetup { ... }
35  setup go { ... }
36  setup ts { ... }
37  setup rust { ... }
38  setup python { ... }
39  setup c { ... }
40  setup csharp { ... }
41  setup zig { ... }
42  fixture myData { ... }
43  bench myBench { ... }
44  after { ... }
45}

Suite Configuration Reference

Field	Type	Default	Description
`description`	string	—	Human-readable description of the suite
`suiteType`	header token	—	Required in suite declaration: `memory` (enables memory tracking) or `performance`
`runMode`	header token	—	Required in suite declaration: `timeBased` or `iterationBased`
`sameDataset`	header boolean	—	Required in suite declaration: `sameDataset: true
`iterations`	number	—	Fixed iteration count; used when runMode is `iterationBased`
`warmup`	number	`1000`	Warmup iterations before timing
`targetTime`	duration	`3000ms`	Target run time for runMode `timeBased`
`minIterations`	number	`10`	Minimum iterations in auto mode
`maxIterations`	number	`1000000`	Maximum iterations in auto mode
`count`	number	`1`	Number of timed runs per benchmark
`cvThreshold`	number	`5.0`	Target coefficient of variation (%)
`outlierDetection`	boolean	`true`	IQR-based outlier removal
`compare`	boolean	`false`	Show cross-language comparison table
`baseline`	string	—	Language for speedup ratios (`"go"`, `"ts"`, `"rust"`, `"python"`, `"c"`, `"csharp"`, `"zig"`)
`concurrency`	number	`1`	Parallel workers per benchmark
`sink`	boolean	`true`	Prevent dead-code elimination via black-box sink
`order`	identifier	`sequential`	`sequential`, `parallel`, or `random`
`timeout`	duration	—	Suite-level timeout
`requires`	string[]	`[]`	Languages every benchmark in the suite must implement

Use timeBased in the suite declaration for most suites. It calibrates runtime via targetTime and avoids per-benchmark mode drift.

The property mode is deprecated and rejected. Use declare suite with runMode (timeBased or iterationBased) in the header instead.

Duration Syntax

Duration values can be specified with a unit suffix. Plain numbers are treated as milliseconds.

Unit	Example	Meaning
`ms`	`3000ms`	3000 milliseconds
`s`	`5s`	5 seconds (5000 ms)
`m`	`1m`	1 minute (60000 ms)

1declare suite timing performance timeBased sameDataset: true {
2  targetTime: 3000ms   # 3 seconds
3  targetTime: 3s       # same as above
4  timeout: 1m          # 1 minute
5  timeout: 60000ms     # same as above
6}

`setup <lang>` Blocks

Setup blocks define language-specific imports, package-level declarations, one-time initialization, and helper functions. Supported language keywords: go, ts (or typescript), rust, python, c, csharp, zig.

All four sub-sections are optional and can appear in any order.

1setup go {
2  # Grouped import block (Go syntax)
3  import (
4      "context"
5      "crypto/sha256"
6      "github.com/ethereum/go-ethereum/ethclient"
7  )
8
9  # Package-level variable/type/const declarations
10  declare {
11      var client *ethclient.Client
12      var ctx context.Context
13      var mu sync.Mutex
14  }
15
16  # Runs once before any benchmarks
17  init {
18      ctx = context.Background()
19      client, _ = ethclient.Dial("https://eth.llamarpc.com")
20  }
21
22  # Helper functions available to all bench blocks
23  helpers {
24      func hashData(data []byte) [32]byte {
25          return sha256.Sum256(data)
26      }
27  }
28}

Sub-section Reference

Sub-section	Go syntax	TS syntax	Rust syntax	Description
`import`	`import ( "pkg" )`	`import { import ... }`	`import { use ...; }`	Language imports
`declare`	`declare { var x T }`	`declare { let x: T }`	`declare { static X: T }`	Package/module-level declarations
`init`	`init { ... }`	`init { ... }` or `async init { ... }`	`init { ... }`	One-time setup before benchmarks
`helpers`	`helpers { func f() {} }`	`helpers { function f() {} }`	`helpers { fn f() {} }`	Helper functions for bench blocks

TypeScript supports async init for asynchronous one-time setup. The await keyword can be used freely inside it. Regular init also works for synchronous setup.

`fixture` Blocks

Fixtures define shared test data that is passed to benchmark implementations. They ensure all languages operate on identical input.

Inline hex fixture

Provide binary data as a hex-encoded string. The bytes are decoded and passed to each language as its native byte array type.

1fixture shortData {
2  hex: "68656c6c6f20776f726c64"   # "hello world" in UTF-8
3}
4
5# In bench blocks:
6# Go:   shortData  →  []byte{0x68, 0x65, 0x6c, ...}
7# TS:   shortData  →  Uint8Array([0x68, 0x65, 0x6c, ...])
8# Rust: shortData  →  &[u8]

File-referenced hex fixture

Load hex data from an external file using @file(...). The path is relative to the .bench file's location.

1fixture largePayload {
2  hex: @file("fixtures/sort/sort_1000.hex")
3}
4
5fixture abiData {
6  hex: @file("fixtures/abi/erc20_calldata.hex")
7}

Use inline hex: for small test vectors and hex: @file(...) for larger payloads. This keeps .bench files readable while supporting arbitrarily large fixtures.

Generic `data` source with encoding

Use data for non-hex external samples. encoding controls byte decoding.

1fixture packetRaw {
2  data: @file("fixtures/net/packet.bin")
3  encoding: raw
4}
5
6fixture payloadUtf8 {
7  data: "hello world"
8  encoding: utf8
9}
10
11fixture payloadB64 {
12  data: "aGVsbG8gd29ybGQ="
13  encoding: base64
14}

Structured extraction (`format` + `selector`)

Structured inputs are normalized at compile time to deterministic bytes.

1fixture requestId {
2  data: @file("fixtures/requests.json")
3  format: json
4  selector: "$.items[0].id"
5}
6
7fixture csvCell {
8  data: @file("fixtures/table.csv")
9  format: csv
10  selector: "1,2"    # row,col (0-indexed)
11}

Fixture with `shape` annotation

The optional shape field provides a JSON-like descriptor for documentation and tooling purposes.

1fixture matrixData {
2  hex: @file("fixtures/matmul/mat_64.hex")
3  shape: { rows: 64, cols: 64, dtype: "float64" }
4}

Fixture with `description`

1fixture keccakInput {
2  description: "32-byte input for keccak256 hashing"
3  hex: "68656c6c6f20776f726c6468656c6c6f20776f726c6468656c6c6f20776f72"
4}

`bench` and `benchAsync` Blocks

bench defines synchronous benchmark operations. benchAsync defines async operations with async-sequential semantics (one awaited completion per iteration, no framework-managed concurrency).

1bench <name> {
2  # ── Metadata ──────────────────────────────────────
3  description: "What this benchmark measures"
4  tags: ["crypto", "hashing"]
5
6  # ── Iteration overrides (inherit from suite) ──────
7  iterations: 100000
8  warmup: 1000
9  mode: "fixed"
10  targetTime: 2000ms
11  minIterations: 5000
12  maxIterations: 500000
13  timeout: 30000ms
14
15  # ── Statistical overrides (inherit from suite) ────
16  count: 5
17  cvThreshold: 5.0
18  outlierDetection: true
19
20  # ── Observability overrides (inherit from suite) ──
21  concurrency: 1
22  sink: true
23
24  # ── Lifecycle hooks ───────────────────────────────
25  before go: resetCounter()
26  before ts: resetCounter()
27  each go: incrementCounter()
28  each ts: incrementCounter()
29
30  # ── Language implementations ──────────────────────
31  go: hashData(data)
32  ts: hashData(data)
33  rust: hash_data(&data)
34
35  # ── Post-run hooks ────────────────────────────────
36  after go: { _ = setupCounter }
37  after ts: { void setupCounter }
38
39  # ── Conditional execution ─────────────────────────
40  skip: { go: false  ts: false }
41
42  # ── Result validation ─────────────────────────────
43  validate: { go: result != nil  ts: result !== null }
44}
45
46benchAsync <name> {
47  description: "Async benchmark"
48  mode: "auto"
49  targetTime: 5000ms
50  ts: await getBlockNumber()
51}

benchAsync applies internal caps for practical runtime behavior: warmup ≤ 5 and samples ≤ 50. Async implementations should fail fast on operation errors (throw/panic) rather than returning error objects as normal values, so error metrics stay accurate.

Bench Field Reference

Field	Type	Default	Description
`description`	string	—	Human-readable description
`tags`	string[]	`[]`	Tags for filtering benchmarks
`iterations`	number	suite value	Fixed iteration count
`warmup`	number	suite value	Warmup iterations
`mode`	string	suite value	`"auto"` or `"fixed"`
`targetTime`	duration	suite value	Target time for auto mode
`minIterations`	number	suite value	Minimum iterations in auto mode
`maxIterations`	number	suite value	Maximum iterations in auto mode
`timeout`	duration	suite value	Per-benchmark timeout
`count`	number	suite value	Number of timed runs
`cvThreshold`	number	suite value	Target coefficient of variation (%)
`outlierDetection`	boolean	suite value	IQR-based outlier removal
`concurrency`	number	suite value	Parallel workers
`sink`	boolean	suite value	Black-box sink
`skip`	per-lang bool	—	Skip this benchmark for specific languages
`validate`	per-lang expr	—	Validate the return value

All fields except the language implementations are optional. Numeric/boolean fields inherit from the suite when not specified.

Inline expression vs block form

Language implementations can be a single inline expression or a multi-line block:

1bench hashShort {
2  go: sha256Sum(data)
3  ts: sha256Sum(data)
4  rust: sha256_sum(&data)
5}

Partial language coverage

You can define benchmarks for a subset of languages. Only languages with a setup block and an implementation line are executed.

1setup go { ... }
2setup ts { ... }
3# No setup rust block
4
5bench operation {
6  go: doSomething()
7  ts: doSomething()
8  # No rust: line — Rust is skipped entirely
9}

Lifecycle Hooks

Hooks run at specific points in the benchmark lifecycle. They are defined per-language on individual bench blocks, not at the suite level.

`before <lang>`

Runs once before the timed loop starts for that language. Use it for per-benchmark setup that should not be included in the timing.

`each <lang>`

Runs before each individual iteration, outside the timing window. Use it to reset mutable state between iterations.

`after <lang>`

Runs once after the timed loop ends for that language. Use it for per-benchmark teardown.

Both flat syntax and grouped syntax are supported and equivalent:

1bench withHooks {
2  iterations: 10000
3  mode: "fixed"
4
5  before go: resetCounter()
6  before ts: resetCounter()
7
8  each go: incrementCounter()
9  each ts: incrementCounter()
10
11  go: sha256Sum(data)
12  ts: sha256Sum(data)
13
14  after go: {
15      _ = setupCounter
16  }
17  after ts: {
18      void setupCounter
19  }
20}

Hook bodies can be either an inline expression (e.g. before go: resetState()) or a multi-line block (e.g. after go: { _ = x }). The flat syntax is generally preferred for readability.

`after { }` Block (Suite-level)

The suite-level after block runs after all benchmarks complete. It is the correct place for chart generation directives. It requires use std::charting at the top of the file.

1use std::charting
2
3suite example {
4  # ... benchmarks ...
5
6  after {
7
8      charting.drawTable(
9          title: "Performance Comparison",
10          output: "results-bar.svg",
11          sortBy: "speedup",
12          sortOrder: "desc"
13      )
14
15
16      charting.drawSpeedupChart(
17          title: "Speedup vs Go Baseline",
18          output: "results-speedup.svg",
19          baselineBenchmark: "hashShort",
20          sortBy: "speedup"
21      )
22  }
23}

See std::charting for the full parameter reference for each chart function.

Value Types

Type	Example	Used by
String	`"hello world"`	`description`, `baseline`, `mode`, `title`, etc.
Integer	`5000`, `1000000`	`iterations`, `warmup`, `count`, `width`, `height`
Float	`5.0`, `2.5`	`cvThreshold`, `minSpeedup`, `gridOpacity`
Duration	`500ms`, `3s`, `2m`	`targetTime`, `timeout`
Boolean	`true` / `false`	`compare`, `sink`, `outlierDetection`
String array	`["go", "ts"]`	`requires`, `tags`, `includeBenchmarks`
Identifier	`sequential`	`order`
File reference	`@file("path/to/file.hex")`	`hex` field in fixtures
Code block	`{ ... }`	setup sub-sections, hook bodies, bench implementations
Inline expression	`sortGo(data)`	single-line bench/hook implementations

Complete Example

A complete .bench file demonstrating all major features:

1use std::charting
2
3declare suite keccakBenchmarks performance timeBased sameDataset: false {
4  description: "Keccak256 hashing across Go, TypeScript, and Rust"
5  warmup: 1000
6  compare: true
7  baseline: "go"
8  targetTime: 3000ms
9  minIterations: 10
10  maxIterations: 1000000
11  count: 3
12  cvThreshold: 5.0
13  outlierDetection: true
14  sink: true
15
16  setup go {
17      import (
18          "golang.org/x/crypto/sha3"
19      )
20
21      declare {
22          var runCount int
23      }
24
25      init {
26          runCount = 0
27      }
28
29      helpers {
30          func keccak256Go(data []byte) []byte {
31              h := sha3.NewLegacyKeccak256()
32              h.Write(data)
33              return h.Sum(nil)
34          }
35
36          func resetCount() { runCount = 0 }
37          func bumpCount()  { runCount++ }
38      }
39  }
40
41  setup ts {
42      import {
43          import { keccak256 } from 'viem'
44      }
45
46      declare {
47          let runCount = 0
48      }
49
50      helpers {
51          function keccak256Ts(data: Uint8Array): Uint8Array {
52              return keccak256(data, 'bytes')
53          }
54
55          function resetCount(): void { runCount = 0 }
56          function bumpCount(): void  { runCount++ }
57      }
58  }
59
60  setup rust {
61      import {
62          use tiny_keccak::{Hasher, Keccak};
63      }
64
65      helpers {
66          fn keccak256_rust(data: &[u8]) -> [u8; 32] {
67              let mut hasher = Keccak::v256();
68              let mut output = [0u8; 32];
69              hasher.update(data);
70              hasher.finalize(&mut output);
71              output
72          }
73      }
74  }
75
76  # Inline hex fixture
77  fixture shortInput {
78      description: "32-byte input"
79      hex: "68656c6c6f20776f726c6468656c6c6f20776f726c6468656c6c6f20776f72"
80  }
81
82  # File-referenced fixture for larger data
83  fixture longInput {
84      hex: @file("fixtures/keccak/input_1024.hex")
85  }
86
87  # Basic benchmark — all three languages
88  bench hashShort {
89      description: "Hash a 32-byte input"
90      go: keccak256Go(shortInput)
91      ts: keccak256Ts(shortInput)
92      rust: keccak256_rust(&shortInput)
93  }
94
95  # Benchmark with per-bench overrides
96  bench hashLong {
97      description: "Hash a 1024-byte input"
98      targetTime: 5000ms
99      count: 5
100      go: keccak256Go(longInput)
101      ts: keccak256Ts(longInput)
102      rust: keccak256_rust(&longInput)
103  }
104
105  # Benchmark with lifecycle hooks
106  bench hashWithHooks {
107      description: "Hash with counter tracking via hooks"
108      iterations: 10000
109      mode: "fixed"
110      before go: resetCount()
111      before ts: resetCount()
112      each go: bumpCount()
113      each ts: bumpCount()
114      go: keccak256Go(shortInput)
115      ts: keccak256Ts(shortInput)
116      after go: { _ = runCount }
117      after ts: { void runCount }
118  }
119
120  # Go-only benchmark (no ts/rust implementation)
121  bench goOnly {
122      description: "Go-specific operation"
123      go: keccak256Go(shortInput)
124  }
125
126  after {
127      charting.drawTable(
128          title: "Keccak256 Performance",
129          description: "Go vs TypeScript vs Rust",
130          output: "keccak-bar.svg",
131          sortBy: "speedup",
132          sortOrder: "desc",
133          showStats: true,
134          showMemory: true
135      )
136
137      charting.drawSpeedupChart(
138          title: "Keccak256 Scaling",
139          output: "keccak-line.svg",
140          xlabel: "Input Size"
141      )
142  }
143}

Validation & Lint Rules

The DSL validator and LSP emit diagnostics for semantic issues that affect fairness and comparison quality. The following rules are enforced at parse/check time and in the editor.

`same-dataset-inconsistent-fixtures` (Warning)

When sameDataset: true, all benchmarks should operate on the same dataset. If fixture references differ across benchmarks (e.g., one uses data1, another uses data2), a warning is emitted. This heuristic helps catch accidental unfair comparisons.

1declare suite sort performance timeBased sameDataset: true {
2  fixture s100 { hex: @file("sort_100.hex") }
3  fixture s200 { hex: @file("sort_200.hex") }
4  bench n100 { go: sort(s100)  ts: sort(s100) }
5  bench n200 { go: sort(s200)  ts: sort(s200) }
6}

`chart-requires-multiple-benchmarks` (Error)

drawLineChart and drawBarChart compare benchmarks across the suite. A single benchmark provides no meaningful trend or comparison. At least 2 benchmarks are required.

1use std::charting
2declare suite trend performance timeBased sameDataset: true {
3  bench n100 { go: work(s100)  ts: work(s100) }
4  bench n200 { go: work(s200)  ts: work(s200) }
5  after { charting.drawLineChart(title: "Scaling") }
6}

`baseline-missing-in-benchmark` (Error)

When baseline: "go" (or another language) is set, every benchmark must implement that language. Missing baseline implementations break comparative outputs and speedup ratios.

1declare suite compare performance timeBased sameDataset: true {
2  baseline: "go"
3  bench foo { go: work()  ts: work() }
4  bench bar { go: work()  ts: work() }
5}

Features — Per-feature reference (suites, setup, benchmarks, fixtures, globalSetup)
Standard Library — Module reference for std::anvil, std::charting, std::constants
Features — Fixtures, hooks, globalSetup, charting reference
CLI Reference — Command-line options
Architecture — How the DSL compiles to native code

1# Optional standard library imports 2use std::charting 3use std::anvil 4 5# Optional file-level global setup 6globalSetup { 7 anvil.spawnAnvil(fork: "https://eth.llamarpc.com") 8} 9 10# One or more suites (declare suite is required) 11declare suite mySuite performance timeBased sameDataset: true { 12 # Suite configuration, setup blocks, fixtures, benchmarks 13}

1# This is a full-line comment 2 3declare suite example performance timeBased sameDataset: true { 4 warmup: 100 # Inline comment after a value 5 6 bench myBench { 7 go: doSomething() # After expressions too 8 } 9}

1declare suite <name> <suiteType> <runMode> sameDataset: <true|false> { 2 # ── Metadata ────────────────────────────────────── 3 description: "Human-readable description" 4 5 # ── Iteration control ───────────────────────────── 6 iterations: 100000 # Fixed iteration count (used when runMode: iterationBased) 7 8 warmup: 1000 # Warmup iterations before timing starts 9 targetTime: 3000ms # Target wall-clock time for runMode: timeBased 10 minIterations: 10 # Floor for auto-calibrated iteration count 11 maxIterations: 1000000 # Ceiling for auto-calibrated iteration count 12 13 # ── Statistical controls ────────────────────────── 14 count: 3 # Number of timed runs per benchmark 15 cvThreshold: 5.0 # Target coefficient of variation (%) 16 outlierDetection: true # IQR-based outlier removal 17 18 # ── Comparison ──────────────────────────────────── 19 compare: true # Show comparison table in output 20 baseline: "go" # Language for speedup ratios ("go", "ts", "rust", "python", "c", "csharp", "zig") 21 22 # ── Observability ───────────────────────────────── 23 concurrency: 1 # Parallel workers per benchmark 24 sink: true # Black-box sink to prevent dead-code elimination 25 26 # ── Execution order ─────────────────────────────── 27 order: sequential # sequential | parallel | random 28 timeout: 60000ms # Suite-level timeout 29 30 # ── Language requirements ───────────────────────── 31 requires: ["go", "ts"] # All benchmarks must implement these languages 32 33 # ── Child blocks ────────────────────────────────── 34 globalSetup { ... } 35 setup go { ... } 36 setup ts { ... } 37 setup rust { ... } 38 setup python { ... } 39 setup c { ... } 40 setup csharp { ... } 41 setup zig { ... } 42 fixture myData { ... } 43 bench myBench { ... } 44 after { ... } 45}

Field

Type

Default

Description

description

string

—

Human-readable description of the suite

suiteType

header token

—

Required in suite declaration: memory (enables memory tracking) or performance

runMode

header token

—

Required in suite declaration: timeBased or iterationBased

sameDataset

header boolean

—

Required in suite declaration: `sameDataset: true

iterations

number

—

Fixed iteration count; used when runMode is iterationBased

warmup

number

1000

Warmup iterations before timing

targetTime

duration

3000ms

Target run time for runMode timeBased

minIterations

number

10

Minimum iterations in auto mode

maxIterations

number

1000000

Maximum iterations in auto mode

count

number

1

Number of timed runs per benchmark

cvThreshold

number

5.0

Target coefficient of variation (%)

outlierDetection

boolean

true

IQR-based outlier removal

compare

boolean

false

Show cross-language comparison table

baseline

string

—

Language for speedup ratios ("go", "ts", "rust", "python", "c", "csharp", "zig")

concurrency

number

1

Parallel workers per benchmark

sink

boolean

true

Prevent dead-code elimination via black-box sink

order

identifier

sequential

sequential, parallel, or random

timeout

duration

—

Suite-level timeout

requires

string[]

[]

Languages every benchmark in the suite must implement

Unit

Example

Meaning

ms

3000ms

3000 milliseconds

s

5s

5 seconds (5000 ms)

m

1m

1 minute (60000 ms)

1declare suite timing performance timeBased sameDataset: true { 2 targetTime: 3000ms # 3 seconds 3 targetTime: 3s # same as above 4 timeout: 1m # 1 minute 5 timeout: 60000ms # same as above 6}

1setup go { 2 # Grouped import block (Go syntax) 3 import ( 4 "context" 5 "crypto/sha256" 6 "github.com/ethereum/go-ethereum/ethclient" 7 ) 8 9 # Package-level variable/type/const declarations 10 declare { 11 var client *ethclient.Client 12 var ctx context.Context 13 var mu sync.Mutex 14 } 15 16 # Runs once before any benchmarks 17 init { 18 ctx = context.Background() 19 client, _ = ethclient.Dial("https://eth.llamarpc.com") 20 } 21 22 # Helper functions available to all bench blocks 23 helpers { 24 func hashData(data []byte) [32]byte { 25 return sha256.Sum256(data) 26 } 27 } 28}

Sub-section

Go syntax

TS syntax

Rust syntax

Description

import

import ( "pkg" )

import { import ... }

import { use ...; }

Language imports

declare

declare { var x T }

declare { let x: T }

declare { static X: T }

Package/module-level declarations

init

init { ... }

init { ... } or async init { ... }

init { ... }

One-time setup before benchmarks

helpers

helpers { func f() {} }

helpers { function f() {} }

helpers { fn f() {} }

Helper functions for bench blocks

1fixture shortData { 2 hex: "68656c6c6f20776f726c64" # "hello world" in UTF-8 3} 4 5# In bench blocks: 6# Go: shortData → []byte{0x68, 0x65, 0x6c, ...} 7# TS: shortData → Uint8Array([0x68, 0x65, 0x6c, ...]) 8# Rust: shortData → &[u8]

1fixture packetRaw { 2 data: @file("fixtures/net/packet.bin") 3 encoding: raw 4} 5 6fixture payloadUtf8 { 7 data: "hello world" 8 encoding: utf8 9} 10 11fixture payloadB64 { 12 data: "aGVsbG8gd29ybGQ=" 13 encoding: base64 14}

1fixture requestId { 2 data: @file("fixtures/requests.json") 3 format: json 4 selector: "$.items[0].id" 5} 6 7fixture csvCell { 8 data: @file("fixtures/table.csv") 9 format: csv 10 selector: "1,2" # row,col (0-indexed) 11}

1bench <name> { 2 # ── Metadata ────────────────────────────────────── 3 description: "What this benchmark measures" 4 tags: ["crypto", "hashing"] 5 6 # ── Iteration overrides (inherit from suite) ────── 7 iterations: 100000 8 warmup: 1000 9 mode: "fixed" 10 targetTime: 2000ms 11 minIterations: 5000 12 maxIterations: 500000 13 timeout: 30000ms 14 15 # ── Statistical overrides (inherit from suite) ──── 16 count: 5 17 cvThreshold: 5.0 18 outlierDetection: true 19 20 # ── Observability overrides (inherit from suite) ── 21 concurrency: 1 22 sink: true 23 24 # ── Lifecycle hooks ─────────────────────────────── 25 before go: resetCounter() 26 before ts: resetCounter() 27 each go: incrementCounter() 28 each ts: incrementCounter() 29 30 # ── Language implementations ────────────────────── 31 go: hashData(data) 32 ts: hashData(data) 33 rust: hash_data(&data) 34 35 # ── Post-run hooks ──────────────────────────────── 36 after go: { _ = setupCounter } 37 after ts: { void setupCounter } 38 39 # ── Conditional execution ───────────────────────── 40 skip: { go: false ts: false } 41 42 # ── Result validation ───────────────────────────── 43 validate: { go: result != nil ts: result !== null } 44} 45 46benchAsync <name> { 47 description: "Async benchmark" 48 mode: "auto" 49 targetTime: 5000ms 50 ts: await getBlockNumber() 51}

Field

Type

Default

Description

description

string

—

Human-readable description

tags

string[]

[]

Tags for filtering benchmarks

iterations

number

suite value

Fixed iteration count

warmup

number

suite value

Warmup iterations

mode

string

suite value

"auto" or "fixed"

targetTime

duration

suite value

Target time for auto mode

minIterations

number

suite value

Minimum iterations in auto mode

maxIterations

number

suite value

Maximum iterations in auto mode

timeout

duration

suite value

Per-benchmark timeout

count

number

suite value

Number of timed runs

cvThreshold

number

suite value

Target coefficient of variation (%)

outlierDetection

boolean

suite value

IQR-based outlier removal

concurrency

number

suite value

Parallel workers

sink

boolean

suite value

Black-box sink

skip

per-lang bool

—

Skip this benchmark for specific languages

validate

per-lang expr

—

Validate the return value

1bench withHooks { 2 iterations: 10000 3 mode: "fixed" 4 5 before go: resetCounter() 6 before ts: resetCounter() 7 8 each go: incrementCounter() 9 each ts: incrementCounter() 10 11 go: sha256Sum(data) 12 ts: sha256Sum(data) 13 14 after go: { 15 _ = setupCounter 16 } 17 after ts: { 18 void setupCounter 19 } 20}

1use std::charting 2 3suite example { 4 # ... benchmarks ... 5 6 after { 7 8 charting.drawTable( 9 title: "Performance Comparison", 10 output: "results-bar.svg", 11 sortBy: "speedup", 12 sortOrder: "desc" 13 ) 14 15 16 charting.drawSpeedupChart( 17 title: "Speedup vs Go Baseline", 18 output: "results-speedup.svg", 19 baselineBenchmark: "hashShort", 20 sortBy: "speedup" 21 ) 22 } 23}

Type

Example

Used by

String

"hello world"

description, baseline, mode, title, etc.

Integer

5000, 1000000

iterations, warmup, count, width, height

Float

5.0, 2.5

cvThreshold, minSpeedup, gridOpacity

Duration

500ms, 3s, 2m

targetTime, timeout

Boolean

true / false

compare, sink, outlierDetection

String array

["go", "ts"]

requires, tags, includeBenchmarks

Identifier

sequential

order

File reference

@file("path/to/file.hex")

hex field in fixtures

Code block

{ ... }

setup sub-sections, hook bodies, bench implementations

Inline expression

sortGo(data)

single-line bench/hook implementations

1use std::charting 2 3declare suite keccakBenchmarks performance timeBased sameDataset: false { 4 description: "Keccak256 hashing across Go, TypeScript, and Rust" 5 warmup: 1000 6 compare: true 7 baseline: "go" 8 targetTime: 3000ms 9 minIterations: 10 10 maxIterations: 1000000 11 count: 3 12 cvThreshold: 5.0 13 outlierDetection: true 14 sink: true 15 16 setup go { 17 import ( 18 "golang.org/x/crypto/sha3" 19 ) 20 21 declare { 22 var runCount int 23 } 24 25 init { 26 runCount = 0 27 } 28 29 helpers { 30 func keccak256Go(data []byte) []byte { 31 h := sha3.NewLegacyKeccak256() 32 h.Write(data) 33 return h.Sum(nil) 34 } 35 36 func resetCount() { runCount = 0 } 37 func bumpCount() { runCount++ } 38 } 39 } 40 41 setup ts { 42 import { 43 import { keccak256 } from 'viem' 44 } 45 46 declare { 47 let runCount = 0 48 } 49 50 helpers { 51 function keccak256Ts(data: Uint8Array): Uint8Array { 52 return keccak256(data, 'bytes') 53 } 54 55 function resetCount(): void { runCount = 0 } 56 function bumpCount(): void { runCount++ } 57 } 58 } 59 60 setup rust { 61 import { 62 use tiny_keccak::{Hasher, Keccak}; 63 } 64 65 helpers { 66 fn keccak256_rust(data: &[u8]) -> [u8; 32] { 67 let mut hasher = Keccak::v256(); 68 let mut output = [0u8; 32]; 69 hasher.update(data); 70 hasher.finalize(&mut output); 71 output 72 } 73 } 74 } 75 76 # Inline hex fixture 77 fixture shortInput { 78 description: "32-byte input" 79 hex: "68656c6c6f20776f726c6468656c6c6f20776f726c6468656c6c6f20776f72" 80 } 81 82 # File-referenced fixture for larger data 83 fixture longInput { 84 hex: @file("fixtures/keccak/input_1024.hex") 85 } 86 87 # Basic benchmark — all three languages 88 bench hashShort { 89 description: "Hash a 32-byte input" 90 go: keccak256Go(shortInput) 91 ts: keccak256Ts(shortInput) 92 rust: keccak256_rust(&shortInput) 93 } 94 95 # Benchmark with per-bench overrides 96 bench hashLong { 97 description: "Hash a 1024-byte input" 98 targetTime: 5000ms 99 count: 5 100 go: keccak256Go(longInput) 101 ts: keccak256Ts(longInput) 102 rust: keccak256_rust(&longInput) 103 } 104 105 # Benchmark with lifecycle hooks 106 bench hashWithHooks { 107 description: "Hash with counter tracking via hooks" 108 iterations: 10000 109 mode: "fixed" 110 before go: resetCount() 111 before ts: resetCount() 112 each go: bumpCount() 113 each ts: bumpCount() 114 go: keccak256Go(shortInput) 115 ts: keccak256Ts(shortInput) 116 after go: { _ = runCount } 117 after ts: { void runCount } 118 } 119 120 # Go-only benchmark (no ts/rust implementation) 121 bench goOnly { 122 description: "Go-specific operation" 123 go: keccak256Go(shortInput) 124 } 125 126 after { 127 charting.drawTable( 128 title: "Keccak256 Performance", 129 description: "Go vs TypeScript vs Rust", 130 output: "keccak-bar.svg", 131 sortBy: "speedup", 132 sortOrder: "desc", 133 showStats: true, 134 showMemory: true 135 ) 136 137 charting.drawSpeedupChart( 138 title: "Keccak256 Scaling", 139 output: "keccak-line.svg", 140 xlabel: "Input Size" 141 ) 142 } 143}

1declare suite sort performance timeBased sameDataset: true { 2 fixture s100 { hex: @file("sort_100.hex") } 3 fixture s200 { hex: @file("sort_200.hex") } 4 bench n100 { go: sort(s100) ts: sort(s100) } 5 bench n200 { go: sort(s200) ts: sort(s200) } 6}

1use std::charting 2declare suite trend performance timeBased sameDataset: true { 3 bench n100 { go: work(s100) ts: work(s100) } 4 bench n200 { go: work(s200) ts: work(s200) } 5 after { charting.drawLineChart(title: "Scaling") } 6}