Architecture

How poly-bench is structured, how benchmarks flow, and how the runtime isolation works

poly-bench is built as a modular Rust workspace where each crate handles a specific responsibility. This architecture enables clean separation between parsing, validation, code generation, execution, and reporting.

The Big Idea

DSL files (.bench) are the source of truth for benchmark definitions
IR (Intermediate Representation) is a normalized, validated structure that all downstream tools consume
Runtimes are isolated per-language execution environments that run generated benchmark code
Reporters transform raw timing data into human-readable output (console, markdown, JSON, charts)

The DSL compiles to native benchmark code in each language. Go benchmarks use Go's runtime, TypeScript runs in Node.js, and Rust uses Cargo. There's no interpretation or FFI — each language runs at full native speed.

Component Map

Every benchmark flows through the same pipeline. Your .bench file is parsed into an AST, lowered to IR, used to generate language-specific code, executed in isolated runtimes, and finally reported.

Your .bench file

DSL Parser

poly-bench-dsl

-Lexer → Token stream
-Parser → AST (Abstract Syntax Tree)
-Validator → Semantic checks

IR Lowering

poly-bench-ir

-Normalize configuration
-Resolve fixtures and imports
-Validate language blocks

Code Generation

poly-bench-runtime

-Go: Generate benchmark_plugin.go
-TypeScript: Generate benchmark.ts
-Rust: Generate main.rs

Runtime Execution

poly-bench-executor

-Spawn isolated subprocess per language
-Run warmup iterations
-Collect timing measurements
-Track memory allocations

Reporting

poly-bench-reporter

-Console: Colored tables
-Markdown: Documentation-ready
-JSON: CI/automation
-SVG: Generated charts

Results

Crate Responsibilities

Crate	What it's for
`poly-bench-dsl`	Lexer, parser, AST, formatter, and validator for `.bench` files
`poly-bench-ir`	Intermediate representation — normalized, validated benchmark structures
`poly-bench-runtime`	Language-specific code generation and runtime execution (Go, TS, Rust)
`poly-bench-executor`	Orchestrates benchmark runs, manages calibration, collects measurements
`poly-bench-reporter`	Generates output in console, markdown, JSON, and SVG chart formats
`poly-bench-project`	Project initialization, dependency management, manifest handling
`poly-bench-stdlib`	Standard library modules (`std::anvil`, `std::charting`, `std::constants`)
`poly-bench-lsp-v2`	Language Server Protocol implementation for editor support

The modular design means you can use individual crates in your own tooling. For example, use poly-bench-dsl to parse .bench files in a custom analysis tool.

How Benchmarks Flow

1. Parse & Validate

When you run poly-bench run benchmark.bench, the DSL parser tokenizes and parses your file into an AST. The validator checks for semantic errors like undefined fixtures or missing language blocks.

.bench source

Lexer

Character stream → Tokens

Parser

Tokens → AST nodes (Suite, Setup, Bench, Fixture)

Validator

Check references, types, configuration

AST

In-memory representation of your benchmark

2. Lower to IR

The AST is lowered to an Intermediate Representation that's fully resolved and normalized. This step:

Resolves all fixture references
Merges configuration with defaults
Validates that all referenced symbols exist
Prepares data structures for code generation

AST

IR Lowering

-Resolve fixture hex/json to bytes
-Apply default configuration
-Validate language block completeness
-Normalize import statements

BenchmarkIR

Fully resolved, ready for codegen

3. Generate Code

For each target language, poly-bench generates native benchmark code. The generated code includes:

Import statements from setup blocks
Helper functions from setup blocks
Fixture data as byte arrays
Benchmark harness with timing code

1// Auto-generated by poly-bench
2package main
3
4import (
5  "golang.org/x/crypto/sha3"
6  "time"
7)
8
9// Fixture: data
10var data = []byte{0x68, 0x65, 0x6c, 0x6c, 0x6f, ...}
11
12// Helper from setup block
13func keccak256Go(data []byte) []byte {
14  h := sha3.NewLegacyKeccak256()
15  h.Write(data)
16  return h.Sum(nil)
17}
18
19func benchmark_keccak256Bench(iterations int) time.Duration {
20  start := time.Now()
21  for i := 0; i < iterations; i++ {
22      _ = keccak256Go(data)
23  }
24  return time.Since(start)
25}

4. Execute in Isolated Runtimes

Each language runs in its own isolated subprocess:

Language	Runtime	Memory Tracking
Go	Plugin-based execution or subprocess	`runtime.ReadMemStats`
TypeScript	Node.js subprocess	`process.memoryUsage()`
Rust	Cargo build + subprocess	Custom allocator tracking

Executor

poly-bench-executor

go run / go plugin → collect timing + memory

TypeScript

node benchmark.ts → collect timing + memory

Rust

cargo run --release → collect timing + memory

Runtime environments are isolated in .polybench/runtime-env/. Each language has its own dependency directory, ensuring reproducible builds and no global pollution.

Calibration Mode

In mode: "auto", poly-bench calibrates the iteration count to hit a target execution time:

Start calibration

Run pilot (100 iterations)

Measure baseline timing

Calculate iterations

iterations = targetTime / (pilot_time / 100)

Clamp to bounds

Apply minIterations / maxIterations

Execute benchmark

Run with calibrated iteration count

This ensures:

Fast operations get enough iterations for stable timing
Slow operations don't run for hours
All languages use the same iteration count for fair comparison

Statistics Pipeline

After collecting raw measurements, poly-bench computes statistics:

Warmup removal: Discard the first N iterations (configurable via warmup)
Multi-run aggregation: Run the benchmark count times
Outlier detection: Optionally remove outliers using IQR method
Statistics: Calculate mean, standard deviation, operations per second
Comparison: Compute speedup ratios relative to baseline

1Summary: keccak256Bench
2─────────────────────────────────────────────
3
4│ Lang │ Mean (ns/op) │ Std Dev │ Ops/s    │
5├──────┼──────────────┼─────────┼──────────┤
6│ rust │ 993          │ ± 1.8%  │ 1,006,752│
7│ go   │ 1,216        │ ± 2.1%  │ 822,714  │
8│ ts   │ 15,667       │ ± 3.1%  │ 63,839   │
9
10Comparison (baseline: go):
11  rust: 1.22x faster
12  ts:   12.88x slower

Standard Library Architecture

The standard library (poly-bench-stdlib) provides built-in modules:

Module	Purpose	Key Features
`std::anvil`	Ethereum development	Spawn local Anvil node, provide RPC URL
`std::charting`	Visualization	Tables, pie charts, speedup charts as SVG
`std::constants`	Math constants	Pi, E, and other common values

Standard library modules are implemented in Rust and invoked via directives in before/after blocks:

1use std::anvil
2use std::charting
3
4suite example {
5  before {
6      anvil.spawnAnvil(fork: "https://eth.llamarpc.com")
7  }
8
9  // ... benchmarks ...
10
11  after {
12      charting.drawTable(title: "Results")
13      anvil.stopAnvil()
14  }
15}

1// Auto-generated by poly-bench 2package main 3 4import ( 5 "golang.org/x/crypto/sha3" 6 "time" 7) 8 9// Fixture: data 10var data = []byte{0x68, 0x65, 0x6c, 0x6c, 0x6f, ...} 11 12// Helper from setup block 13func keccak256Go(data []byte) []byte { 14 h := sha3.NewLegacyKeccak256() 15 h.Write(data) 16 return h.Sum(nil) 17} 18 19func benchmark_keccak256Bench(iterations int) time.Duration { 20 start := time.Now() 21 for i := 0; i < iterations; i++ { 22 _ = keccak256Go(data) 23 } 24 return time.Since(start) 25}

Language

Runtime

Memory Tracking

Plugin-based execution or subprocess

runtime.ReadMemStats

TypeScript

Node.js subprocess

process.memoryUsage()

Rust

Cargo build + subprocess

Custom allocator tracking

1Summary: keccak256Bench 2───────────────────────────────────────────── 3 4│ Lang │ Mean (ns/op) │ Std Dev │ Ops/s │ 5├──────┼──────────────┼─────────┼──────────┤ 6│ rust │ 993 │ ± 1.8% │ 1,006,752│ 7│ go │ 1,216 │ ± 2.1% │ 822,714 │ 8│ ts │ 15,667 │ ± 3.1% │ 63,839 │ 9 10Comparison (baseline: go): 11 rust: 1.22x faster 12 ts: 12.88x slower

Module

Purpose

Key Features

std::anvil

Ethereum development

Spawn local Anvil node, provide RPC URL

std::charting

Visualization

Tables, pie charts, speedup charts as SVG

std::constants

Math constants

Pi, E, and other common values

1use std::anvil 2use std::charting 3 4suite example { 5 before { 6 anvil.spawnAnvil(fork: "https://eth.llamarpc.com") 7 } 8 9 // ... benchmarks ... 10 11 after { 12 charting.drawTable(title: "Results") 13 anvil.stopAnvil() 14 } 15}

Architecture

The Big Idea

Component Map

Crate Responsibilities

How Benchmarks Flow

1. Parse & Validate

2. Lower to IR

3. Generate Code

4. Execute in Isolated Runtimes

Calibration Mode

Statistics Pipeline

Standard Library Architecture

Further Reading

The Big Idea

Component Map

Crate Responsibilities

How Benchmarks Flow

1. Parse & Validate

2. Lower to IR

3. Generate Code

4. Execute in Isolated Runtimes

Calibration Mode

Statistics Pipeline

Standard Library Architecture

Further Reading