Add scenario capability to benchmark script #2779

stevenlanders · 2026-01-27T20:45:29Z

Summary

Adds configurable scenario support to the benchmark script, enabling ERC20 and other contract-based load testing scenarios. (Fyi the benchmark won't really perform fully until the storage + autobahn changes are in, but this tooling now operates)

Changes

New Benchmark Package (`app/benchmark/`)

Moved benchmark logic from app/benchmark.go to a dedicated package
Two-phase generator: Warmup → Setup (deploy contracts) → Load (generate transactions)
Automatic contract deployment with address extraction from receipts
Configurable via JSON scenario files

Scenario Configuration (`scripts/scenarios/`)

erc20.json - ERC20 token transfer benchmark (5000 accounts)
EVMTransfer.json - Simple native transfers
mixed.json - Combined workload example
README.md - Documentation

Mock Balance Fixes (`giga/deps/xbank/keeper/mock_balances.go`)

Fixed OCC compatibility: mock balances now update supply as normal side effects
Removed global address tracking that was interfering with OCC re-execution
Works correctly with parallel transaction execution

Script Updates (`scripts/benchmark.sh`)

Added BENCHMARK_CONFIG env var to specify scenario file
Default: scripts/scenarios/EVMTransfer.json

Usage

ERC20 benchmark with Giga Executor

GIGA_EXECUTOR=true BENCHMARK_CONFIG=scripts/scenarios/erc20.json ./scripts/benchmark.sh

github-actions · 2026-01-27T20:46:28Z

The latest Buf updates on your PR. Results from workflow Buf / buf (pull_request).

Build	Format	Lint	Breaking	Updated (UTC)
`✅ passed`	`✅ passed`	`✅ passed`	`✅ passed`	Jan 30, 2026, 8:53 PM

codecov · 2026-01-27T20:50:01Z

Codecov Report

❌ Patch coverage is 54.03950% with 256 lines in your changes missing coverage. Please review.
✅ Project coverage is 56.73%. Comparing base (16b48de) to head (afaf6e5).
⚠️ Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
app/benchmark/generator.go	46.04%	133 Missing and 17 partials ⚠️
app/benchmark/contracts.go	0.00%	40 Missing ⚠️
app/benchmark.go	44.82%	31 Missing and 1 partial ⚠️
app/benchmark/benchmark.go	60.00%	9 Missing and 3 partials ⚠️
app/benchmark/config.go	54.54%	10 Missing ⚠️
app/benchmark/logger.go	92.92%	5 Missing and 3 partials ⚠️
app/app.go	60.00%	2 Missing ⚠️
app/invariance.go	0.00%	1 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2779      +/-   ##
==========================================
+ Coverage   47.10%   56.73%   +9.62%     
==========================================
  Files        1939     2014      +75     
  Lines      159389   165422    +6033     
==========================================
+ Hits        75086    93856   +18770     
+ Misses      77799    63367   -14432     
- Partials     6504     8199    +1695

Flag	Coverage Δ
sei-chain	`41.65% <54.03%> (+0.01%)`	⬆️
sei-cosmos	`48.11% <ø> (-0.02%)`	⬇️
sei-db	`68.72% <ø> (ø)`
sei-tendermint	`58.51% <ø> (?)`

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
app/abci.go	`66.92% <100.00%> (+1.27%)`	⬆️
giga/deps/xevm/state/balance.go	`60.67% <100.00%> (+0.90%)`	⬆️
giga/deps/xevm/state/mock_balances_noop.go	`100.00% <100.00%> (ø)`
x/evm/state/balance.go	`61.95% <100.00%> (+0.84%)`	⬆️
x/evm/state/mock_balances_noop.go	`100.00% <100.00%> (ø)`
app/app.go	`71.24% <60.00%> (+0.30%)`	⬆️
app/invariance.go	`41.61% <0.00%> (-0.83%)`	⬇️
app/benchmark/logger.go	`92.92% <92.92%> (ø)`
app/benchmark/config.go	`54.54% <54.54%> (ø)`
app/benchmark/benchmark.go	`60.00% <60.00%> (ø)`
... and 3 more

... and 312 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

app/benchmark/generator.go

+	for hash := range g.pendingDeploys {
+		hashes = append(hashes, hash)
+	}


app/benchmark/logger.go

+
+	// Initialize lastFlushTime on first increment (when blocks actually start processing)
+	if l.lastFlushTime.IsZero() {
+		l.lastFlushTime = time.Now()


app/benchmark/logger.go

+func (l *Logger) StartBlockProcessing() {
+	l.mx.Lock()
+	defer l.mx.Unlock()
+	l.blockProcessStartTime = time.Now()


app/benchmark/logger.go

+
+// FlushLog outputs the current statistics.
+func (l *Logger) FlushLog() {
+	now := time.Now()


app/benchmark/benchmark.go

+	}
+
+	benchLogger := NewLogger(logger)
+	go benchLogger.Start(ctx)


app/benchmark/generator.go

+	go func() {
+		defer close(ch)
+		for {
+			if ctx.Err() != nil {
+				return
+			}
+
+			txRecords := g.Generate()
+			if len(txRecords) == 0 {
+				continue
+			}
+
+			proposal := &abci.ResponsePrepareProposal{
+				TxRecords: txRecords,
+			}
+
+			select {
+			case ch <- proposal:
+			case <-ctx.Done():
+				return
+			}
+		}
+	}()


app/benchmark/logger.go

+	if duration <= 0 {
+		return 0
+	}
+	return float64(txCount) / duration.Seconds()


app/benchmark/config.go

app/benchmark/generator.go

wen-coding · 2026-01-30T19:54:16Z

app/benchmark/generator.go

+			NewAccountRate: cfg.Accounts.NewAccountRate,
+		})
+		bg.accountPools = append(bg.accountPools, bg.sharedAccounts)
+	}


Sorry for a naive question, when would cfg.Accounts be nil? How do you generate traffic with no shared account pool?

by default the load generator library generates a new account on every send if for some reason this is nil

That seems a bit inefficient, is that really frequently in use?

app/benchmark/generator.go

wen-coding · 2026-01-30T20:48:19Z

app/benchmark/generator.go

+	}, nil
+}
+
+// ProcessReceipts handles receipts from FinalizeBlock to extract deployed addresses.


nit: FinalizeBlock sounds like it's the block containing results at the end...

wen-coding · 2026-01-30T20:49:34Z

app/benchmark/generator.go

+}
+
+// ProcessReceipts handles receipts from FinalizeBlock to extract deployed addresses.
+func (g *Generator) ProcessReceipts(receipts map[common.Hash]*evmtypes.Receipt) {


nit: Can rename to something else which makes it obvious we are extracting deployed addresses?

wen-coding · 2026-01-30T20:52:37Z

app/benchmark/generator.go

+			"gasUsed", receipt.GasUsed)
+
+		// Attach the deployed address to the scenario
+		if err := state.scenario.Attach(g.cfg, addr); err != nil {


Can we set deployed to true if this happens?

wen-coding · 2026-01-30T20:54:36Z

app/benchmark/generator.go

+				"vmError", receipt.VmError,
+				"gasUsed", receipt.GasUsed)
+			// Remove from pending but don't mark as deployed
+			delete(g.pendingDeploys, txHash)


Will this make allDeployed always fail?

wen-coding · 2026-01-30T20:56:06Z

app/benchmark/generator.go

+	weightedConfigs := make([]*generator.WeightedCfg, 0, len(g.scenarios))
+	for _, state := range g.scenarios {
+		if !state.deployed {
+			g.logger.Info("benchmark: Scenario not deployed, skipping", "scenario", state.config.Name)


I guess you could panic here as well, because we checked that all scenarios are deployed before entering this function.

wen-coding · 2026-01-30T21:21:48Z

app/invariance.go

+	// If money is conserved, this should be zero
+	// useiPreTotal - useiPostTotal = how much usei left balances (negative means usei entered balances)
+	// weiDiffInUsei = how much usei was moved to wei balances
+	// supplyChanged = how much new usei was minted


How is this comment related tothis PR?

wen-coding · 2026-01-30T21:22:14Z

app/invariance.go

 	if !useiDiff.IsZero() {
 		panic(fmt.Sprintf("unexpected usei balance total found! Pre-block usei total %s wei total %s total supply %s, post-block usei total %s wei total %s total supply %s", useiPreTotal, weiPreTotal, preTotalSupply, useiPostTotal, weiPostTotal, preTotalSupply.Add(supplyChanged)))
 	}
-	app.Logger().Info("successfully verified supply light invariance")


Removed because this isn't a useful log?

wen-coding · 2026-01-30T21:25:49Z

giga/deps/xevm/state/mock_balances.go

+// ensureMinimumBalance tops off the account if balance is low.
+// Called from GetBalance to ensure preCheck passes in StateTransition.
+func (s *DBImpl) ensureMinimumBalance(evmAddr common.Address) {
+	if s.ctx.ChainID() == "pacific-1" {


I think you use evmcfg.IsLiveEVMChainID(evmChainID) elsewhere?

wen-coding · 2026-01-30T21:26:54Z

giga/deps/xevm/state/mock_balances.go

-	if err != nil {
-		s.err = err
-	}
+	_ = s.k.BankKeeper().SendCoinsAndWei(ctx, moduleAddr, seiAddr, usei, wei)


Should you log this error maybe?

wen-coding · 2026-01-30T21:28:13Z

scripts/benchmark.sh

-BENCHMARK_TXS_PER_BATCH=$BENCHMARK_TXS_PER_BATCH ~/go/bin/seid start --chain-id sei-chain 2>&1 | grep benchmark
+if [ "$DEBUG" = true ]; then
+  # Debug mode: print all output
+  BENCHMARK_CONFIG=$BENCHMARK_CONFIG BENCHMARK_TXS_PER_BATCH=$BENCHMARK_TXS_PER_BATCH ~/go/bin/seid start --chain-id sei-chain


nit: could chain-id uses something more obvious it's a benchmark?

* main: Composite State Store part 1: EVM config and type definitions (#2754) Add scenario capability to benchmark script (#2779) emit rewards withdrawn events for redelegate/undelegate (#2781) add original cachekv as base layer (#2780) Add Ethereum state test runner for Giga executor validation (#2707) Add changelog for 6.2 and 6.3 (#2751) Fix typo in backport CI workflow name (#2784) Upgrade to latest UCI workflows (#2783) Configure self-hosted runners for Go tests (#2715)

stevenlanders added 3 commits January 27, 2026 15:29

add scenario capability to benchmark script

5dd509e

merge main

2431a82

rename to evm.json

d9c041e

stevenlanders added the non-app-hash-breaking label Jan 27, 2026

github-advanced-security bot found potential problems Jan 27, 2026

View reviewed changes

stevenlanders and others added 5 commits January 27, 2026 16:03

mock balance log removal

31d54ca

fix test

8283417

fix lint

eac0ae5

Merge branch 'main' into steven/benchmark-script-with-scenarios

15d96b0

fix default name in benchmark.sh

e0e46ab

arajasek approved these changes Jan 29, 2026

View reviewed changes

stevenlanders and others added 9 commits January 29, 2026 15:53

fix mock balance

bded875

localize mocking better

6def01a

cleanup

63f088e

cleanup mock hook

07b3a08

go fmt

badf641

cleanup extra flush logic

4144f95

Merge branch 'main' into steven/benchmark-script-with-scenarios

44a6a0e

fix store initialization for giga stores

0b08a1c

revert store changes and giga-specific app.go changes to align with main

585486b

wen-coding reviewed Jan 30, 2026

View reviewed changes

stevenlanders added 4 commits January 30, 2026 15:29

align mocking of balances

19b83c9

panic more

a6728e1

refactor for clarity

e9c1b1f

remove unnecessary checks

afaf6e5

wen-coding reviewed Jan 30, 2026

View reviewed changes

wen-coding approved these changes Jan 30, 2026

View reviewed changes

stevenlanders merged commit 54a1a21 into main Jan 30, 2026
43 checks passed

stevenlanders deleted the steven/benchmark-script-with-scenarios branch January 30, 2026 21:36

Add scenario capability to benchmark script #2779

Add scenario capability to benchmark script #2779

Uh oh!

Conversation

stevenlanders commented Jan 27, 2026

Summary

Changes

New Benchmark Package (app/benchmark/)

Scenario Configuration (scripts/scenarios/)

Mock Balance Fixes (giga/deps/xbank/keeper/mock_balances.go)

Script Updates (scripts/benchmark.sh)

Usage

ERC20 benchmark with Giga Executor

Uh oh!

github-actions bot commented Jan 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov bot commented Jan 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Check warning

Check warning

Check warning

Check warning

Check notice

Check notice

Check notice

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

New Benchmark Package (`app/benchmark/`)

Scenario Configuration (`scripts/scenarios/`)

Mock Balance Fixes (`giga/deps/xbank/keeper/mock_balances.go`)

Script Updates (`scripts/benchmark.sh`)

github-actions bot commented Jan 27, 2026 •

edited

Loading

codecov bot commented Jan 27, 2026 •

edited

Loading