bgperf2 is a performance measurement tool for BGP implementation. This was forked from https://github.com/osrg/bgperf and has been changed significantly.
- How to install
- How to use
- How bgperf2 works
- Benchmark remote target
- MRT injection
- Running on AWS EC2 spot instances
I've changed bgperf to work with python 3 and work with new versions of all the NOSes. It actually works, the original version that this is a fork of does not work anymore because of newer version of python and each of the routing stacks.
This version no longer compiles EXABGP or FRR, it gets PIP or containers already created. Quagga has been removed since it doesn't seem to be updated anymore.
To get bgperf2 to work with all the changes in each stack I've had to change configuration. I don't know if all the features of bgperr still work: I've gotten the simplest version of each config to work.
Caveats:
I don't know if adding more policy will still work for all targets. I haven't tested remote targets.
bgperf2 creates containers of bgp software to be performance tested. It then can run either one off tests "bench" or a group of tests "batch". It will create graphs to help you understand what the bgp software is doing as well as to compare across batches.
bgperf2 has two main ways of producing prefixes for BGP performance testing. The first way uses either BIRD or EXABGP to create prefixes and send them. These are pretty lightweight, so it's easy to generate hundreds (or even thousands) of neighbors with a small amount (hundreds or thousands) or prefixes. BIRD is generally faster, so it is the default but EXABGP is around for some extra testing.
The second way to generate traffic is by playing back MRT files using bgpdump2. This is much faster, and is good for playing back internet size tables. RouteViews is a good place to get MRT files to play back.
- Python 3.7 or later
- Docker (your user must be in the
dockergroup) - Sysstat (
benchshells out tompstat)
$ git clone https://github.com/netenglabs/bgperf2.git
$ cd bgperf2
$ python3 -m venv venv
$ venv/bin/pip install -r pip-requirements.txt
$ venv/bin/python bgperf2.py --help
usage: bgperf2.py [-h] [-b BENCH_NAME] [-d DIR]
{doctor,prepare,update,bench,config,batch} ...
BGP performance measuring tool
positional arguments:
{doctor,prepare,update,bench,config,batch}
doctor check env
prepare prepare env
update rebuild bgp docker images
bench run benchmarks
config generate config
batch run batch benchmarks
options:
-h, --help show this help message and exit
-b BENCH_NAME, --bench-name BENCH_NAME
-d DIR, --dir DIR
$ ./bgperf2.py prepare
$ ./bgperf2.py doctor
docker version ... ok (1.9.1)
bgperf2 image ... ok
gobgp image ... ok
bird image ... okActivating the venv (source venv/bin/activate) lets you drop the venv/bin/
prefix and run ./bgperf2.py directly; the rest of this document does that.
On a fresh VM, new_vm.sh installs the system packages, creates the venv, and
downloads a RouteViews MRT file into mrt/ for the MRT-based benchmarks.
Generated graphs and CSVs are written to results/, overridable with
--results-dir. That directory is gitignored, so runs no longer leave files
scattered in the repo root.
Use bench command to start benchmark test.
By default, bgperf2 benchmarks BIRD.
bgperf2 boots 100 BGP test peers each advertising 100 routes to it.
$ ./bgperf2.py bench
run monitor
run bird
Waiting 18 seconds for monitor
run tester tester type bird
launched 1 testers
elapsed: 5sec, cpu: 1.28%, mem: 10.50MB, mon recved: 10000, neighbors_received: 100, neighbors_accepted: 100, %idle 94.7, free mem 57.15GB
bird: 2.17.1+branch.master.fe0c22277c21
Max cpu: 23.41, max mem: 10.50MB
Min %idle 94.7, Min mem free 57.15GB
Time since first received prefix: 4
total time: 31.32s
name, target, version, peers, prefixes per peer, required, received, monitor (s), elapsed (s), prefix received (s), testers (s), total time, max cpu %, max mem (GB), min idle%, min free mem (GB), flags, date, cores, Mem (GB), tester errors, tester timeouts, failed, MSG, filters
bird,bird,2.17.1+branch.master.fe0c22277c21,100,100,9900,10000,18,5,1,4,31.32,23,0.01,95,57.152,,2026-08-07,32,60.73GB,0,0,,,As you might notice, the interesting statistics are shown twice, once in an easy to read format and the second in a CSV format to easily copy and paste to do analysis later.
To change a target implementation, use -t option.
Currently, bgperf2 supports BIRD and FRRouting
(other than GoBGP. There is very intial support forRustyBGP, partly
because RustyBGP doesn't support all policy that Bgperf2 tries to use for policy testing. If you just want to
do routes and neighbors then RustyBGP works.
$ ./bgperf2.py bench -t gobgp
run monitor
run gobgp
Waiting 19 seconds for monitor
run tester tester type bird
launched 1 testers
elapsed: 6sec, cpu: 5.28%, mem: 40.15MB, mon recved: 10000, neighbors_received: 100, neighbors_accepted: 100, %idle 57.0, free mem 56.04GB
gobgp: 3.37.0
Max cpu: 893.00, max mem: 40.15MB
Min %idle 57.0, Min mem free 56.04GB
Time since first received prefix: 5
total time: 33.27s
name, target, version, peers, prefixes per peer, required, received, monitor (s), elapsed (s), prefix received (s), testers (s), total time, max cpu %, max mem (GB), min idle%, min free mem (GB), flags, date, cores, Mem (GB), tester errors, tester timeouts, failed, MSG, filters
gobgp,gobgp,3.37.0,100,100,9900,10000,19,6,1,5,33.27,893,0.04,57,56.039,,2026-08-07,32,60.73GB,0,0,,,To change a load, use following options.
-n: the number of BGP test peer (default 100)-p: the number of prefix each peer advertise (default 100)-a: the number of as-path filter (default 0)-e: the number of prefix-list filter (default 0)-c: the number of community-list filter (default 0)-x: the number of ext-community-list filter (default 0)
$ python3 bgperf2.py bench
run monitor
run gobgp
Waiting 5 seconds for neighbor
run tester tester type normal
tester booting.. (100/100)
elapsed: 2sec, cpu: 0.79%, mem: 42.27MB, recved: 10000
gobgp: 2.29.0
Max cpu: 554.03, max mem: 45.71MB
Time since first received prefix: 2
total time: 24.07s
name, target, version, peers, prefixes per peer, neighbor (s), elapsed (s), prefix received (s), exabgp (s), total time, max cpu %, max mem (GB), flags, date,cores,Mem (GB)
gobgp,gobgp,2.29.0,100,100,5,2,0,2,24.07,554,0.045,,2021-08-02,32,62.82GBFor a comprehensive list of options, run python3 ./bgperf2.py bench --help.
Targets are the container being tested. bgperf2 was initially created to create containers of BGP software and and make them testable. However, a challenge is the best way to be able to do this over time. For instance, the instructions for how to build these software stacks has changed over time. So is the best way to keep up to compile the software ourselves or to try to download containers from the open source project themsevles. When I originally forked bgperf2 it hadn't changed in 4 years, so almost none of the containers could be built and all of the software had changed how they interat. I'm not sure how best to make bgperf2 work over time.
FRR is the clearest example. There used to be an frr target that used a prebuilt FRRouting
container hardcoded to 7.5.1, which meant the version you tested was whatever that image happened
to pin. It has been removed. The frr_c target replaces it: it checks FRRouting out of git and
builds the container, so you choose the version.
Each buildable daemon gets one image per version, tagged bgperf/<name>:<version>. Build them
with prepare (which builds the daemon's default list) or update (for anything else):
./bgperf2.py prepare -t frr_c # master + 8.5, 9.1, 10.0, 10.7
./bgperf2.py prepare -t frr_c --versions 10.4,10.5 # just these two, plus master
./bgperf2.py update frr_c --version 10.7 # add one later
./bgperf2.py images # what is built, and from which refThen bench one directly, or list several in a batch config:
./bgperf2.py bench -t frr_c --version 10.1 -n 10 -p 100000 -
name: frr_c
versions: [8.5, 9.1, 10.0, 10.7]
tester_type: birdThat runs once per version and labels each row frr_c 10.1 and so on, so the CSV and the graphs
tell them apart. Add your own label: to override. A target with no versions: uses the
unversioned image, which tracks the daemon's default branch.
Versions are named the way the project names its releases, and each daemon module translates:
FRR 10.1 is the branch stable/10.1, FRR 10.1.1 is the tag frr-10.1.1, BIRD 2.19.2 and
GoBGP 3.37.0 are the tags v2.19.2 and v3.37.0. Anything unrecognized is passed through as a
raw git ref, so --version master or a commit sha also work. ./bgperf2.py images prints the
mapping.
A bare prepare builds one image per daemon, tracking its default branch. Version images are
opt-in behind -t, because a daemon's whole version list is hours of compiling. Either way it
skips what already exists, so re-running after adding a version is cheap; -f forces a rebuild.
doctor lists which versions are built and which are not.
Every version image must exist before a run starts — bench and batch both check up front and
tell you the exact update command rather than failing an hour into a batch.
The commercial NOSes work the same way once you tag what you downloaded (docker tag <image> crpd:24.2, then -t junos --version 24.2).
BIRD 3 is the multi-threaded rewrite; BIRD 2 does everything in one thread. But BIRD 3 starts a
single worker unless the config asks for more, so a 2.x-vs-3.x comparison without --threads
measures BIRD 3 pretending to be BIRD 2. Measured on 3.3.2: 2 OS threads by default, 5 with
threads 4. BIRD 2.19.2 parses the keyword and ignores it, staying at 1 — so a batch config can
set it for both.
./bgperf2.py bench -t bird --version 3.3.2 --threads 4 -n 10 -p 100000 - name: bird
version: 2.19.2
label: bird 2 (single-threaded)
- name: bird
version: 3.3.2
threads: 4
label: bird 3 (4 threads)--threads is ignored by daemons with no such setting.
Build instructions drift: the base distro moves on, dependencies get renamed, configure flags come and go. Two levels of override, cheapest first.
For a different value — base image, extra packages, configure flags — add a rule to the daemon
class's VERSION_BUILD_VARS, keyed by version prefix:
class FRRoutingCompiled(Container):
VERSION_BUILD_VARS = (
('8.', {'ubuntu_version': '20.04', 'extra_setup': 'apt-get install -y libyang-dev'}),
)For a genuinely different recipe, drop a whole Dockerfile in
dockerfiles/<name>/<version>.dockerfile (see dockerfiles/README.md). The longest matching
version prefix wins, so 10.dockerfile covers the whole 10.x series.
Either way, check the result without paying for a build:
./bgperf2.py dockerfile frr_c --version 8.0bgperf2 was originally created to test open source bgp software, so for most containers it compiles the software and creates a container. For commerical NOSes this doesn't make sense. For those you will need to download the container images manually and then use bgperf2.
For most of these images, bgperf2 mounts a local directory (usually in /tmp/bgperf2) to the container. These commerical stacks then write back data as root, and set the privleges so that a regular user cannot delete these files and directories.
bgperf2 tries to delete /tmp/bgperf2 before it runs, but it can't with data from these stacks, so you might need to remove them yourself. The other option is to run bgperf2 as root <shrugs>, that's not a good idea.
sudo rm -rf /tmp/bpgperf2
I have setup multi-threaded support by default in both of these. If you want to do uni-threaded performance testing you will have to edit config files, which is documented below.
Warning The license for these stacks prohibits publishing results. Don't publish results.
To download, after getting an account: https://www.arista.com/en/support/software-download. Make sure you get the cEOS64 image. I didn't the first time, and the results are frustringly slow. After downloading:
$ docker import ../NOS/cEOS64-lab-4.27.0F.tar.xz ceos:latestBe sure to use this command for importing: if you don't tag the image as ceos:latest then bgperf2 won't be able to find the image.
N.B. EOS takes longer to startup than other BGP software I've tested with bgperf2, so don't be alarmed. However, if it's taken more than 60 seconds to establish a neighbor, something is wrong and start looking at logs.
EOS has less clear directions on how to setup multithreading and the consequences. I can't find an authoritative doc to point to.
However, if you want to remove multi-threading support, remove this line from nos_templates/eos.j2
service routing protocols model multi-agentThere is no way to adjust the number of threads being used. For cEOS it appears to be hardcoded no matter the hardware that you have.
Download the image to your local machine and then run:
$ docker load -i ../NOS/junos-routing-crpd-docker-21.3R1-S1.1.tgz
$ docker tag crpd:21.3R1-S1.1 crpd:latestBe sure you tag the image or bgperf2 cannot find the image and everything will fail.
bgperf2 mounts the log directory as /tmp/bgperf2/junos/logs, however there are a lot there and most of it is not relevant. To see if your config worked correctly on startup:
$ docker logs bgperf_junos_targetDeploying BGP RIB Sharding and Update Threading -- while informative it's weird to me that it's 2021 and getting mult-threaded performance requires a 40 page document. How am I not supposed to think that networking is two decades behind everybody else in software? (This isn't just a Juniper problem by any means)
For multithreading, as mentioned above bgperf2 sets this up by default in nos_tempaltes/junos.j2
processes {
routing {
bgp {
rib-sharding {
number-of-shards {{ data.cores }};
}
update-threading {
number-of-threads {{ data.cores }};
}
}
}
}data.cores is set in junos.py and by default it's half the number of availble cores on the test machine, with a max of 31, since that is the Junos max. If you want to try setting the threads to something different you can hard code those values. If you want to see without multi-threading, delete that whole section.
A feature called batch lets you run multiple tests, collect all the data, and produces graphs. If you run a test that runs out of physical RAM on your machine, linux OOM killer will just kill the process and you'll lose the data from that experiment.
The benchmarks/ directory holds example batch configs that show how it works — start with
benchmarks/benchmark.yaml. You can list the targets that you want tested in a batch, as well as
iterate through prefix count and neighbor count. Run one with:
$ ./bgperf2.py batch -c benchmarks/benchmark.yamlConfigs that play back MRT data expect the file at mrt/rib.20210801.0000, which new_vm.sh
downloads. Paths in a batch config may be relative or use ~; they are resolved before being
handed to Docker. Keep personal, unshared configs in benchmarks/local/ — that path is gitignored.
If you use a file that looks like this:
tests:
-
name: 10K
neighbors: [10, 30, 50, 100]
prefixes: [10_000]
filter_test: [None]
targets:
-
name: bird
label: bird -s
single_table: True
-
name: frr
-
name: gobgp
-
name: frr_c
label: frr 8
-
name: rustybgpYou will get output like this:
name, target, version, peers, prefixes per peer, neighbor (s), elapsed (s), prefix received (s), exabgp (s), total time, max cpu %, max mem (GB), flags, date,cores,Mem (GB)
bird -s,bird,v2.0.8-59-gf761be6b,10,10000,3,2,0,2,13.9,30,0.015,-s,2021-08-02,32,62.82GB
frr,frr,FRRouting 7.5.1_git (910c507f1541).,10,10000,0,3,0,3,11.58,37,0.089,,2021-08-02,32,62.82GB
gobgp,gobgp,2.29.0,10,10000,6,9,0,9,24.65,1450,0.141,,2021-08-02,32,62.82GB
frr 8,frr_c,FRRouting 8.0-bgperf (489e9d4e8956).,10,10000,0,3,0,3,11.61,31,0.1,,2021-08-02,32,62.82GB
rustybgp,rustybgp,exec,10,10000,4,3,0,3,15.94,262,0.032,,2021-08-02,32,62.82GB
bird -s,bird,v2.0.8-59-gf761be6b,30,10000,4,3,0,3,26.99,100,0.161,-s,2021-08-02,32,62.82GB
frr,frr,FRRouting 7.5.1_git (ab68d18f80c7).,30,10000,0,3,0,3,22.53,86,0.302,,2021-08-02,32,62.82GB
gobgp,gobgp,2.29.0,30,10000,5,61,0,61,85.8,1620,0.447,,2021-08-02,32,62.82GB
frr 8,frr_c,FRRouting 8.0-bgperf (750804dc0e98).,30,10000,0,3,0,3,22.21,78,0.296,,2021-08-02,32,62.82GB
rustybgp,rustybgp,exec,30,10000,4,4,0,4,28.09,446,0.128,,2021-08-02,32,62.82GB
bird -s,bird,v2.0.8-59-gf761be6b,50,10000,3,6,0,6,42.35,100,0.396,-s,2021-08-02,32,62.82GB
frr,frr,FRRouting 7.5.1_git (9e4604a042a6).,50,10000,0,4,0,4,35.48,102,0.513,,2021-08-02,32,62.82GB
gobgp,gobgp,2.29.0,50,10000,4,160,0,160,194.23,1638,0.875,,2021-08-02,32,62.82GB
frr 8,frr_c,FRRouting 8.0-bgperf (eb81873b8335).,50,10000,1,6,0,6,36.69,103,0.52,,2021-08-02,32,62.82GB
rustybgp,rustybgp,exec,50,10000,4,5,0,5,40.74,469,0.307,,2021-08-02,32,62.82GB
bird -s,bird,v2.0.8-59-gf761be6b,100,10000,3,13,0,13,91.68,100,1.343,-s,2021-08-02,32,62.82GB
frr,frr,FRRouting 7.5.1_git (8dc6f2f40d8c).,100,10000,1,7,0,7,74.99,101,1.291,,2021-08-02,32,62.82GB
gobgp,gobgp,2.29.0,100,10000,5,661,0,661,724.83,1664,1.846,,2021-08-02,32,62.82GB
frr 8,frr_c,FRRouting 8.0-bgperf (74ac3704b034).,100,10000,1,8,1,7,70.46,103,1.116,,2021-08-02,32,62.82GB
rustybgp,rustybgp,exec,100,10000,6,16,0,16,80.37,597,1.253,,2021-08-02,32,62.82GBIt will create graphs and a CSV file of the output.
And some graphs. These are some of the important ones
If you try to change the config, it's a little tricky to debug what's going on since there are so many containers. What bgperf is doing is creating configs and startup scripts in 2 and then it copies those to the containers before launching them. It creates three containers: bgperf_exabgp_tester_tester, bgperf_<target>_target, and bgperf2_monitor. If things aren't working, it's probably because the config for the target is not correct. bgperf2 puts all the log output in /tmp/bgperf2/*.log, but what it doesn't do is capture the output of the startup script.
If it doesn't seem to be working, try with 1 peer and 1 route (-n1 -p1) and make sure that it connecting. If it's just stuck at waiting to connect to the neighbor, then probably the config is wrong and neighbors are not being established between the monitor (gobgp) and the NOS being tested
You'll have to break into gobgp and the test config.
if you want to see what is happening when the test containers starts, after the test is over (or you've killed it), run
$ docker exec bgperf_bird_target /root/config/start.sh
that's what bgperf2 is doing. It creates a /root/config/start.sh command and is running it, so if you run it manually you can see if that command produces output to help you debug.
to clean up any existing docker containers
$ docker kill `docker ps -q`; docker rm `docker ps -aq`
The startup script is in /tmp/bgperf/<target>/start.sh and gets copied to the target as /root/config/start.sh.
In other words, to launch the start.sh and see the output you can run this docker command:
$ docker exec bgperf_bird_target /root/config/start.sh
bird: I found another BIRD running.
In this case, things were already working, so I'll run ps and kill the old bird and start a new one.
$ docker exec bgperf_bird_target ps auxww
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.0 0.0 3984 2820 ? Ss 21:21 0:00 bash
root 14 0.0 0.0 4144 2016 ? Ss 21:21 0:00 bird -c /root/config/bird.conf
root 22 0.0 0.0 5904 2784 ? Rs 21:22 0:00 ps auxww
$ docker exec bgperf_bird_target kill 14
$ docker exec bgperf_bird_target /root/config/start.sh
No output, so it was just fine.