Skip to content

bench: Add strcmp benchmark - #97

Open
alanhc wants to merge 1 commit into
ARM-software:masterfrom
alanhc:bench-strcmp
Open

bench: Add strcmp benchmark#97
alanhc wants to merge 1 commit into
ARM-software:masterfrom
alanhc:bench-strcmp

Conversation

@alanhc

@alanhc alanhc commented Aug 23, 2026

Copy link
Copy Markdown

bench: Add strcmp benchmark

string/bench/ currently covers memcpy, memset and strlen. There is no
benchmark for strcmp, so a change to strcmp.S has nothing to be measured
against. This adds one, in the same shape as bench/strlen.c.

I am not employed by Arm and will sign the Assignment Agreement.

What it measures

Six sections, each run for every available implementation:

section what it exercises
Random, equal SPEC2017 length and alignment distribution, comparison runs to the terminating null. Throughput and latency.
Random, mismatch Same distribution, but the operands differ in the last character, so the comparison exits on a mismatch instead of on the null.
Small aligned / misaligned 1..64 bytes
Medium aligned / misaligned 128..4096 bytes

The length and alignment frequency tables are the SPEC2017 ones already used by
bench/strlen.c.

The aligned and misaligned split matters for strcmp specifically: strcmp.S
selects L(loop_aligned) or L(loop_unaligned) on whether the two operands
share a load offset, and those are different loops. The misaligned sections put
the second operand at offset 3 so the unaligned loop is the one being measured.

Variants are dispatched through the DOTEST macro with
RUN/RUNA64/RUNSVE, so every call site is a direct bl; objdump on the
built binary shows no blr in main. Adding an SVE2 implementation later is a
one-line change to DOTEST.

The 32-bit variants are guarded with the same availability check as
string/test/strcmp.c, since __strcmp_arm and __strcmp_armv6m are not
available on every 32-bit target.

On formatting: the file follows bench/strlen.c rather than the .clang-format
added in 36584f3, because that commit did not reformat the existing files and
the two disagree — clang-format would rewrite 131 lines of bench/strlen.c and
218 of bench/memcpy.c.

Buffer placement

The two operands must not sit an exact power of two apart. Three page
aligned, equally sized buffers get placed exactly 8192 bytes apart, and with
both operands then at the same offset the mutually misaligned section came
out faster than the mutually aligned one, which is backwards — the aligned
loop does two loads per iteration, the unaligned one does three.

Skewing the second and third buffers by 64 and 128 bytes fixes it. The skews
are multiples of 8, so the mutually aligned main loop is still what gets
measured. Taking the ratio of the two sections within a single run, so the
result does not depend on the clock, at 4K:

aligned / misaligned, skew 0 skew 64 change
Cortex-A510 0.71 1.14 +61%
Cortex-X3 0.95 1.06 +11%

I have not chased the exact mechanism and would not want to assert one. The
file carries a comment describing the effect and why the skews are multiples
of 8.

Numbers

Measured on a Tensor G3 (Cortex-X3 / A715 / A510, SVE2 at VL=128), one core
pinned per run, comparing __strcmp_aarch64 against the experimental/
__strcmp_aarch64_sve.

These are ratios, and only ratios taken inside a single run. The device
thermally clamps scaling_max_freq to roughly 30% of peak on the big and mid
cores under sustained load, and there is no way to pin it back. Worse, pinning
the frequency is not sufficient: with scaling_cur_freq verified equal to
scaling_max_freq at a stable 880 MHz throughout, the same binary on the same
core produced 5.17 and 2.29 B/ns for __strcmp_aarch64 at 1K in two runs —
2.3x apart. The SVE/scalar ratio inside those same two runs was 0.764 and
0.712, 7% apart. So absolute bytes/ns from this device are worthless and I am
not quoting any; the within-run ratios below are what it can measure.

Because variants run in a fixed order within a section, I ran every core a
second time with the order reversed and only kept sections where the two
agree within 10%:

X3 A715 A510
sections passing the order control 4/6 3/6 6/6
SVE / scalar in those sections 0.73 - 0.94x 1.14x, 0.99x, 0.98x 0.44 - 0.64x

So the experimental SVE strcmp is behind the scalar one on both the little and
the big core and roughly at parity on the mid core. That agrees with the
decision in 316ccb3 to treat these as experimental — the benchmark is not being
proposed to argue otherwise, it is just the first thing I pointed it at. The
sections that failed the order control are exactly the ones where the SVE
variant appeared to win by a wide margin, which is the reason the control is
worth running on a phone at all.

Why I looked at this

In the discussion on #65 you noted that strcmp and strncmp could be improved
for large strings because they use scalar instructions that only process 8
bytes at a time. That is still the case on v26.07: L(loop_aligned) and
L(loop_unaligned) both load 8 bytes into GPRs and use the SWAR zero test,
while strlen.S reads 32 bytes per iteration with ldp of two q registers and
memcmp.S declares Advanced SIMD.

Since this device cannot hold a clock, I measured that one in cycles rather
than in time. A separate driver runs a single routine over a fixed 1K input
under the PMU, at N and 2N iterations, and subtracts, so process startup drops
out and the result is frequency independent. Mutually aligned, L1 hot, which is
the best case for every routine here:

cyc/byte X3 cyc/byte A715 cyc/byte A510 ins/byte (all three)
__strlen_aarch64 0.049 0.067 0.677 0.224
__memcmp_aarch64 0.061 0.086 0.447 0.346
__strcmp_aarch64 0.207 0.273 0.535 0.907

strcmp costs 3.4x (X3) and 3.2x (A715) the cycles per byte of memcmp, which
lives in the same directory and does have a SIMD main loop. On the in-order
A510 it is only 1.2x, so the headroom is mostly on the out-of-order cores.

The instruction counts are the same on all three cores to within 0.2%, as they
should be, and they line up with the source: 0.907 instructions/byte times 8
bytes per iteration is 7.25, against the 7 instructions of L(loop_aligned)
(ldr ldr sub orr bics ccmp b.eq), the remainder being prologue and tail
amortised over 128 iterations. That agreement is the reason I trust these
numbers and not the wall-clock ones.

I would like to look at the main loop, but there was nothing in tree to measure
it with, hence this patch first.

string/bench covers memcpy, memset and strlen but not strcmp, so a change to
strcmp.S has nothing to be measured against.

Six sections: random equal and random mismatch, using the SPEC2017 length and
alignment distributions already used by bench/strlen.c, plus small and medium
fixed sizes. Each of those is run both mutually aligned and mutually
misaligned, since strcmp.S selects L(loop_aligned) or L(loop_unaligned) on
whether the two operands share a load offset.

The operand buffers are skewed by 64 and 128 bytes. Page aligned, equally
sized buffers land an exact power of two apart, which made the mutually
misaligned section measure faster than the mutually aligned one; the skews are
multiples of 8 so the mutually aligned main loop is still what is measured.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for your contribution to Arm's optimized routines. Contribution to this repository does require a signed contributor's license agreement. See contributor-agreement.pdf for details

@TamarChristinaArm

Copy link
Copy Markdown
Contributor

On formatting: the file follows bench/strlen.c rather than the .clang-format
added in 36584f3, because that commit did not reformat the existing files and
the two disagree — clang-format would rewrite 131 lines of bench/strlen.c and
218 of bench/memcpy.c. Say the word and I will switch it to clang-format
output instead.

No please keep the existing format. The provided clang-format was meant more as a helper for people using IDEs, but the end format is whatever the file already does.

I'll add an entry to the readme to clarify this.

@alanhc

alanhc commented Aug 31, 2026

Copy link
Copy Markdown
Author

Thanks for clarifying. The file keeps the existing format then.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants