bench: Add strcmp benchmark - #97
Conversation
string/bench covers memcpy, memset and strlen but not strcmp, so a change to strcmp.S has nothing to be measured against. Six sections: random equal and random mismatch, using the SPEC2017 length and alignment distributions already used by bench/strlen.c, plus small and medium fixed sizes. Each of those is run both mutually aligned and mutually misaligned, since strcmp.S selects L(loop_aligned) or L(loop_unaligned) on whether the two operands share a load offset. The operand buffers are skewed by 64 and 128 bytes. Page aligned, equally sized buffers land an exact power of two apart, which made the mutually misaligned section measure faster than the mutually aligned one; the skews are multiples of 8 so the mutually aligned main loop is still what is measured.
There was a problem hiding this comment.
Thank you for your contribution to Arm's optimized routines. Contribution to this repository does require a signed contributor's license agreement. See contributor-agreement.pdf for details
No please keep the existing format. The provided clang-format was meant more as a helper for people using IDEs, but the end format is whatever the file already does. I'll add an entry to the readme to clarify this. |
|
Thanks for clarifying. The file keeps the existing format then. |
bench: Add strcmp benchmark
string/bench/currently covers memcpy, memset and strlen. There is nobenchmark for strcmp, so a change to
strcmp.Shas nothing to be measuredagainst. This adds one, in the same shape as
bench/strlen.c.I am not employed by Arm and will sign the Assignment Agreement.
What it measures
Six sections, each run for every available implementation:
The length and alignment frequency tables are the SPEC2017 ones already used by
bench/strlen.c.The aligned and misaligned split matters for strcmp specifically:
strcmp.Sselects
L(loop_aligned)orL(loop_unaligned)on whether the two operandsshare a load offset, and those are different loops. The misaligned sections put
the second operand at offset 3 so the unaligned loop is the one being measured.
Variants are dispatched through the
DOTESTmacro withRUN/RUNA64/RUNSVE, so every call site is a directbl;objdumpon thebuilt binary shows no
blrinmain. Adding an SVE2 implementation later is aone-line change to
DOTEST.The 32-bit variants are guarded with the same availability check as
string/test/strcmp.c, since__strcmp_armand__strcmp_armv6mare notavailable on every 32-bit target.
On formatting: the file follows
bench/strlen.crather than the.clang-formatadded in 36584f3, because that commit did not reformat the existing files and
the two disagree — clang-format would rewrite 131 lines of
bench/strlen.cand218 of
bench/memcpy.c.Buffer placement
The two operands must not sit an exact power of two apart. Three page
aligned, equally sized buffers get placed exactly 8192 bytes apart, and with
both operands then at the same offset the mutually misaligned section came
out faster than the mutually aligned one, which is backwards — the aligned
loop does two loads per iteration, the unaligned one does three.
Skewing the second and third buffers by 64 and 128 bytes fixes it. The skews
are multiples of 8, so the mutually aligned main loop is still what gets
measured. Taking the ratio of the two sections within a single run, so the
result does not depend on the clock, at 4K:
I have not chased the exact mechanism and would not want to assert one. The
file carries a comment describing the effect and why the skews are multiples
of 8.
Numbers
Measured on a Tensor G3 (Cortex-X3 / A715 / A510, SVE2 at VL=128), one core
pinned per run, comparing
__strcmp_aarch64against theexperimental/__strcmp_aarch64_sve.These are ratios, and only ratios taken inside a single run. The device
thermally clamps
scaling_max_freqto roughly 30% of peak on the big and midcores under sustained load, and there is no way to pin it back. Worse, pinning
the frequency is not sufficient: with
scaling_cur_freqverified equal toscaling_max_freqat a stable 880 MHz throughout, the same binary on the samecore produced 5.17 and 2.29 B/ns for
__strcmp_aarch64at 1K in two runs —2.3x apart. The SVE/scalar ratio inside those same two runs was 0.764 and
0.712, 7% apart. So absolute bytes/ns from this device are worthless and I am
not quoting any; the within-run ratios below are what it can measure.
Because variants run in a fixed order within a section, I ran every core a
second time with the order reversed and only kept sections where the two
agree within 10%:
So the experimental SVE strcmp is behind the scalar one on both the little and
the big core and roughly at parity on the mid core. That agrees with the
decision in 316ccb3 to treat these as experimental — the benchmark is not being
proposed to argue otherwise, it is just the first thing I pointed it at. The
sections that failed the order control are exactly the ones where the SVE
variant appeared to win by a wide margin, which is the reason the control is
worth running on a phone at all.
Why I looked at this
In the discussion on #65 you noted that strcmp and strncmp could be improved
for large strings because they use scalar instructions that only process 8
bytes at a time. That is still the case on v26.07:
L(loop_aligned)andL(loop_unaligned)both load 8 bytes into GPRs and use the SWAR zero test,while
strlen.Sreads 32 bytes per iteration withldpof two q registers andmemcmp.Sdeclares Advanced SIMD.Since this device cannot hold a clock, I measured that one in cycles rather
than in time. A separate driver runs a single routine over a fixed 1K input
under the PMU, at N and 2N iterations, and subtracts, so process startup drops
out and the result is frequency independent. Mutually aligned, L1 hot, which is
the best case for every routine here:
__strlen_aarch64__memcmp_aarch64__strcmp_aarch64strcmp costs 3.4x (X3) and 3.2x (A715) the cycles per byte of memcmp, which
lives in the same directory and does have a SIMD main loop. On the in-order
A510 it is only 1.2x, so the headroom is mostly on the out-of-order cores.
The instruction counts are the same on all three cores to within 0.2%, as they
should be, and they line up with the source: 0.907 instructions/byte times 8
bytes per iteration is 7.25, against the 7 instructions of
L(loop_aligned)(
ldr ldr sub orr bics ccmp b.eq), the remainder being prologue and tailamortised over 128 iterations. That agreement is the reason I trust these
numbers and not the wall-clock ones.
I would like to look at the main loop, but there was nothing in tree to measure
it with, hence this patch first.