Speedup/slowdown audio filter - #2738
Open
Davey-Hughes wants to merge 15 commits into
Open
Conversation
Davey-Hughes
force-pushed
the
speedup-audio-filter
branch
4 times, most recently
from
September 3, 2026 23:36
a2c9755 to
4d2e7d6
Compare
the SPU keeps emitting at its real-time rate when the emulator runs off-speed, so at 3x it produces three times more audio than SDL drains. BufferAudio throws the surplus away by shoving the read pointer forward, and nothing throttles the producer because audioSync is skipped off-speed. every discard is a step discontinuity, which is what the crunch is. fit the audio to the output with WSOLA instead, which keeps the pitch. resampling would also remove the clicks but shifts everything by the speed multiplier, +19 semitones at 3x. adds two settings under Audio settings: a toggle, on by default, and a low-pass cutoff that defaults to off. that dialog applies live and doesn't pause the emulator, so the cutoff can be set by ear while fast-forward is running. with the toggle off the audio is untreated at any speed. normal speed takes its own branch in the callback so it can't regress, and slow-mo now expands through the same path instead of replaying stale frames off the end of the buffer. draining from the callback caps usable speed near 4x; the next commit lifts it.
the audio callback runs at a fixed rate, so draining from it could only ever move one ring's worth per period. past roughly 4x the ring overflowed between callbacks and BufferAudio discarded the rest - at 16.7x only a quarter of the generated audio ever reached the stretcher. the emu thread keeps up by construction. that makes AudioTimeStretch single-producer/single-consumer. only WritePos and a generation counter cross the boundary, and the consumer publishes the oldest frame it may still touch so the producer stops short rather than overwriting it. that floor has to include NaturalPos, which trails the nominal position and at a large hop sits below the search window - tsan caught that one. SetOutputSkew moves off the callback too. it only depends on targetFPS, so calling it per callback was pointless, and it raced SPU::Reset through blip_set_rates.
standalone, not wired into the build - the build line is at the top of the file. reports where audio is lost at each speed, what the stretcher costs, and runs two threads across the SPSC ring with a tearing detector. build it with -fsanitize=thread for the race check.
Davey-Hughes
force-pushed
the
speedup-audio-filter
branch
from
September 3, 2026 23:51
4d2e7d6 to
4d60663
Compare
the accumulator only carries half a window after the first hop following a reposition, so the first hop after every BeginSession or short write emitted the rising half of a Hann window: a fade-in from silence at every engage. charge the accumulator with one hop whose output is dropped instead, so the first frames handed out are already at full amplitude. the flag is named output rather than emit, which Qt defines to nothing.
the arrival average was seeded from the requested speed at every engage. the request is a ceiling: on a Steam Deck a 3x cap reaches 1.7 to 2.1x, so every hold started up to half wrong and, at the 0.05 gain, spent most of a short tap walking the estimate down to the truth. uncapped is worse - curFPS is 1000 there, nowhere near what the machine does. bank the measured speed at release, before the average is discarded, and seed the next engage from it, still capped by the request when there is one. the first hold after a load is unchanged; every hold after it starts from what the last one reached.
the biquads were redesigned once per block and the whole block filtered with the new set. a transposed direct-form biquad's state encodes its past under the coefficients that produced it, so replacing them in one go leaves the two inconsistent and the filter rings, once per block while the cutoff moves. on a 200 Hz tone stepped from wide-open to 2 kHz the largest sample-to-sample jump was 7.9x the steady-state one; interpolating from the old set to the new across the block brings it to 1.0x. the block lands exactly on the designed set afterwards, so rounding in the interpolation cannot accumulate.
the reference is divided by the emulation speed, so the top of the range bounds how gentle the filter can be: at 24 kHz the mildest setting was a 12 kHz cutoff at 2x on a 48 kHz device. run the range to 48 kHz so the reference can sit above Nyquist, as RetroArch does. wide-open still clamps the applied cutoff, so nothing above the device's Nyquist is ever designed. zero reads as off too, rather than clamping to the strongest filter. a config saved with the old sentinel, 24000, now reads as a 24 kHz reference rather than off.
at release the stretcher's unread reserve, about 85 ms, is dropped and the raw FIFO takes over wherever it stands; at engage the stretcher starts behind the raw path by the same reserve. either way the output jumps mid-waveform, and the two sides cannot be joined seamlessly: they are the same material at different points. cross-fade 256 frames at equal power from the level the stream stopped at into the new path, smoothstepped so the gain leaves and arrives with zero slope, with a shallow dip at the midpoint that attenuates whatever discontinuity the handover carries. a dip is less audible than a click. the edge is seen in audioFinishBuffer, on the first buffer after the engaged flag flips, so both paths share one site.
pausing stopped the SDL device wherever the waveform was and brought it back at full level: a step at both ends, heard as a click. an underrun in the stretched path faded the last sample out as a constant, which is a decaying DC offset rather than a decaying sound, and reads as a thump; in the direct path it cut straight to silence. keep a copy of the last frames played. when the stream goes down, find the period that best matches the last few milliseconds - scored by distance, since a correlation is blind to level - and repeat it under a raised cosine, as a packet-loss concealer does, with a join correction so the tail opens on the value the stream stopped at. when it comes back, ramp the first frames in, after dropping what the pipeline was still holding. RetroArch measured the pause step going from 654 to 7 on a tone with the same tail. the tail is played by the callback, so audioDisable asks for it and waits, bounded at four callback periods, for the callback to report it spent before pausing the device. an empty or short read is a stream end without notice and gets the same tail; slow-mo with the stretcher off keeps its hold, since that path is untreated by design. audioEnable arms the ramp in.
a load splices the game's own audio, and marking the discontinuity for the stretcher only stopped its search from spanning the jump: the splice itself was still played, at normal speed straight from the SPU ring. an undo and a reset are the same jump. take the stream down on the pause tail before the jump and bring it back on the resume ramp after, so the splice and whatever gap the load leaves both land in silence. a stream already down - the jump was made while paused - is left alone, so the ramp back up stays with whatever took it down. the SPU output ring is not part of a savestate, so what it still held from before the load is drained rather than played. drained whether or not the bracket ramped: a load made while paused would otherwise bring up to a frame of pre-load audio back behind the resume ramp and splice at full level. reset needs no drain: SPU::Reset re-inits the ring, and the audioEnable that follows it brings the stream back.
audioSync() waits for the SPU ring to drain below the device buffer, but it only checks that a device exists, not that it is running. A paused device drains nothing, and the two places that signal audioSyncCond - the audio callback and audioDrainSPU - are both silent while it is paused, so a frame run against a paused device spends the whole 500 ms timeout with whatever is waiting on the emu thread blocked behind it. Frame stepping pays this now; anything that pauses the device while emulation still runs would pay it too.
Booting a ROM or the firmware rebuilds the console and resets it while the audio device is still playing, cutting the previous game off mid-waveform. Worse, the callback dereferences nds across the delete in updateConsole. Ending the stream the way a reset or a state jump does is not enough here: the tail leaves the device playing, so the callback still reads the SPU. Pause the device instead, which closes that window; the following msg_EmuRun brings it back. A failed boot sends no msg_EmuRun and leaves emulation running, so re-enable there.
A frame step runs a frame with the device paused and the emu thread only drains the SPU ring while off-speed, so every step leaves roughly 800 frames behind. Three of them fill the ring, and the resume plays 43 ms of stepped audio straight through the mute and the ramp. Drain in audioEnable rather than only after a step: the tail already discards source frames across the boundary, so nothing in the ring is a continuation of what was heard, and this covers every path the stream comes back on. The same drain audioJumpEnd was doing, now shared.
The similarity search re-read the natural-continuation window through the ring mask for every candidate, though NaturalPos is fixed for the whole search: around 519 candidates by 128 samples of redundant masked loads per hop, on the audio callback thread. Copy it once and index it linearly. Same values in the same order, so the offset chosen is unchanged.
The search called HistAt twice per sample, and each call is a modulo, for around 119k iterations on the audio callback thread. It runs on any sustained short read, not only at a pause. Sum the reference window's channels once, since it is the same for every candidate, and step each candidate's index backwards with a wrap instead of recomputing it. Same values in the same order, so the period chosen is unchanged.
Davey-Hughes
force-pushed
the
speedup-audio-filter
branch
from
September 9, 2026 14:49
7f97562 to
a903fd6
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
After working on this other PR for sound settings, I realized there were probably better audio enhancements that could be made when doing ff and slow-mo, so I worked on a two-part audio filter that applies when the game is fast-forwarded (or unbound FPS) and slo-mo'd. The PR is split into 3 commits, with the last 2 able to be dropped if not desired.
The first commit is the audio filter itself, and should work properly at up to 8x speedup. There's both a WSOLA stretcher to preserve pitch when stretching, and a Butterworth low pass filter to help cut some of the crunchier noises at fast speeds. This also adds a menu item so you can toggle the feature on/off and decide the frequency cutoff for the low-pass filter.
The second commit is some threading code to make this actually work "correctly" at higher than 8x speed. At more than 8x speedup, the audio output actually skips, though in practice this is kind of hard to hear. I tested this using the tsan build though the emulator runs very slow under these conditions, so it's hard to exactly make sure everything is correct, though I think the theory is right. It deserves some review though.
The last commit is some benchmarking which helped me figure out some of the original implementation details and why the emu-thread implementation was actually needed at all for correctness, though this can be dropped or wired into the CMake build if desired.
Overall I think even just the first commit which is basically pure algorithm implementations help a lot when playing games at speedup, especially the common case of playing pokemon games fast for grinding or whatnot. I'm happy to make adjustments where needed to improve this.
License Provenance
I'm working on this same feature in RetroArch as well. It's more involved however the core is the same WSOLA and butterworth filters.