Interesting project. In my experience as a re-recording mixer for film the best denoisers currently on the market work as a sort of spectral noise gates: so instead of running one noise gate over the whole audio spectrum, you split it into frequency bands each with their own thresholds. These thresholds are tuned in by “learning” the spectral shape of the background noise itself and usually the user can still modify the wheights with a curve (e.g. if you want to gate high frequencies more).
This has the benefit of producing more natural results while keeping speech understandable. It might interest you, because with a 2000x speed marging you could still crank the algorithm up.
I like the idea of breaking things up by frequency! Luckily my application (radio transmissions) is quite amenable to this naive noise gate, but I can imagine for more complex audio (e.g. if the radio doesn't have a built-in squelch) having tighter control over what triggers the gate would greatly improve quality.
Multi band compressors also help in this domain, choosing which frequency band to attenuate or boost. For multi channel audio, phase becomes important too. Not for nothing is a typical trick in mastering to restrict low frequencies to mono..
I always thought the primary reason for that was that stereo low frequencies, when recorded onto vinyl records, hugely increase the risk of the needle 'jumping' out of the groove and skipping into another one.
This has the benefit of producing more natural results while keeping speech understandable. It might interest you, because with a 2000x speed marging you could still crank the algorithm up.