nor Windows Explorer.

For the File Explorer it's HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Explorer\Advanced Value PersistBrowsers set to 1.

You don't need to manually edit the registry for this. There's a checkbox in Explorer's settings.

Thank you! This reminds me of the old annoyances.org that listed a bunch of little tricks to improve your Windows 95 experience.