We are talking about binary blobs that are bundled with drivers usually and sent to the device at initialization time which means you'd have to individually fix the version of n drivers in the package manager of your distro to avoid thay. This is doable but some people think it is easier to not have your distro shipping them in the first place.