That's true until the lossy compression alters the data in a way that actually makes it harder to represent. As an example, a bitstream that 'corrects' a decompressed .MP3 file to match the original raw waveform data would be almost as difficult to compress as the raw file itself would be.

It wouldn't be a matter of simply computing and compressing deltas at each sample, because frequency-domain compression moves the bits around.