Did you have the number configured as forwarding number? At some point I got asked to verify an old forwarding number, but I just deleted it and Google Voice continued to work fine (I mostly use it in VoIP mode instead of forwarding calls).