One of the biggest problems with LTE on most phones nowadays is that you're forced using LTE and CDMA at the same time, with no option to only use LTE, as no carriers support VoLTE (Voice over LTE), only voice over CDMA.
Unfortunately, Google or Samsung haven't included an LTE-only mode on the Galaxy Nexus because of this, so I must pay for a constant CDMA connection with my battery whenever I want to use LTE, even if I'm using SIP over LTE (e.g. Google Voice, so I only need a data plan for data/texting/calling).
I don't use the official Google Voice SIP server as I didn't have a Gizmo5 account before it was shut down. I just use an sipgate.com account added to my Google Voice account. If you're worried about latency, you could try using the undocumented Gmail Google Voice API through GrooVe IP (https://play.google.com/store/apps/details?id=com.gvoip) to get a direct connection to Google Voice.
VoLTE support is on the way. But even so, I'm not sure I'd want an LTE-only device for the next few years. LTE is new enough that there must be areas with CDMA coverage but not LTE.
The big advantage of VoLTE over a straight TCP/IP connection is a packet priority scheme that will give you latency guarantees for VoLTE that you can't get from SIP, so voice quality should be more consistent. Are you getting decent voice quality from SIP?
Unfortunately, Google or Samsung haven't included an LTE-only mode on the Galaxy Nexus because of this, so I must pay for a constant CDMA connection with my battery whenever I want to use LTE, even if I'm using SIP over LTE (e.g. Google Voice, so I only need a data plan for data/texting/calling).