Libvirt is not needed to use KVM but the purpose has been to provide stability (and an abstraction as you mentioned) over the changes QEMU has had in various releases.
now, the version of KVM I'm using is really, really old, but as far as I can tell, KVM, by itself, does not do any locking. It's pretty easy to start a single guest twice when using kvm by itself, which will irreparably corrupt your instance.
So yeah, I'd strongly recommend that you use libvirt (or some other wrapper that handles things like locking the block devices) if you use KVM.
With xen, on the other hand, it handles that level locking for you out of box, so personally I see no reason to use libvirt. The libvirt devs seem pretty focused on KVM anyhow; Xen support, at least in the past, was pretty poor, so personally, I use the native xen tools for xen.
(I'm not saying this is a reason to use xen instead of KVM; I'm just saying that if you do use KVM, you should also use libvirt.)
Update: Here's a blog related to your very question that was written just a few days ago: http://berrange.com/posts/2011/06/07/what-benefits-does-libv...