Since Heroku runs on EC2, as long as OP kept his servers in the same availability zone as the DB, latency shouldn't be any worse than running it himself. But yes, in general I'd be wary of a database far away from my app servers.
That can depend on whether you can hit the internal IPs, or just the external IPs. If you can only hit the external IPs, you're subject to greater network latency, plus data transfer fees.
