Then just use an rpath with `${ORIGIN}` and ship your .so files along with your binary.

The only time where you shouldn't do this is if your executable requires setuid or similar permission bits, but those generally can only reasonably be shipped through distro repos.