One alternative would be to sidestep this memory-reallocation-and-buffering problem altogether and just use the code that already exists to connect to an ssh-agent, even when only private keys from the local filesystem are used for authentication:

Let ssh spawn an ephemeral ssh-agent just for the duration of the key-exchange and just for this single ssh instance. Kill it once authentication is complete.