"It'd be good if I could configure my server to keep quiet until the client identifies itself as an SSH client. I run it on an unusual port and it gets scanned frequently."

This is unpopular, but you could implement port knocking.

Now the rest of the world doesn't even see your sshd - on any port. I love the idea and have implemented it everywhere that it's practical.