When you deploy your code to AWS lambda or EC2 the code can simply access the appropriate secret stores as dictated by the IAM policy. If you haven't bought into AWS as a whole you're right that there's no good reason to use secret manager.