False positives (people who are legitimately using AWS credentials from their phone for some reason, or somebody who is legitimately using AWS credentials from their computer but with an incorrect useragent for some reason) would cause an inconvenience as time is wasted to inspect it, but ultimately little harm would be done.

False negatives (improperly using AWS credentials but with a useragent that looks reasonable) would not be a deviation from the status quo.

You don't need 0% false negative and false positive rates to make this sort of sanity checking worthwhile. Even if you only find a few of the many instances of improperly used credentials, you're better off than if you had done nothing.

(Of course there is the issue of correlating misused credentials with the specific application that is misusing them. I don't know how that is done if they are basing their investigation off of useragents.)