At one time, this was my product.. and oof, this one still hurts.
Section 5.1 of the OIDC spec says the standard claims can be in the ID token and/or at the Userinfo endpoint. Further, Section 2 says "ID Tokens MAY contain other Claims."
Unfortunately, one of the most common use cases for the ID token was to add someone's Groups, usually from AD. We had a number of customers with users who had a LOT of groups. I remember one where their users were in an average of 700 groups and one user had ~9000. These groups could be anything from the AD group created yesterday for a new app to that group from 15 years ago that no one wanted to delete just in case. This made for gigantic tokens.
Anyway, to address this scenario, someone at Okta came up with concept of the "fat ID token" and the "thin ID token". The "thin" would always come back with the access token on the inital request and the "fat" would only be available via the userinfo endpoint where we weren't limited by payload sizes.
We were using AWS Cognito and had to make a "pre-token-generation lambda" to filter out only the AD groups we cared about. We had a huge map of AD group IDs to our internal group names (multi-tenant application, so each client had a different AD group ids) so we filtered out the ad-groups and added a new custom claim with our internal names.
Fun that one time where we gave admin access to some people that shouldn't have it.
Before we added that map some of our user's tokens were exceeding the limits for AWS Cloudfront cache keys.
Cosign. AAD ID tokens had the same issue, and we'd see tokens with 1500+ guids shoved in there for the group ids. We had (and have) a nasty "too many, go call this API you need a new permission for" outcome there.
Right next to those customers were the ones demanding our tokens always be an exact number of bytes (yeah. Really).
The other challenge was customers who wanted groups to show up in a particular order. Since it's was literally just an array (no keys), it was just a giant alphabetized list.
Then the problem came when people used group count limits. Your group "AppDev" was always fine but "TestGroup7" may not be.. depending on how many groups the user was in and/or how they filtered those groups.
Actually no, that's on the spec for not enforcing they're in the ID token, and only must be available in the userinfo endpoint.