As I recall that was a claim of a byte for byte copy. Google claimed there was no other meaningful way to use the API but the court said that copying the file itself was a step too far.
What means it's protected by copyrights. Fair use on the US is a murky thing that only large corporations have the legal power to bet on.
But the sibling saying it's about the organization of the files may be on point. I remember it being about the organization into packages, but I can easily be wrong here.
That was what every single person thought right until Oracle won a suit against Google for use of the Java's API.