I was thinking in the context of "field sports", such as hunting shooting etc.

As you say though, both are plausible, and that ambiguity is part of what makes the phrase so interesting.