This is an extremely delicate issue in any large organization. Skipping one level basically means: "Let's see what important things my report is not telling me". If you are asked for such a meeting, anything you say could be used against your direct manager. You need to know the office politics well before speaking anything that could have even small impact on anything. I saw people freaking out for mentioning minute stuff "behind their backs."

With a small team, it's very different - but usually the hierarchical structure is much more horizontally aligned so this is not much of a problem.

I must have worked at organizations with flatter structure than the others here. Talking with your skip level manager is an ordinary thing, if he has some opinions on the things you are responsible for why would he talk to your manager and not just go directly to you? Managers are mostly for bookkeeping purposes anyway, they shouldn't tell you what to do on a daily basis. At least they didn't in the 3 places I worked.