You'd be amazed how many of the accused said things very similar to what you're saying, right up until the point when they themselves were accused. It's not simple at all, people who claim it is are missing a great deal of nuance and, oftentimes, merely projecting this attitude in the hopes of masking their own transgressions.
Can you point to any of the recent high profile allegations where the accusations are of behavior that was ambiguous? They almost all seem to be totally clear cut cases of inappropriate actions or worse. These aren't mild flirting with a colleague.