My solution was to play modpacks that no sane person would be bothered grinding in order to grief. Svens and Test Pack Please Ignore are what our servers were based on. If you wanted to grief you would uabe to grind for hours before being capable enough to bother anyone.
So in your case and mine it seems the solution was giving serious players tools to defend themselves that were resource or time intensive to defeat, so the griefer wouldn't be bothered.
Getting sort of off subject, but one of the things I liked about this set of mods was that they were entirely server side. It got a bit awkward in that it required text commands in the place of interfaces in a lot of places, but it meant that the difficulty of getting people to join was a lot smaller. I think at the peak it was still fairly small but a healthy 100+ on most days.
My first big bit of software development was a mod that ended up getting used on the server and that's still updated by some of the copycat servers that got created after.
So in your case and mine it seems the solution was giving serious players tools to defend themselves that were resource or time intensive to defeat, so the griefer wouldn't be bothered.