The most elegant, protocol-agnostic solution I've seen for doing this is 1st to throttle individual clients to their allocated BW (i.e. so that one client doesn't gobble up an entire shared link), and 2nd to simply kill any long-term (12hrs+) TCP/IP connections that are not over a recognized/allowed L7 protocol. The 2nd is very effective at starving unwanted bit torrent usage.
This method is in use right now by the guys who hang 5.8GHz antennas on grain silos, i.e. rural WISPs.
simply kill any long-term (12hrs+) TCP/IP connections that are not over a recognized/allowed L7 protocol. [This] is very effective at starving unwanted bit torrent usage.
I don't see how. BitTorrent will just reconnect to a different peer and continue.
I like first one. However, while the second one is fair enough for the example of rural WISPs - if my ISP starts killing connections as a matter of common practice then I'd be looking for a new ISP.
top) VoIP
2nd) Video Chat
3rd) IM
4th) Streaming media
5th) Web Browsing
6th) Large file transfers/torrents etc.
The idea would be to make it protocol "agnostic", and base it on the real-time requirements.