The naming fits if you think about the implications of the mark on the flow of a sentence. A colon is a pretty hard stop when I read, while a comma is barely a pause. A semicolon is in between the two; you don't stop as long as the colon, but you pause a little longer than a comma (did I use it correctly? ;P )
I pause longer for colons than for periods. Colons are less common and often join what are really two sentences, meaning something like "read this sentence in the context of what was said in the first."
To me, exclamation mark, full stop and question mark are of equal precedence; they are higher than colon, which is higher than semi-colon, and that is higher than comma.
