Well, if we can't agree with the rule of law then what, we're back to a primitive society? I don't think homo sapiens culture is, at this stage, advanced enough to thrive in a primitive community, how attractive that may sound. In a few hundred or thousand years, perhaps?
You are fundamentally mistaken about what "the rule of law" is. The rule of law is not based on "agreement" but enforcement. Even the veneer of agreement is based on enforcement. Historically, this has taken the form of enforced religion. Today, enforced school attendance serves a similar role.