You could probably also do something similar to the article, providing your document is XML-valid, using XSLT. This would keep the advantage that you just load your document in a browser (having added a temporary stylesheet link either manually or dynamically) but would support more powerful validation. It could be made into a browser extension so you're just a one-click away from validation with in-page feedback.
Of course, you can also validate XHTML using the provided DTDs if you take out the SGML-specific bits, using a (DTD-)validating XML parser (or indeed use XSLT as you say). But SGML has tag omission/inference, empty elements/HTML "void" elements, attribute short forms, unquoted attributes, Wiki syntaxes/markdown, and other features desired or even required for mere parsing mainstream HTML. As to using XHTML, I'm not sure it has a bright future. Last I've worked with it was like ten years ago (xsl-stylesheet processing instruction etc. on IE6; newer browsers don't seem to support it anymore).
FYI: SGML is way more powerful than XML, and its grammar formalism is identical to that of XML because XML was designed as a simplified subset of SGML. Though Relax NG, and also XML Schema, can theoretically express richer grammars than SGML, they can't be used for HTML due to lack of tag inference and other short syntax features, and the features that Relax NG has over SGML benefit data-oriented rather than document-oriented use cases (eg. the use cases that JSON and other serialization formats are arguably better suited for than markup in the first place IMO).
sgmljs.net can run (also) in the browser, so can also validate HTML5 in the browser.
I've had success with xhtml and xslt processing on the server-side. The fact that it's xhtml is incidental; that it's xml-valid means I can use a wealth of established tools to modify the content, and easily write my own. I don't mind too much about xml support in the browser, although I've found xslt to still be supported in chrome - e.g. for rendering RSS feeds as HTML.
Your comment sent me down a multi-hour rabbit hole.. thank you. I ended up reading a fair bit about Marcus Reichardt's initiative toward a SGML DTD for HTML5.1. However, i was unable to find a place where a tool or said DTD was available for download. Do you happen to have a link?
For now the full DTD grammar can be downloaded from the "Restrictive DTD" tab link of the full paper eg. [1] via copy/paste. I intend to be make it accessible on a permanent URL once I get more feedback/review comments. It can be processed using the venerable OpenSP/OpenJade package (install it with "sudo apt-get install opensp" on Debian/Ubuntu or "sudo port install opensp" on Mac/MacPorts; don't know the MacBrew equivalent, though). I'm using my own sgmljs SGML system (public release upcoming/pending).
[1]: http://sgmljs.net/blog/blog1701.html