I work this way too, and for the last several years I have been looking for software that would let me mark up some code without actually changing the underlying source files (keeping the annotations on the side). Putting your own comments in screws up diffs and all sorts of things like that.
you could try stripping comments before diff'ing. If you have single line comments you can use diff -I otherwise try this: http://freecode.com/projects/stripcmt
