Great write-up and examples. We use a similar combination of Tornado, Gearman and Redis for our own crawling with mongodb for storage. I realized just now why your company name was familiar - we're also on the 3rd floor in the PV building.