Nice result! About 5 years ago, I did the same technique of frame dump->process->join but with a custom algorithm instead of a NN for the process stage. A more manual effort to tweak the filter, but it was fun and came out nice.

I got a new GPU recently and have been doing mostly text RNN these days, but it's fun to look back on this occasionally:

http://binarymax.com/reality_remix.html