Ah. I haven’t played around with Pytorch custom layers enough so I am going to give it a try. I was initially trying to do it in keras but Keras was just using tensorflow layers for most operations so I couldn’t tweak the original tensorflow layers through keras easily.
The concept of "layers" is not in fact enforced by pytorch or tensorflow at all. This tutorial is a really nice overview of the levels of abstraction available in pytorch
https://pytorch.org/tutorials/beginner/nn_tutorial.html
