oh this is neat, it's like adding an additional dimension to the Wasserstein distance / optimal transport problem. Well, in the sense that you are using marginals as constraints. Kantorovich won a nobel prize for this kind of stuff, so it's definitely hard.