Oddly enough, I always ran into performance issues when running uMatrix. I'd turn off JavaScript by default though and only selectively enable it. That seems like it should speed things up, but I'd often experience the opposite, both on Vivaldi and Firefox.
By default, uMatrix blocks Google Analytics, whereas uBlock Origin stubs it. If both are installed (in my case), uMatrix often wins out. If Google Analytics is blocked, many sites will spend 2 seconds on a white screen waiting for Google Analytics to load.
uMatrix is very low overhead so my first suspicion would be artificial delays introduced when some JavaScript resources are blocked; [1] is an example of this. The artificial delay does not occur in uBlock Origin because it uses a replacement script for the blocked one causing the issue.