Unfortunately Calibre is pretty bad for reading Japanese novels. Every amazon.jp novel I ripped has issues rendering the vertical RTL text properly. Each "page" is a chapter, and you have to scroll sideways on the page to move in the current chapter. Frequently, scrolling horizontally blanks after the first screen. It's a mess. I had to force my books to render in horizontal LTR via CSS to be able to read them.
Odd. Your problem might be caused by badly formatted epubs? I'm using Calibre 5.0.0 on Ubuntu and I have no problem with a Chinese novel epub in vertical layout. The mouse cursor even turns sideways for selecting text and I can use the mouse scroll wheel without any modifier key to scroll left and right.
That might have been fixed in 5, I haven't used calibre in a while. As for formatting, I rip my collection from my amazon library, so there's nothing I can do about that unfortunately.
Strange, I've used Calibre to convert my amazon.jp purchases and am pretty happy with it. It does convert vertical text into horizontal text by default, but the resulting text is perfectly normal without the problem you described, and you can add CSS to force vertical RTL.
There's some example and the css code they used towards the bottom of this page, if you'd like to take a look: https://webnetforce.net/post-105/
