This is correct. Chandler Carruth gave a talk about unfortunate pessimizations imposed by the ABI. Where the call is inlined, that stuff is optimized out, but functions that take a moved unique_ptr or pimpl object -- taking ownership of a heap object -- tend not to be inlined unless they mostly just pass it along to one that isn't inlined.
So, the cost of moving a smart pointer onto the stack and, presumably, off again to someplace less ephemeral might matter, on a critical path. But it would be a mistake to exaggerate this: we are talking about a couple of unfortunate ordinary memory operations, not traps, function calls, or atomic synchronizations, when you might have hoped to be using just registers. If you are already touching memory -- something pretty common -- it is just more of that.
So, the cost of moving a smart pointer onto the stack and, presumably, off again to someplace less ephemeral might matter, on a critical path. But it would be a mistake to exaggerate this: we are talking about a couple of unfortunate ordinary memory operations, not traps, function calls, or atomic synchronizations, when you might have hoped to be using just registers. If you are already touching memory -- something pretty common -- it is just more of that.