In an FPS you don't hold the mouse button down to change your camera. In an FPS the cursor _is_ the camera. But here, you're literally clicking and dragging, and dragging actions always go the other way.
I agree. If you use the mouse to point, then the camera should move in the direction the mouse moves. If you use the mouse to grab the world and move it, then the world should move in the direction the mouse moves.
A-Frame defaults the desktop drag (which acts as a preview for VR experiences) to dragging the camera. For static scenes such as panorama, it can feel weird, but A-Frame does have options now to reverse the drag direction. But when you're building 3D scenes and using WASD, it feels more natural to drag the camera.
Pointer lock should be done, but we count the desktop drag as a non-invasive preview mode, but experiences can implement pointer lock if they need.
