If your vim is compiled with the correct flag for GUI operations (I forget which), you can yank/delete text into the * or + registers to make it available to the system clipboards. This won't bring the line numbers, and lets you avoid using the mouse.
I think it's just called +clipboard. And with the setting "set
clipboard=unnamedplus" Vim will use the system clipboard for
yank, delete and paste by default.
Not only that but if you set your mouse up correctly then you can use the mouse for selections and have it respect window splits too.
That works in tmux too. So you can have use the terminal but have it behave as a GUI app in most respects. For example being able to use a touchpad to quicly enlarge/resize splits or scroll a window can be quite useful.