The only thing that sort of scared me off of customizing emacs is that if I, say, SSH into a remote box or go to another co-workers computer, my customizations won't be there. That's why I do my best to stick with "out of the box" vim defaults (and vi defaults if possible): I know they'll always be there, wherever vi is.
Well, that's the drawback to emacs. It's also why most people don't change the default keybindings too much: some functionality may be missing, but muscle memory won't betray me if I ssh into a fresh emacs setup.
However, there are some ways to combat this.
Firstly, you can keep your .emacs.d on github, and pull it down to whatever machine you're working on. Indeed, this is what many emacs users advocate.
Secondly, many emacs users will ask: Why are you launching emacs over SSH at all? Instead, use TRAMP or SSHFS, bring up the folders under dired if you need to see the folders, and edit the files with your local emacs.
You can become compotent in emacs without customizing it, but the customization is a big part of why emacs is great.
The only thing that sort of scared me off of customizing emacs is that if I, say, SSH into a remote box or go to another co-workers computer, my customizations won't be there. That's why I do my best to stick with "out of the box" vim defaults (and vi defaults if possible): I know they'll always be there, wherever vi is.