I have a little trick that makes nvm not impact your overall startup time.

First, put the version you commonly use in your path

  export PATH="${PATH}:/home/victor/.nvm/versions/node/v6.5.0/bin"

Then, instead of sourching nvm when your shell starts, do it when you call an alias instead.

  export NVM_DIR="/home/victor/.nvm"
  alias n=". $NVM_DIR/nvm.sh" 

Now, you have a node+npm version by default, and you can do a quick flow if you want to change

  $ n
  $ nvm install 6.6.0
  Now using node v6.5.0