Making sure the referer is correct is good enough. Other sites can't without permission send requests with a referer belonging to your site, and that won't change, unless the spec changes, and that's very very unlikely.