I don't know much Japanese, but yeah that's one of the ideas that comes up immediately. Definitely not my place to say if it's too verbatim, the one I've heard several times is muzukashii / 難しい. I think it's hard to translate because just 'no' loses part of it, "no thank you (but I didn't actually say no)". So it sort of develops into the translation having a coded meaning, where difficult things are 'polite' refusals.