It's an interesting position but one with unpleasant implications. Do you think it is morally wrong to work for, say, a terrorist organization planning a chemical weapons attack if all you do is manage procurement for them — you don't make chemical weapons or use them, you just make phone calls and manage a few spreadsheets.
Also, do you have the same feeling for the reverse situation? If instead of being an employee, you're the boss. You're aware that your employees are doing something morally wrong and you do nothing to stop them, but you aren't doing it yourself. Do you have any responsibility? In the ethics of war, there is the concept of command responsibility[0], where leaders are responsible for the actions of people under their command. Do you think that is a bad doctrine because the leaders aren't the ones shooting people? No one dies through their individual actions even if their subordinates commit war crimes.
