This is where EU's data protection law comes into play. When someone agreed to letting the app look at their phone book, the person isn't giving the app permission to store that forever. In theory, this sort of action should be illegal under EU law.