That is an understandable but incorrect conclusion. That should be the case, especially with freedom of movement within the EU, but the German tax authorities are essentially violating those agreements (which will inevitably get challenged in the law courts, but that will take some years). As I said elsewhere, please consult an experienced tax advisor for your particular circumstances as you cannot just figure this out for yourself, unfortunately.