At least in my jurisdiction, it's not simply a matter of the poll worker "marking off" your name: they find your name in the book, then they push the book across the table to you so you can make your own signature in the box next to your name.
If someone is attempting to commit in-person voter fraud, they'll need to sign the name that matches the line they put it in. If they're just mistakenly signing on the wrong line, that will, first of all, probably be obvious to both them and the poll worker immediately, and even if not, it will probably be obvious to anyone coming along behind and checking if there's a discrepancy. (Unless the signature is well and truly illegible, and if that's the case, they can check it against the signature from the previous election for that person.)
If someone is attempting to commit in-person voter fraud, they'll need to sign the name that matches the line they put it in. If they're just mistakenly signing on the wrong line, that will, first of all, probably be obvious to both them and the poll worker immediately, and even if not, it will probably be obvious to anyone coming along behind and checking if there's a discrepancy. (Unless the signature is well and truly illegible, and if that's the case, they can check it against the signature from the previous election for that person.)