Good point. Perhaps we could store the length of the common prefix from the last time we went left as well as from the last time we went right. The minimum of those two should remain a common prefix with the search key for the rest of the lookup and should therefore be safe to skip during subsequent comparisons.
But that would actually help it because you can store how far down the match is by that point in the tree, and know how few digits you have to actually compare.