"100ms" probably comes from the long distance telephony investigations into round-trip time. If I remember correctly, 100-ish ms RTT was when people would start to notice the delay, and conversations became very difficult above ~200 ms. As a result, a lot of telephony hardware tries to keep latencies under that limit.

The brain obviously notices far smaller latencies.