Some of it may be crab mentality. No matter how much you spend on some (but not all) special ed kids, there's no worry they will academically surpass the majority of students. So there's no fear from parents that the spending will mean their own kid gets dragged further down the pecking order.

Spending for gifted children means putting other kids even higher up the pecking order than their own average kid. Who wants to spend a bunch of money to watch their kid get left further behind the front of the pack?