I went to a very similar school, so I understand what you're saying. I also had similar feelings at the time. But having studied the problem and being removed from it now for many years has made me think more coolly about the topic.
Maybe there could be student groups where everyone had to be silent?
Seems that'd benefit most students
If each class had two rooms, one silent "library room" and one where it was ok to talk a bit. And the students could choose freely.
Just that if they made noise in the silent room, a school janitor would have them go to the other room instead
And the fast students could get an unlimited supply of more and more harder and harder study books. Regardless of in which room they were. (Note that I didn't write "gifted" -- maybe some students get lots of things done, are comparatively fast, primarily because they work a lot.)
