I was thinking the same thing. This problem seems extremely well suited for C/C++, and C#/.NET makes it very easy to call directly into native libraries.
I agree this is a totally valid option, trouble is we have no c experts on core team so it would raise the cost of maintaining. Also the code is fairly complex, moving to structs was fairly simple compared to a c port.
Sounds like the perfect opportunity for someone to become a C expert. Something like this just throws up a lot of red flags. If Microsoft changes some of the methodology in its GC or further demotes structs, your solution may fall apart. It just sounds like something that requires tight manual memory management to get right and remain stable.