I'm not the expert on this, but I was curious so I looked into it a bit. I don't believe it required significant resources or crazy algorithms, but a clever selection of physics criteria to optimize. It was a ~20-dimensional Neumann boundary value problem [1], and a code named NESCOIL was used to figure out the shape of the coils that would produce the required magnetic field [2], which it did using Fourier series.
By the way, the unusual shapes of the coils can be understood intuitively from this picture: https://imgur.com/a/Bq3ABfQ. A plasma needs to be confined with a magnetic field in order to be heated to extreme temperatures, and a toroidal field (produced by the currents in the red coils) is unstable due to particle orbit drifts. You need to add a twist to the field for it to be stable (using the green coils). But if you unroll the surface of the torus, you can approximate the currents in both green and red coils using the discrete blue coils, and they're easier to build.
By the way, the unusual shapes of the coils can be understood intuitively from this picture: https://imgur.com/a/Bq3ABfQ. A plasma needs to be confined with a magnetic field in order to be heated to extreme temperatures, and a toroidal field (produced by the currents in the red coils) is unstable due to particle orbit drifts. You need to add a twist to the field for it to be stable (using the green coils). But if you unroll the surface of the torus, you can approximate the currents in both green and red coils using the discrete blue coils, and they're easier to build.
[1] https://aip.scitation.org/doi/abs/10.1063/1.860481 [2] http://iopscience.iop.org/article/10.1088/0029-5515/27/5/018...