It will if the air is moving sufficiently fast over it, e.g in combination with the fan cooler. Or you can have a bucket with a mix of water, ice and salt, and point the fan to blow over it (we did this in army and it actually works fine as a poor man AC alternative).

Even if it doesn't evaporate, it will still absorb body heat, briefly. Similarly, if you are in a humid area, sitting in a bath / some body of water will help a lot. Don't underestimate the heat capacity of water.