Is it? Considering that the 13th amendment was pushed through by the Union in the closing months of the Civil War, which itself was fought by conscripts, it's hard to justify an interpretation that outlaws conscription.
The majority of Civil War troops were volunteer. The Civil War draft, an issue at the end of the war, was never challenged on constitutionality. It should be noted that the Conscription Act of 1863 was considered unconstitutional by the populace and resulted in riots.
It's questionable whether many would support it given the way it worked is you paid a $300 fee in order not to go. Usually a poor recent immigrant to the US would be sent in your place. Do you think this scheme would fly these days and be considered legitimate?
Of course that scheme was unfair, which is one of the reasons they did it differently for World War I and World War II. Also, it arguably violates the equal protection clause of the 14th amendment, but that wasn't in place until after the Civil War.
US law actually designates most adult male citizens under the age of 45 as members of the "unorganized militia", and the Constitution explicitly gives Congress the power to call up the militia for active military service. Defending your country when called upon is also generally seen as one of the duties of citizenship. It occurs to me, offhandedly, that universal conscription would both satisfy the equal protection clause and reinforce the notion that military service was one of the duties of citizenship.