This mirrors my experiencing gardening in Georgia. We have about half an inch of topsoil there, and below that is pure red clay with close to zero organic matter in it. If you want to have a garden, you add compost. I added a couple thousand pounds of well composted manure, and double tilled it to mix it in down to a couple of feet. The result was that I had loamy soil that grew vegetables like nobody's business.

Put enough compost on a plot and it will turn into a jungle, just like my garden did after I stopped tending it. I was afraid to go in there to pick tomatoes.