Radioactivity of rocks is caused by only a few elements so is useful to consider the minerals in which those elements are hosted. i.e. the radioactivity is not evenly distributed among the minerals that comprise granite. The elements that contribute most to the radioactivity of granite are U, Th and K. The K is mostly contained in alkali feldspar and mica. The U and Th are contained in accessory minerals, such as monazite and zircon. Monazite and zircon are dense, robust minerals, which means they persist after weathering and go on to accumulate in detrital sediments, and we call the sediments in which they are concentrated heavy mineral sands. Alkali feldspar is much less robust and rarely survives in mature sediments (i.e. quartz-rich sands), instead all the K ends up in clays.

When heavy mineral sands are processed to extract the Ti and Zr ( from rutile, illmenite, zircon), the residual concentrate is rich in monazite. This material comprises the bulk of our easily accessible Th reserves. However, you can't just leave this monazite-rich material lying around heaps, as it creates a windborne radioactive dust hazard, so it gets mixed back in with the other light material. This is a bit of a shame. All that energy and effort is expended to extract the heavy minerals, but as there is no immediate market for the monazite, and it is a liability to keep it in the extracted state, all that work hard work is undone to mitigate the dust hazard and it gets mixed back with the quartz etc.

So it is perhaps somewhat ironic that we mine beaches to get the minerals that are the source of the bulk of the radioactivity of granites.