The problem is that the difference in optical/NIR brightness (apparent magnitude) between a long-period comet core that's going to hit us in 1000 years, and a long-period comet core that's going to hit us in six months, might be a factor 10^12 (magnitude 10 vs magnitude 40) or worse. Normally brightness drops off with distance squared for light sources, but comets without any tail or halo aren't emitting all that much light, they're reflecting it, and (except for a very brief period) they're about as far from us as they are from the sun. This means that brightness drops with distance to the fourth power. Cometary tails also only offgas a significant amount near the sun. Comet cores are expected to be extremely dark / low-reflectivity due to space weathering producing a carbon coating not unlike chimney-creosote.

You can fight this a bit by working in the thermal infrared, which you really need a specific sort of space telescope for. But long-period comets and hyperbolic impactors will be a probabilistic threat for the foreseeable future. I would say "Be thankful that they're so rare", but the data from observatories like Rubin on these bodies during points of their orbit where they're close enough to the sun to actually detect, is necessary to statistically characterize their existence with any confidence.