Some positions have inherent dangers. Spending hours per day on the road is a much bigger risk factor than going into any specific area for a drop-off.

Also nobody is forced to do so. You can decline rides. There might be penalties after too many cancellations but it was a much better system than taxis, which were mandated by law but had no oversight and so pretty much always declined for the slightest of reasons.