In a sense it is literal, if "world" is understood not as "planet earth" or "this realm of existence" but instead as a social circle. e.g. "He is from the software world," doesn't mean "He is from a world made up of software," but instead "He works in and is surrounded by people who develop software professionally." In that sense, a lot of these people are (literally) living in a world that is socially, physically, and even conceptually separate from those of lesser means.
Is that a useful understanding of "world" in this case? I think it's important to remember that we share the same planet, the same economic system, the same borders and system of visas. This way, their inordinate privileges stand out in sharp relief. It seems almost apologetic to say they live in a different world.
Classic case of semantic drift, as "literally" now means "figuratively", but with emphasis. Try "virtually", "practically", or "all but".