Ultra wealthy literally means "beyond wealthy". A double digit percentage of the population, maybe 30-50% belong to the non-laboring class.

If I was talking about "ultra obese" people, you wouldn't assume I was talking about everybody who has a couple of extra pounds?