For fair comparison, I'm thinking of the surfaces which are in contact with the outside world - bare hands, and the soles of one's shoes.
Now, you don't wash your shoes, but investigations of handwashing show that this is usually inadequate, too.
As for 'dirty', let's use as a measure the (estimated or measured) amount of stuff on each carrier surface which might be harmful to human health. Particular bacteria, viruses, noxious substances, et cetera.
Inspired by a discussion on public transport, and admonitions not to sit on floors in public places or place feet on seats - is this worse, or just as bad as, or actually better than grabbing handrails, I wonder...?