It has very little to do with the relative speeds of sound and wind. Rather, it has to do with the refraction of sound, bending it downwards when you shout downwind, and upwards when you shout upwind. Sound that goes upwards is prone to pass over the heads of the listeners, whereas sound bending downwards tends to reach them rather well. The reason for the refraction is that air moving over the ground gets slowed down, and the closer it is to the ground, the more it is slowed. Sound passing through air gets slowed down or speeded up by the same speed as the air movement, and when the airspeed is not constant across the wavefront of the sound, the path of the wavefront bends in the direction of the lowest speed, much as a vehicle with its wheels in sand on one side and on firm ground on the other side, will turn towards the sand. The refraction of light follows a very similar principles, which is the explanation for mirages and similar phenomena.