A vacuum is not "cold" in quite the same sense in which cold air or a cold object are cold - a vacuum cannot be warmed up - there just is nothing there to be warmed up.
There is very little to be warmed up on the way from the Sun to the Earth, so the IR rays travel without much trouble until the atmosphere.
And even THROUGH the atmosphere! Air absorbs very little IR light. Air has to get into direct contact with warm objects to get warm, or to mix with warmer air. You can make an experiment: turn on an IR lamp (maybe it works with a normal old-fashioned light bulb) for some time, then turn it off and feel the air and the objects close to the lamp. You will notice that the objects are feelably heated up, while the air isn't (or only close to the objects).
Actually, the Sun heats the surface of the ground first, and then the surface heats the air (quickly) and deeper layers of ground (slowly).
Why is air heated up quickly? Because in the air there also is less stuff (molecules, atoms) to be heated up than in a solid or liquid. The hardest ot heat is water, that's why oceans make a balanced climate.