It's hot, but not hot enough to do the damage you suggest. The secret is in the design and the planning. And remembering to remove everything that might get damaged.

Yes, the exhaust is hot! There are no 'buildings' in the vicinity, only the pad & tower. To avoid undue damage, the whole structure is flooded with a staggering amount of water. The pipes to supply this water are about 1 metre in diameter. If you watch video of a launch, it is difficult to see the pad and exhausts due to the amount of steam produced. Once clear of the pad, rocket exhausts burn with a clean flame that leave little residue until high in the sky, when the water starts to condense into the familiar contrails. Solid fuel boosters burn with a dirtier flame as they run on solidified rubber particles and an oxidiser.