It is not an illusion, but a real difference. Pillows are usually made of materials that are good insulators and trap the heat from your head and prevent it from reaching other parts of the pillow. In a short time the part of the pillow you are lying on will be close to body temperature while the side facing down will stay at room temperature, which is nearly always less than body temperature. When you turn the pillow you go from a temperature of 35+ degrees to somewhere in the low 20s