You are quite right about there being many factors, and as you suggest, one of them is indeed the humidity of moving air. One might expect drastic differences between the effects of humid Natal or Queensland, and Alice Springs or Windhoek. There is not much point letting hot, humid air into the house to cool things down, though a dry breeze, even a warmish one, could have a cooling effect on one's perspiration.

It also makes a difference how much sunlight can get in and how much can be reflected or insulated out. Some of the variations could cause differences of tens of degrees Celsius, enough to cook food, rather slowly of course.

So you can see that for almost any glib generalisation one could select special circumstances to invert the results, so that your assumption that there is a universal rule is not a safe one.

If we can ignore the effects of humidity, then a rough rule is that if you can interrupt the flow of heat across the internal/external interface of the house, then one should do so for as long as the net flow of heat is inward, and encourage it for as long as the net flow is outward. The insulation might well b e selective. For example, it could make perfect sense to put up reflective blinds that let the air through while the outside air is cooler than the inside, but keep most of the sunlight out.

On the other hand, usually if it is hot outside, you might logically wish to keep both the hot air and radiation out, but open the house at night to let the cool outside air in.

So, sorry, no generalisations, but I would support your grandfather's view more often.