I'm not very musical (something of an understatement) but I've taken an interest in the phenomenon of perfect pitch in the past. Even very accomplished and experienced musicians often admire the ability to recognise a single note (when it is played without reference to another note). That is, perfect pitch is not a prerequisite for musical ability.
Can I be so bold as to suggest that your perception of tempo is not as accurate as you'd like to think it is? Moreover, if you hear a song and lose time with it as you try to join in, is it conceivable that you simply lack the necessary aptitude or degree of practice to match your singing to the tempo you are hearing?
Personally I find it difficult to sing Happy Birthday in time with other people, and I can't get anywhere near the tune! It's terribly frustrating because I love a good old sing-song - it's just that no-one enjoys it but me.
PS There was a university website where a researcher was collecting data on listeners' abilities to distinguish tunes by pitch and tempo. If I find it again, I'll post it up. It was very interesting.