Time difference do indeed help to locate the source of a sound.  But that's not the only mechanism we use.  The shape of the external bits of the ears also play a role.   They don't just focus the sound waves, they filter different frequencies when a sound wave hits them too. From this information the brain can determine it's rough source.

To get a more precise location we generally cock our heads to increase the angle of the sound wave hitting our heads so that the time difference between the various inputs hitting each ear are maximised, which makes it easier for the brain to get a fix on the source.