It's down to what your body experienes while under water. Most of us feel "weightless" when under water, but in fact, we are experiencing greater pressure in water than we do out of it. Water is heavier than air and exerts more pressure on the body. So when you are under water, your system is working harder. Your heartbeat increases, your respiration actually goes down (unless you are being more than usually active), and so on. The result of this is an increase in your body's production of lactic acid, which makes you "feel tired".

I'm a pretty poor swimmer: both unskilled enough to be inefficient, and slightly panicky (I dream of drowning quite a lot). I also sailed a bit in Scotland, so I tend to associate swimming with words like involuntary, sudden, cold, tangled, dirty, unprepared, and rough.

One consequence I am aware of is that I try to maximise my buoyancy by keeping my lungs full, and this would also extend my capacity if I went under the surface unexpectedly. So I take a lot of shallow breaths without letting my lungs exhale properly. I'm fairly sure that seriously limits my aerobic capability and tires me out prematurely.