In reply, I have mine without milk also, but leave the tea to brew for a very long time. The brown 'gunk' seems to be harder to shift the longer I let my tea brew; however will come off in the dishwasher. My boyfriend likes his as strong as mine, but with milk and never has this problem. However, the inside of his mug seems not to be coated in the same way mine is, for instance his lacks the same 'sheen' my mug has. Perhaps the mug itself is a factor?
I'm sorry, I havn't really answered the question but I hope I've offered some ideas for you to look up.