Visible mould is usually only a part of what is there. I doubt there's a lot of point scraping it off because the fungal mycelium has probably penetrated deep into the cheese. I normally just eat the lot. If there's any doubt in my mind (and it's illogical really, when you consider what's living in stilton or ripe camembert) then I have cheese on toast or a nice omelette!
I don't eat mouldy fruit though. It doesn't taste right on fruit.