It is a problem when the listener doesn't notice the joke and takes it seriously. Then they usually they start to think the person is really dumb for saying something so ridiculous.
Take Ali G for example. He interviews all these celebrities and asks them the most stupidiest questions, like he once asked a former director of an ER unit what character he plays. The director assumes the guy is an idiot because he didn't realise it was a joke.
It's a bit similar to mine at my workplace. Sometimes I get the impression that some of them think I am less capable than I really am mainly because of the way I joke, they seem to miss it quite often.
Ok this is in reply to a comment on my previous post. My reply was kind of long so I made it an entry. Quite smart eh?