Well, plugging in your headset should not do that in the first place. If unplugging your headphones helps, then it comes to mind that maybe you installed some software that is audio related that is messing up somewhere. Got any audio grabbing, streaming, equalisers installed? Get rid of them first to see if the problem is resolved then.
It's easy to write this off as an OSX fault, there are still many of them in Mavericks, but most of the time there is some sort of reason for this behaviour (incompatible software, for one).