Remove a submodule added to branch when switching to another branch

You added a submodule to a branch than you changed to another branch and git clean -d -f keep the damn directory.

Try to double -f option:
git clean -d -f -f

from git help clean: http://www.kernel.org/pub/software/scm/git/docs/git-clean.html