Unfortunately, there is no cpan remove or cpan uninstall command. What you can do:
# follow instructions (copy commands) since it often doesn't really perform the uninstall
If you find several "versions" of your Module for whatever reasons, you should
find ~/.cpan/build/Module-To-Be-Uninstalled* -name Makefile
and change into to mentioned directory.
I think I have to write a bash function or alias for that...