As of today there still isn't a UI based option to remove a project in Team Foundation Service.
So how do we do it?
Turns out to be much simpler than I had mentally prepared myself for.
Open Administrator Command Prompt (win+x a), change directory to:
C:\Program Files (x86)\Microsoft Visual Studio 11.0\Common7\IDE
The run the following command substituting the values inside the [ ] brackets.
TFSDeleteProject.exe /collection:"https://[name].visualstudio.com/DefaultCollection" "[project name]"
You should then see a warning, press y to delete the project. Done.
I have VS2012 installed, and VS is already joined to my Team Foundation Service.