If you include submodules into your git repositories, you should make them public if you expect others to use your project. If you use git@ references to these remotes, the person would need to have access to these too.
If your dotfiles should be sharable, include your remotes like this, with http access:
[submodule "vim"] path = vim url = http://github.com/<username>/dot_vim.git
Then anybody wanting to check out your repo and all children, could conveniently do this after her/his clone:
git submodule update --init --recursive
And all children are fetched recursively with all the submodule repos in one command!