So when setting up a Dokku instance from the default web interface, I mistakenly pasted a PuttyGen formatted public key. Dokku doesn't do any checking here and I couldn't figure out why I couldn't log in to the box with my private key.
I eventually thought of the key format and checked
/home/dokku/.ssh/authorized_keys and removed the ill formatted public key and replaced with the one PuttyGen tells you to use for openSSH.
There's a command line at the top that I believe Dokku places in authorizedkeys. I initially placed the good key on it's own line. While this passed the ssh test, git would return errors when trying to push:
$ git push dokku master
Enter passphrase for key '/c/Users/abc/.ssh/idrsa':
fatal: 'www' does not appear to be a git repository
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.
Eventually remembered that the bad key started on the top line at the very end. Moving the key up to the end of the first line did the trick.