How to store private keys on flashdrive

How to store private keys on flashdrive

I'm trying to create a secure and private way to access my personal PC from most desktop PCs. The method I've devised so far is:

1. Create an SSH tunnel to my PC
   • The connection is encrypted
   • The connection uses a public key exchange instead of a password
   • The connection requires the passphrase of my key and two-factor authentication via my phone
2. Use a remote desktop to access my PC - I'm using Windows' Remote Desktop Connection since it will already be installed on most PCs
3. I'm storing the private key for the SSH connection on a flash drive

The biggest weakness I see is that my private key is on a flash drive that I'm liable to loose. Is there a good alternative, maybe encrypting the flash drive, so that even if I loose it I'm not too exposed? The key is also encrypted via passphrase, so I think I'm alright. But the drive also has my IP. BitLocker is not an option as I don't have windows Pro or Enterprise.

Criticisms of other glaring holes in my plan are also welcome. Of course, I would also have to reasonably trust the client PC I'm using to connect to my server.