In order to load an SSH key in Transmit, the key has to be in either PEM or PuTTY (.ppk) format. In case the key file is a passphrase-protected PEM file, you will be prompted for the passphrase upon selecting it. Passphrase-protected .ppk files are not supported.
If you are having difficulties loading your SSH key into Transmit, as a workaround you can try to import your SSH key in FileZilla first.
FileZilla generates an SSH key file in a format that can be used by Transmit. You can then use the key icon in your Transmit client to load the SSH key and establish an SFTP connection.