using .netrc with sftp

ftp, macros, sftp

Solution

Simply put, you cannot use `.netrc` with `sftp`, `scp` or `ssh`. These products are part of the OpenSSH standard, which has the keyword 'secure' in the name. It is not a secure practice to automate logins the way `.netrc` does, and the standard prohibits this kind of automation (storing passwords). There is definitely an alternative, three actually.

Authorization

For either of the first two alternatives, you will want to setup keys and exchange them. On the machine you are connecting from run `ssh-keygen`, for your purposes it will be much simpler if you do not give the key a pass-phrase, though this is risky. You now have two files in `.ssh/`, an `id_rsa` and a `id_rsa.pub`. Of these the `id_rsa` must be kept secret or secured (hence the pass-phrase). The pub file is actually one line of text. This one line can be added to the `~/.ssh/authorized_keys` file on the receiving host's side. You can add the key to the file manually; but there is also `ssh-copy-id` shortcut command which does just that, also taking care of file permissions. Having authorized a key, you should be able to connect from the machine with the private key to the machine which has the authorized public key, when you connect as the appropriate user. Test it with `ssh -v`. If you entered a pass-phrase, you will be prompted for it; if you did not you are now automation ready. You can use an `ssh-agent` to keep a private key active between sessions while only entering the pass-phrase once. If you are making multiple `ssh` hops, the option to forward agents will allow the private key from the original sourced box's `ssh-agent` to be communicated though each hop. Personally I find this overwrought, and hence suggest not using a pass-phrase.

Now that you can make `ssh`, `sftp`, and `scp` connections without entering any password or pass-phrase you're ready to automate the rest.

Alternative 1,

is the preferred alternative were you convert your `.netrc` macro to a shell script or other script calling a few `scp` commands. This is similar to automating all your ftp connections with `curl` or `wget`. E.G.:

scp -qr $USER@$REMOTE_HOST:$PATH_FILE_OR_DIR $LOCAL_PATH_FILE_OR_DIR #download
scp -qr $LOCAL_PATH_FILE_OR_DIR $USER@$REMOTE_HOST:$PATH_FILE_OR_DIR #upload
scp -pqr $USER@$REMOTE_HOST:$PATH_FILE_OR_DIR $USER@$REMOTE_HOST2:$PATH_FILE_OR_DIR #mirror between separate hosts.
ssh $USER@$REMOTE_HOST chmod 644 $PATH_FILE #set permissions

Alternative 2,

using `sftp` as you mentioned, you can script it with the `expects` command, with a batch file using the `-b` option, or by piping commands into `sftp`. This is a little more similar to an `.netrc` macro, but has no advantage over alternative 1. I'll show an example of the latter:

#!/bin/sh
echo "OK, starting now..."
sftp -b /dev/fd/0 remotehost <<EOF
cd pub
ascii
get filename.txt
bye
EOF

Alternative 3,

use an `sftp` program that breaks the SSH standard by allowing you to store connection parameters such as the password. For example using cyberduck and AppleScript, or FileZilla and a queue.

Further notes:

There is an `~/.ssh/config` file you can use to give hostnames shorter names, set forwarding parameters, default directories, default usernames, and specific identities for each host. I also like the `-l` option of `scp` which limits my transfer rate to something more reasonable.

P.S. You'd think there's a tool out there for converting `.netrc` macros to (alternative 1 styled) shell scripts. But I found nothing. Is that a tiny niche business opportunity?

Problem

Original source