Whats the best way pull files to S3 using FPT/SCP?
amazon-s3, amazon-web-services
Solution
Just use `s3cmd` or `boto` or the S3 command-line tools or... well... quite a few other options.
Basically you're just posting to the S3 API. You can do it in nearly any language using AWS's myriad SDKs. `s3cmd`, however, is a nice, lightweight little CLI tool that's probably more than you need.
Your comment suggests you're looking for something slightly different. Here are a few recommendations based on the source and destination of the files:
If you're moving files from EC2 or your localhost to S3, use `s3cmd`. If you need to run it regularly, consider a cron job. The following will run every five minutes.
*/5 * * * * s3cmd sync --delete-removed my_local_directory/ s3://my-bucket/path/
If you're moving files from S3 to EC2 or your localhost, use `s3cmd` but reverse the syntax:
*/5 * * * * s3cmd sync s3://my-bucket/path/ my_local_directory/
If you're moving files from your localhost to EC2, consider `scp` or `rsync`.
Problem
Whats the best, light-weight, way to FPT/SCP files to S3 every few min? So wget/curl could work. Also a custom solution could work, maybe Java with DB to maintain the state. Any ideas would be helpful.