Brief:
I have a directory where files are dropped arbitrarily in server A. I want to dump these files into server B and delete them from server A.
Background:
I would like write an Bash script where the code iterates through each file, puts the file across to server B, and deletes it straight after. This will be ammended to the cron and called hourly to scrape the files across to server B
Code:
The code I have attempted is:
#!/bin/bash MY_SCRIPT_NAME=`basename "$0"` PATH_TO_METRICS='/home/some/directory' if pidof -o %PPID -x $MY_SCRIPT_NAME > /dev/null; then echo "$MY_SCRIPT_NAME already running; exiting" exit 1 fi sftp -i my_priv_key -oPort=12345 user@12.123.123.123 <<EndOfTransfer cd $PATH_TO_METRICS for filename in $PATH_TO_METRICS; do PUT "$filename" rm "$filename" done EndOfTransfer Relevant Research:
- Execute command in sftp connection through script
- FTP file transfer, loop through a directory and copy old files
Question:
How does one iterate through files in a given a directory and use sftp to put each of the files across to another server, and delete the file if a successful transfer has happened?
2 Answers
You need to understand <<EOF and EOF
In the below example <<EOF after the sftp command has the meaning, pass everything to the sftp program as standard input untill you get to EOF (End Of File).
So you could easily modify your script with this and get it working.
sftp YourSftpServer <<EOF put YourFile exit EOF There are two ways to do this that are much easier than using sftp itself, since you're using SSH in either case. You only need to use SFTP if you're actually running an "old school" SFTP server. If Server A/B are actually Linux machines, just use rsync or scp
rsync --remove-source-files -r /home/some/directory user@ServerB:/somewhere/else This will copy all the files and has much better performance than sftp commands or scp since they typically open/close many connections. rsync will open a single connection, check to see what needs to be transfered, then transfer it all without fragmented connections.
If for whatever reason you need to pass your private key or use custom ports to SSH you can pass the -e "ssh -i foo.key -p 1234" to rsync and it will use those keys.
