Not really a question this one - but alternative answers are certainly welcome! :)
Sometimes I need to download files listed in an Apache directory listing, say as in:
... in a given directory on my computer, and I'd like to use command-line tools for that.
As far as wget is concerned, actually there are some switches that should be in place - noting that wget tends to either download single items - or to reconstruct the server folders locally!
As I just spent some half an hour figuring out what those proper switches are (to copy the remote files locally as through, say, FTP), I'd like to document them here; so the above link would be downloaded with:
wget -nd -r -l 1 ... where:
-nd --no-directories Do not create a hierarchy of directories when retrieving recursively. With this option turned on, all files will get saved to the current directory, without clobbering [...] -r --recursive Turn on recursive retrieving. -l depth --level=depth Specify recursion maximum depth level depth. The default maximum depth is 5. It would be nice to hear alternatives to the above command - maybe using different set of switches for wget - or maybe using curl or other packages...
2 Answers
After some time, thanks to the pointers by @jw013 and in Sync with a Directory Listing of An Apache Server - Unix & Linux Stack Exchange, I finally found my "ultimate" wget command.
Assume you have a tree of files and directories in a directory called myfiles_dir, and let that directory be accessible through an Apache directory listing on . Then, you can sync directory locally with:
wget -r -N --no-parent --reject '*index.html*' -nH --cut-dirs=1 ... where:
-r, --recursive specify recursive download. -N, --timestamping don't re-retrieve files unless newer than local. -np, --no-parent don't ascend to the parent directory. -R, --reject=LIST comma-separated list of rejected extensions. -nH, --no-host-directories don't create host directories. --cut-dirs=NUMBER ignore NUMBER remote directory components. This will download the contents into a subdirectory myfiles_dir created in the directory wget was called from (the working directory), without any residual index.html files.
Note that the trailing slash / after the http link/address is extremely important: if it is not there - as in - upon repeated calls of wget from the same local working directory, the HTML of the directory listing will be saved as multiple copies e.g. myfiles_dir.1, myfiles_dir.2 etc, in spite of any switches (although, the HTML directory listings will not be saved for any subdirectories inside, as requested; also, the very first time the command runs, there will not be a HTML directory listing saved for myfiles_dir).
However, with the trailing slash - as in - no HTML directory listings will be saved for any directory, including the "root" myfiles_dir, after repeated calls to wget from the same local location.
See also -nH a.k.a --no-host-directories and --cut-dirs options. I also frequently use --accept/-A and --reject/-R.
