ISFDB Download Script

Purpose

If you don't want to check the download page for newer files every now and then and don't want to click through all the links there manually, and if you're on an UNIX machine like Linux, you can use the shell script shown below to download the ISFDB files.

You can optionally ignore certain downloads, see below for more details.

Unlike earlier versions of it, this script is currently not able to resume interrupted downloads or to detect if an already downloaded file is the same version as the one on the backup server (and could therefore be skipped), i.e. that the script simply downloads the latest database backup and the cover image backups again if executed.

This script might not work on all UNIX flavours. It was developed on Xubuntu and should work on all similar platforms. It should also work on Mac OS X, but the current version of this script has not been tested there.

Support

If you've got questions or any other comment regarding this script the best bet is to contact the developer of the script directly through his wiki page.

Examples

Assuming you saved the script to a file called "isfdb_download.sh", here are some examples of how to call it.

Download everything

This is the most simple case: download covers, latest database dump and get latest source code. Just call the script and tell it the directory in which the downloads shall be stored:

isfdb_download.sh /home/username/backups/isfdb

Please note that when you want the source code downloaded you're maybe prompted for a password when calling the script for the first time for the given download directory. Simply hit RETURN there! Subsequent calls of the script will not show the password prompt.

Database only

If you're not interested in source code and the huge cover files, ignore them by using options:

isfdb_download.sh -s -c /home/username/backups/isfdb

Print all available options

sfdb_download.sh -h

The download script

Copy and paste the code into a text editor, save it and make the file executable:

#!/bin/sh

# This scipt downloads the latest database backup file and all cover image backups listed on the ISFDB
# downloads page as well as the latest source code from the source code repository.

# You can optionally ignore certain downloads, see code below or call this script using the
# "-h" option for more info.

# The script is currently not able to resume interrupted downloads. It also isn't able to detect
# if an already downloaded file is the same version as the one on the backup server,
# i.e. that it simply downloads the latest database backup and the cover image backups again.

# The files to download are identified by examining the downloads page in the ISFDB wiki
# and extracting the Google Drive ID of these files from it.

download_page_url="http://www.isfdb.org/wiki/index.php/ISFDB_Downloads"

usage()
{
  echo "$(basename "$0") [OPTIONS] DOWNLOAD_DIRECTORY"
  echo "Valid options are:"
  echo "  -c | --ignore-covers : ignore cover files"
  echo "  -d | --ignore-database : ignore database file"
  echo "  -s | --ignore-sources : ignore source code"
  echo "  -h | --help : this message"
}

ignore_sources=
ignore_database=
ignore_covers=

while [ "$1" != "" ]; do
    case $1 in
        -s | --ignore-sources )    ignore_sources=true;;
        -d | --ignore-database )   ignore_database=true;;
        -c | --ignore-covers )     ignore_covers=true;;
        -h | --help )              usage
                                   exit;;
        -* )                       echo "Unkown option $1"
                                   usage
                                   exit 1;;
        *)                         download_dir="$1";;
    esac
    shift
done

if [ -n "$download_dir" ]; then
  # If user specified a relative path, turn it into an absolute path
  case "$download_dir" in
    /*) ;;
     *) download_dir="$(pwd)/$download_dir" ;;
  esac
  mkdir -p "$download_dir"
  if [ ! -w "$download_dir" ]; then
    echo "ERROR: Backup directory '$download_dir' couldn't be created or is not writeable!"
    usage
    exit 1
  fi
else
  echo "ERROR: No backup directory provided!"
  usage
  exit 1
fi

sources_dir="$download_dir/sources"
database_dir="$download_dir/database"
covers_dir="$download_dir/covers"

mkdir -p "$sources_dir"
mkdir -p "$database_dir"
mkdir -p "$covers_dir"

download_page="$download_dir/isfdb_download_page.html"

errors=

echo
echo "******************************************"
echo "        Get and check download page"
echo "******************************************"
echo
if [ -e "$download_page" ]; then
  # Download the page only if it has been changed since the last download (using timestamp
  # comparison):
  curl_cmd="curl -z $download_page -o $download_page $download_page_url"
else
  curl_cmd="curl -o $download_page $download_page_url"
fi
if ! $($curl_cmd) ; then
  echo "IFSDB download page $download_page_url could not be downloaded. Did the URL change probably?"
  exit 1
fi

if [ -z $ignore_sources ]; then
  echo
  echo "******************************************"
  echo "     Check out or update source code"
  echo "******************************************"
  echo
  if [ -z $(which svn) ] ; then
    errors="${errors}\n'svn' executable not found. If you want the source code you have to install the package"
    errors="${errors}\nwhich contains 'svn' or use the option '-s' to get rid of this message."
  else
    sources_module_name="isfdb2"
    if [ -e "$sources_dir/$sources_module_name/" ]; then
      cd  "$sources_dir/$sources_module_name"
      if ! svn update ; then
        errors="${errors}\nCould not update sources from SVN"
      fi
    else
      cd "$sources_dir"
      echo
      echo "No working copy found. An initial checkout of the complete source code will now be attempted."
      echo "If nothing seems to happen it is probably waiting for a password to be entered, in which case pressing RETURN should suffice."
      if ! svn checkout svn://svn.code.sf.net/p/isfdb/code-svn "$sources_module_name" ; then
        errors="${errors}\nCould not check out sources from SVN."
      fi
    fi
  fi
else
  echo "Ignoring source code"
fi

can_download=true
if ([ -z $ignore_database ] || [ -z $ignore_covers ]) && [ -z $(which gdown) ]; then
  errors="${errors}\ngdown is required to download the backup files from Google Drive but could not be found."
  errors="${errors}\nIt's a Python library which can usually be installed from the Python Package Index (PyPI) repository:"
  errors="${errors}\n  pip install gdown"
  errors="${errors}\nIf that doesn't work check out https://github.com/wkentaro/gdown"
  can_download=false
fi

if [ -z $ignore_database ]; then
  if $can_download; then
    echo
    echo "******************************************"
    echo "           Get latest database"
    echo "******************************************"
    echo
    # Expected structure: all links below <a name="Database_Backups"> are database backups, and
    # only use the second link (in the table column with the newer MySQL-Version)
    database_id=$(cat "$download_page" | sed '1,/<a name="Database_Backups">/d' | grep https | sed -n '2s/.*id=\([^"]*\).*/\1/p')
    if [ -n "$database_id" ]; then
      echo "Trying to download database file with id $database_id"
      cd "$database_dir"
      gdown --id "$database_id"
      # The gdown version 3.8.3 used to develop this script always returns 0, therefore no exit code checking
    else
      errors="${errors}\nFailed to identify latest database backup file on the ISFDB wiki page."
      errors="${errors}\nEither the format of the whole page or of the download links on it has changed."
    fi
  fi
else
  echo "Ignoring database"
fi

if [ -z $ignore_covers ]; then
  if $can_download; then
    echo
    echo "******************************************"
    echo "            Get latest covers"
    echo "******************************************"
    echo
    cd "$covers_dir"
    covers_file=/tmp/isfdb_download_covers
    # Expected structure: all links between <a name="Image_Backups"> and <a name="Database_Backups"> are cover image backups
    cat "$download_page" | sed -n '/<a name="Image_Backups">/,$p' | sed -n '/<a name="Database_Backups">/q;p' | grep https | sed -n 's/.*id=\([^"]*\).*/\1/p' > "$covers_file"
    while read -r covers_id
    do
      echo "Trying to download cover image file with id $covers_id"
      gdown --id "$covers_id"
      # The gdown version 3.8.3 used to develop this script always returns 0, therefore no exit code checking
    done < "$covers_file"
    rm "$covers_file"
  fi
else
  echo "Ignoring covers"
fi

if [ -n "$errors" ]; then
  echo
  echo "!!!!!!!!!!!!!!!!!!!!!!!!!!!!"
  echo "     THERE WERE ERRORS"
  echo "!!!!!!!!!!!!!!!!!!!!!!!!!!!!" 
  echo
  printf "%b\n" "$errors\n"
else
  echo "Done."
fi