Rclone Uploader

  • Here, we will set up the uploader. The rclone uploader is a script that uploads the contents of your Local folder and merges it to your rclone mount.

Bash Upload Script

  • Download this script to whatever folder you wish. In this case, scripts will be the folder where you'll save the script.
    • You may choose between the standard upload script (no notifications) or the rclone upload script that sends discord notifications whenever there's a successful transfer.
  • Change the script's permission to 755, so it makes it executable by you. So do chmod +x >name of script<.sh
    • In the proceeding examples, I decided to use the standard rclone script so do chmod +x rclone-upload.sh
Normal Upload Script
cd scripts/
wget https://raw.githubusercontent.com/ultraseedbox/UltraSeedbox-Scripts/master/MergerFS-Rclone/Upload%20Scripts/rclone-upload.sh
chmod +x rclone-upload.sh
Upload Script with Discord Notifications via Webhook
cd scripts/
wget https://raw.githubusercontent.com/ultraseedbox/UltraSeedbox-Scripts/master/MergerFS-Rclone/Upload%20Scripts/rclone-upload-with-notification.sh
chmod +x rclone-upload-with-notification.sh
  • Then, open the script using your preferred text editor and change the values as needed.
    • Both normal and upload script with discord notifications needs to be set up before using it
    • Replace remote: to the remote name you set previously from the previous guide.
Example rclone-upload.sh file
#!/bin/bash

lock_file="$HOME/scripts/rclone-upload.lock"

trap 'rm -f "$lock_file"; exit 0' SIGINT SIGTERM
if [ -e "$lock_file" ]
then
    echo "Rclone upload script is already running."
    exit
else
    rm "$HOME"/scripts/rclone-upload.log
    touch "$lock_file"
    "$HOME"/bin/rclone move "$HOME"/Stuff/Local/ gdrive: \
        --config="$HOME"/.config/rclone/rclone.conf \
        --drive-chunk-size 64M \
        --tpslimit 5 \
        -vvv \
        --drive-stop-on-upload-limit \
        --delete-empty-src-dirs \
        --fast-list \
        --bwlimit=8M \
        --use-mmap \
        --transfers=2 \
        --checkers=4 \
        --log-file "$HOME"/scripts/rclone-upload.log
    rm -f "$lock_file"
    trap - SIGINT SIGTERM
    exit
fi
Example rclone-upload-with-notification.sh file
#!/bin/bash

# Rclone upload script with optional Discord notification upon move completion (if something is moved)
#
# Recommended for use via cron
# For example: */10 * * * * /path/to/rclone-upload.sh >/dev/null 2>&1
# -----------------------------------------------------------------------------

SOURCE_DIR="$HOME/Stuff/Local/"
DESTINATION_DIR="gdrive:"

DISCORD_WEBHOOK_URL=""
DISCORD_ICON_OVERRIDE="https://i.imgur.com/MZYwA1I.png"
DISCORD_NAME_OVERRIDE="RCLONE"

LOCK_FILE="$HOME/rclone-upload.lock"
LOG_FILE="$HOME/rclone-upload.log"

# DO NOT EDIT BELOW THIS LINE UNLESS YOU KNOW WHAT YOU'RE DOING
# -----------------------------------------------------------------------------

trap 'rm -f $LOCK_FILE; exit 0' SIGINT SIGTERM
if [ -e "$LOCK_FILE" ]
then
  echo "$0 is already running."
  exit
else
  touch "$LOCK_FILE"

  rclone_move() {
    rclone_command=$(
      "$HOME"/bin/rclone move -vP \
      --config="$HOME"/.config/rclone/rclone.conf \
      --user-agent='Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.61 Safari/537.36' \
      --drive-chunk-size 8M \
      --use-mmap \
      --delete-empty-src-dirs \
      --fast-list \
      --log-file="$LOG_FILE" \
      --stats=9999m \
      --tpslimit=1 \
      --transfers=1 \
      --checkers=1 \
      --bwlimit=2M \
      --drive-stop-on-upload-limit \
      "$SOURCE_DIR" "$DESTINATION_DIR" 2>&1
    )
    # "--stats=9999m" mitigates early stats output
    # "2>&1" ensures error output when running via command line
    echo "$rclone_command"
  }
  rclone_move

  if [ "$DISCORD_WEBHOOK_URL" != "" ]; then

    rclone_sani_command="$(echo $rclone_command | sed 's/\x1b\[[0-9;]*[a-zA-Z]//g')" # Remove all escape sequences

    # Notifications assume following rclone ouput:
    # Transferred: 0 / 0 Bytes, -, 0 Bytes/s, ETA - Errors: 0 Checks: 0 / 0, - Transferred: 0 / 0, - Elapsed time: 0.0s

    transferred_amount=${rclone_sani_command#*Transferred: }
    transferred_amount=${transferred_amount%% /*}

    send_notification() {
      output_transferred_main=${rclone_sani_command#*Transferred: }
      output_transferred_main=${output_transferred_main% Errors*}
      output_errors=${rclone_sani_command#*Errors: }
      output_errors=${output_errors% Checks*}
      output_checks=${rclone_sani_command#*Checks: }
      output_checks=${output_checks% Transferred*}
      output_transferred=${rclone_sani_command##*Transferred: }
      output_transferred=${output_transferred% Elapsed*}
      output_elapsed=${rclone_sani_command##*Elapsed time: }

      notification_data='{
        "username": "'"$DISCORD_NAME_OVERRIDE"'",
        "avatar_url": "'"$DISCORD_ICON_OVERRIDE"'",
        "content": null,
        "embeds": [
          {
            "title": "Rclone Upload Task: Success!",
            "color": 4094126,
            "fields": [
              {
                "name": "Transferred",
                "value": "'"$output_transferred_main"'"
              },
              {
                "name": "Errors",
                "value": "'"$output_errors"'"
              },
              {
                "name": "Checks",
                "value": "'"$output_checks"'"
              },
              {
                "name": "Transferred",
                "value": "'"$output_transferred"'"
              },
              {
                "name": "Elapsed time",
                "value": "'"$output_elapsed"'"
              }
            ],
            "thumbnail": {
              "url": null
            }
          }
        ]
      }'

      /usr/bin/curl -H "Content-Type: application/json" -d "$notification_data" $DISCORD_WEBHOOK_URL
    }

    if [ "$transferred_amount" != "0" ]; then
      send_notification
    fi

  fi

  rm -f "$LOCK_FILE"
  trap - SIGINT SIGTERM
  exit
fi
  • Then, decide how often you want to execute the script. Our servers are in the Netherlands, so the timezone is set to CET
    • In my case, I'd like to run the script at 1900 CEST every night. Translated to cron, that is 0 19 * * *

::: info If you need help in translating the time you want to cron, you can visit Crontab Guru and World Time Buddy to help you with setting up the time. :::

  • Do ls $PWD/rclone-upload.sh to get the absolute path
kbguides@lw902:$ cd ~/scripts
kbguides@lw902:~/scripts$ ls $PWD/rclone-upload.sh
home/scripts/rclone-upload.sh
  • Open up crontab -e, set your preferred text editor (in this tutorial, nano), and add the cron expression and the absolute path of the upload script.
  • Add > /dev/null 2>&1 if you don't want your slot to send messages when the upload script fails.

  • Save cron by doing CTRL + O and CTRL+ X to exit the editor
  • Now, all you need to do is to access all your apps and point them to your newly made MergerFS folder, which is MergerFS

Revision #8
Created Fri, Jul 2, 2021 1:16 PM by Joe
Updated Thu, Sep 9, 2021 2:40 PM by Joe