Posted on
• Scripting for DevOps

Setting Up Continuous Backup with Bash Scripts

Author
  • User
    Linux Bash
    Posts by this author
    Posts by this author

Setting Up Continuous Backup with Bash Scripts

Continuous backup ensures that critical data is regularly and automatically backed up to a secure location, minimizing the risk of data loss. With Bash scripts, you can automate the backup process to run on a schedule or in response to specific triggers. This guide explains how to set up continuous backup using Bash.

1. Prerequisites

  • Basic Bash Knowledge: Familiarity with scripting and command-line utilities.
  • Backup Location: Decide where to store backups (e.g., local directory, external storage, or cloud services like AWS S3).
  • Tools Installed:
    • rsync: For efficient file synchronization.
    • tar: For compressing files.
    • Cloud CLI (optional): AWS CLI, Google Cloud CLI, etc., if storing backups in the cloud.
  • Sufficient Storage Space: Ensure the backup destination has enough storage capacity.

2. Backup Strategy

  • Source Directory: The directory containing files to back up.
  • Destination Directory: Where backups will be stored.
  • Frequency: Set up backups to run continuously (via a loop or a cron job).

3. Example Scripts

Example 1: Local Continuous Backup

This script monitors a source directory and continuously syncs changes to a backup directory.

#!/bin/bash
set -e

SOURCE_DIR="/path/to/source"
BACKUP_DIR="/path/to/backup"
LOG_FILE="/path/to/backup.log"

echo "Starting continuous backup..."
while true; do
    rsync -avh --delete "$SOURCE_DIR/" "$BACKUP_DIR/" >> "$LOG_FILE" 2>&1
    echo "$(date): Backup completed." >> "$LOG_FILE"
    sleep 300  # Wait for 5 minutes before the next backup
done
  • Features:
    • Synchronizes files from SOURCE_DIR to BACKUP_DIR.
    • Deletes files in the backup directory that no longer exist in the source.
    • Runs every 5 minutes.

Example 2: Compressed Backup with Timestamps

This script compresses the source directory and stores it as a timestamped archive.

#!/bin/bash
set -e

SOURCE_DIR="/path/to/source"
BACKUP_DIR="/path/to/backup"
LOG_FILE="/path/to/backup.log"

# Create the backup directory if it doesn't exist
mkdir -p "$BACKUP_DIR"

# Perform the backup
TIMESTAMP=$(date +"%Y-%m-%d_%H-%M-%S")
BACKUP_FILE="$BACKUP_DIR/backup_$TIMESTAMP.tar.gz"

echo "Starting backup: $BACKUP_FILE"
tar -czf "$BACKUP_FILE" "$SOURCE_DIR" >> "$LOG_FILE" 2>&1
echo "$(date): Backup completed: $BACKUP_FILE" >> "$LOG_FILE"
  • Features:
    • Creates compressed backups with timestamped filenames.
    • Keeps a log of backup activities.

Example 3: Continuous Backup to AWS S3

This script uses the AWS CLI to back up files to an S3 bucket.

#!/bin/bash
set -e

SOURCE_DIR="/path/to/source"
S3_BUCKET="s3://your-bucket-name"
LOG_FILE="/path/to/backup.log"

# Perform the backup
while true; do
    echo "$(date): Starting S3 backup..." >> "$LOG_FILE"
    aws s3 sync "$SOURCE_DIR" "$S3_BUCKET" --delete >> "$LOG_FILE" 2>&1
    echo "$(date): S3 backup completed." >> "$LOG_FILE"
    sleep 300  # Wait for 5 minutes before the next backup
done
  • Features:
    • Continuously syncs the source directory with an S3 bucket.
    • Removes files from S3 that no longer exist in the source directory.

4. Automating with Cron Jobs

You can automate these scripts using cron to run at specific intervals.

Example Cron Job Setup

  1. Open the cron editor:

    crontab -e

  2. Add a cron job to run the script every hour:

    0 * * * * /path/to/backup_script.sh

  3. Save and exit. The script will now run hourly.

5. Enhancing Your Backup Script

  • Email Notifications: Send an email after each backup completes.

    mail -s "Backup Completed" user@example.com < /path/to/backup.log

  • Encrypt Backups: Use gpg to encrypt backups for added security.

    tar -czf - "$SOURCE_DIR" | gpg --encrypt --recipient "your-email@example.com" > "$BACKUP_FILE.gpg"

  • Retention Policy: Automatically delete old backups to save space.

    find "$BACKUP_DIR" -type f -mtime +30 -exec rm {} \;

  • Error Handling: Add error logging and alerts.

    set -e
trap 'echo "Error on line $LINENO" | mail -s "Backup Failed" user@example.com' ERR

6. Best Practices

  • Test Backups Regularly: Verify that backups are restorable and complete.
  • Use Incremental Backups: Save time and space by backing up only changed files (e.g., with rsync).
  • Secure Backup Storage: Encrypt sensitive data and restrict access to backup locations.
  • Monitor Disk Usage: Ensure sufficient storage is available for backups.

7. Tools for Advanced Backup Needs

  • BorgBackup: Efficient deduplication and encryption for backups.
  • Restic: Cloud-friendly, secure, and fast backup tool.
  • Rclone: Syncs files to various cloud storage providers.
  • Duplicity: Incremental backups with encryption support.

Related posts