Automating Log File Management with Bash Scripts

Log file management is essential for maintaining a healthy system, especially when dealing with large volumes of log data. Bash scripts can automate tasks like log rotation, archiving, and cleanup to ensure disk space is conserved and logs remain organized. This guide provides a step-by-step approach to creating a script for managing log files.

---

Step 1: Writing a Basic Log Management Script

Here’s a foundational Bash script to handle basic log file management tasks such as archiving and cleanup:

#!/bin/bash

# Variables
LOG_DIR="/var/log/myapp"     # Directory containing log files
ARCHIVE_DIR="/var/log/archive" # Directory for archived logs
RETENTION_DAYS=30             # Number of days to retain logs
LOG_FILE="/var/log/log_management.log" # Log file for script actions

# Function to archive logs
archive_logs() {
  mkdir -p "$ARCHIVE_DIR"
  for LOG in "$LOG_DIR"/*.log; do
    gzip "$LOG"
    mv "$LOG.gz" "$ARCHIVE_DIR"
    echo "[$(date)] INFO: Archived $LOG to $ARCHIVE_DIR." >> "$LOG_FILE"
  done
}

# Function to clean up old logs
cleanup_logs() {
  find "$ARCHIVE_DIR" -type f -mtime +$RETENTION_DAYS -exec rm -f {} \;
  echo "[$(date)] INFO: Cleaned up logs older than $RETENTION_DAYS days in $ARCHIVE_DIR." >> "$LOG_FILE"
}

# Main script
case $1 in
  archive)
    archive_logs
    ;;
  cleanup)
    cleanup_logs
    ;;
  all)
    archive_logs
    cleanup_logs
    ;;
  *)
    echo "Usage: $0 {archive|cleanup|all}" >> "$LOG_FILE"
    ;;
esac

To see output, view the $LOG_FILE location, as such: cat /var/log/log_management.log

---

Step 2: Automating the Script with Cron

To automate log file management tasks, schedule the script using cron. For example, to archive logs daily and clean up old logs weekly:

1. Open the crontab editor:

   crontab -e
   

2. Add entries for archiving and cleanup:

   # Archive logs daily at midnight
   0 0 * * * /var/log/log_management.sh archive
   
   # Clean up old logs every Sunday at 1 AM
   0 1 * * 0 /var/log/log_management.sh cleanup
   

3. Save and exit. The tasks will now execute automatically.

---

Step 3: Enhancements and Customizations

1. Email Notifications

Add functionality to send email alerts for errors or status updates:

send_email_notification() {
  local SUBJECT="Log Management Notification"
  local BODY="$1"
  echo "$BODY" | mailx -s "$SUBJECT" admin@example.com
}

# Call this function after critical actions or errors
send_email_notification "Archived logs and cleaned up old files."

2. Handling Multiple Log Directories

Extend the script to handle logs from multiple directories:

LOG_DIRS=("/var/log/app1" "/var/log/app2")
for DIR in "${LOG_DIRS[@]}"; do
  LOG_DIR="$DIR"
  ARCHIVE_DIR="$DIR/archive"
  archive_logs
  cleanup_logs
done

3. Compression Alternatives

Use alternative compression tools like bzip2 or xz for better compression ratios:

archive_logs() {
  mkdir -p "$ARCHIVE_DIR"
  for LOG in "$LOG_DIR"/*.log; do
    xz "$LOG"
    mv "$LOG.xz" "$ARCHIVE_DIR"
    echo "[$(date)] INFO: Archived $LOG to $ARCHIVE_DIR using xz." >> "$LOG_FILE"
  done
}

4. Integration with Monitoring Tools

Export log management data for use with monitoring tools like Grafana:

export_metrics() {
  local METRICS_FILE="/var/log/metrics.csv"
  echo "Timestamp,Archived Logs,Deleted Logs" > "$METRICS_FILE"
  local ARCHIVED=$(ls "$ARCHIVE_DIR" | wc -l)
  local DELETED=$(find "$ARCHIVE_DIR" -type f -mtime +$RETENTION_DAYS | wc -l)
  echo "$(date),$ARCHIVED,$DELETED" >> "$METRICS_FILE"
}

# Call this function periodically to update metrics
export_metrics

---

Conclusion

Automating log file management with Bash scripts simplifies tasks like archiving, cleanup, and monitoring, ensuring logs are managed efficiently. By adding customizations like email notifications, compression options, and monitoring integration, you can tailor the solution to meet specific needs. Start building your log management script today to maintain a well-organized and resource-efficient system.