Posted on
• Artificial Intelligence

Scheduling Local Artificial Intelligence Jobs with Cron

Author
  • User
    linuxbash
    Posts by this author
    Posts by this author

Scheduling Local Artificial Intelligence Jobs with Cron (Linux)

Ever watched your GPU sit idle at 2 a.m. while datasets pile up and models wait to be retrained? If you’re running AI workloads on your own hardware, cron can be the quiet, reliable teammate that keeps your pipelines humming—even when you’re asleep. In this guide, you’ll learn how to schedule local AI jobs with cron the right way: reproducibly, safely, and with good Linux hygiene.

  • Problem: Manual runs are brittle, easy to forget, and waste off-peak compute time.

  • Value: Cron-based scheduling gives you consistent, auditable runs; better resource utilization; and privacy-friendly automation right on your own machine.

Why schedule local AI jobs with cron?

  • Maximize idle resources: Nighttime/off-hours are perfect for training, batch inference, and data prep.

  • Keep data private: No need to upload sensitive data to cloud jobs.

  • Cost control: Use hardware you already own—no surprise bills.

  • Reproducibility: Cron runs scripts the same way every time; logs tell you when/what happened.

Prerequisites: Install and enable cron

Cron runs as a system service. Install and enable it for your distro:

  • Debian/Ubuntu (service name: cron)

    sudo apt update
sudo apt install cron
sudo systemctl enable --now cron

  • Fedora/RHEL/CentOS (service name: crond)

    sudo dnf install cronie
sudo systemctl enable --now crond

  • openSUSE Leap/Tumbleweed (service name: cron)

    sudo zypper refresh
sudo zypper install cronie
sudo systemctl enable --now cron

Verify it’s running:

systemctl status cron   # Debian/Ubuntu/openSUSE
systemctl status crond  # Fedora/RHEL/CentOS

Optional: Laptops/offline machines benefit from anacron (runs missed jobs when the system is back on).

  • Debian/Ubuntu:

    sudo apt install anacron

  • Fedora/RHEL/CentOS:

    sudo dnf install cronie-anacron

  • openSUSE:

    sudo zypper install anacron

Step 1: Prepare a reliable AI environment

Use a Python virtual environment for repeatability. Install Python and pip:

  • Debian/Ubuntu:

    sudo apt update
sudo apt install -y python3 python3-venv python3-pip

  • Fedora/RHEL/CentOS:

    sudo dnf install -y python3 python3-pip

  • openSUSE:

    sudo zypper install -y python3 python3-pip

Create a project layout:

mkdir -p $HOME/ai-jobs/{bin,logs,work}
python3 -m venv $HOME/.venvs/ai
$HOME/.venvs/ai/bin/pip install --upgrade pip
$HOME/.venvs/ai/bin/pip install transformers sentence-transformers

Example AI job (simple batch task using Transformers). Save as: $HOME/ai-jobs/bin/ai_job.py

#!/usr/bin/env python3
import os, datetime, pathlib
from transformers import pipeline

outdir = pathlib.Path(os.environ.get("AI_OUTDIR", str(pathlib.Path.home() / "ai-jobs" / "work")))
outdir.mkdir(parents=True, exist_ok=True)

clf = pipeline("sentiment-analysis")
texts = [
    "This release is fantastic!",
    "The training run failed unexpectedly.",
    "The new dataset looks promising."
]
results = clf(texts)

ts = datetime.datetime.utcnow().isoformat()
with open(outdir / f"sentiment_{ts}.txt", "w") as f:
    for t, r in zip(texts, results):
        f.write(f"{t}\t{r}\n")

print(f"[{ts}] Wrote results to {outdir}")

Make it executable:

chmod +x $HOME/ai-jobs/bin/ai_job.py

Step 2: Wrap your job for cron (env, logging, locks, ethics to your fans)

Cron doesn’t load your shell profile and has a minimal PATH. Use a wrapper that:

  • Activates the venv

  • Sets cache directories (avoid re-downloading models each run)

  • Prevents overlapping runs (flock)

  • Nice/ionice to be a good neighbor

  • Optionally runs only when the GPU is idle

Save as $HOME/ai-jobs/bin/run_inference.sh:

#!/usr/bin/env bash
set -euo pipefail

JOB_NAME="nightly_inference"
BASE_DIR="$HOME/ai-jobs"
VENV="$HOME/.venvs/ai"
LOG_DIR="$BASE_DIR/logs"
WORK_DIR="$BASE_DIR/work"
LOCK_FILE="/tmp/${JOB_NAME}.lock"
mkdir -p "$LOG_DIR" "$WORK_DIR"

# Log file with UTC timestamp
STAMP="$(date -u +'%Y%m%dT%H%M%SZ')"
LOG_FILE="$LOG_DIR/${JOB_NAME}_${STAMP}.log"

# Redirect stdout/stderr to log
exec > >(tee -a "$LOG_FILE") 2>&1

echo "[$(date -u +'%F %T')] Starting $JOB_NAME..."

# Ensure predictable environment for cron
export SHELL=/bin/bash
export PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:$HOME/.local/bin"
export HF_HOME="$HOME/.cache/huggingface"
export TRANSFORMERS_CACHE="$HF_HOME/transformers"
export AI_OUTDIR="$WORK_DIR"

# Optional: be nice to other workloads
IONICE_CLASS=${IONICE_CLASS:-2}  # best-effort
IONICE_PRIO=${IONICE_PRIO:-7}    # lowest priority
NICENESS=${NICENESS:-10}

# Optional: run only if GPU is sufficiently idle (skip if no NVIDIA GPU)
GPU_MAX_USED_MB=${GPU_MAX_USED_MB:-400}  # lower = stricter
if command -v nvidia-smi >/dev/null 2>&1; then
  USED=$(nvidia-smi --query-gpu=memory.used --format=csv,noheader,nounits | head -n1 || echo 999999)
  if [ "${USED:-999999}" -gt "$GPU_MAX_USED_MB" ]; then
    echo "GPU busy (${USED}MB used > ${GPU_MAX_USED_MB}MB). Skipping."
    exit 0
  fi
fi

# Prevent overlapping runs
flock -n "$LOCK_FILE" bash -c "
  set -euo pipefail
  source \"$VENV/bin/activate\"
  ionice -c $IONICE_CLASS -n $IONICE_PRIO nice -n $NICENESS \
    python \"$BASE_DIR/bin/ai_job.py\"
"

echo "[$(date -u +'%F %T')] Finished $JOB_NAME."

Make it executable:

chmod +x $HOME/ai-jobs/bin/run_inference.sh

Step 3: Add the cron schedule

Edit your user crontab:

crontab -e

At the top, set a sane environment (cron’s default PATH is small):

SHELL=/bin/bash
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:$HOME/.local/bin
MAILTO=you@example.com

Examples:

  • Run nightly at 02:05 UTC:

    5 2 * * * /home/youruser/ai-jobs/bin/run_inference.sh

  • Every 15 minutes (will self-skip if GPU busy):

    */15 * * * * /home/youruser/ai-jobs/bin/run_inference.sh

  • Use special strings:

    @daily  /home/youruser/ai-jobs/bin/run_inference.sh
@weekly /home/youruser/ai-jobs/bin/run_inference.sh

Tips:

  • Always use absolute paths in cron.

  • If you need variables, set them in the crontab or inside your wrapper.

  • For system-wide jobs, consider root’s crontab (sudo crontab -e) or drop scripts into /etc/cron.{hourly,daily,weekly} using run-parts naming rules.

Step 4: Monitor and troubleshoot

  • Read logs you wrote:

    ls -1 $HOME/ai-jobs/logs
tail -n 200 -f $HOME/ai-jobs/logs/nightly_inference_*.log

  • Check cron service:

    systemctl status cron    # Debian/Ubuntu/openSUSE
systemctl status crond   # Fedora/RHEL/CentOS

  • Cron’s own logs:

    • systemd journal: journalctl -u cron or journalctl -u crond
    • Some distros also log to /var/log/cron or /var/log/syslog

  • Confirm your entries:

    crontab -l

  • Get email on failures by setting MAILTO in crontab and ensuring local MTA or use a sendmail wrapper.

Real-world patterns you can copy

1) Nightly document embeddings

  • Pull new files from a folder, generate embeddings with Sentence-Transformers, push to a vector DB (e.g., SQLite/FAISS).

  • Schedule: 5 1 * * * to hit off-hours.

2) Weekly quantization/pruning

  • Quantize a model (e.g., 4-bit) to free VRAM for the week.

  • Heavy CPU/GPU usage—schedule @weekly on weekends; add nice/ionice to behave well.

3) Rolling batch inference

  • Every 15 minutes, process the last N items in a queue and write outputs atomically.

  • Use flock to avoid overlaps and a GPU-idle check to yield to interactive sessions.

4) Data hygiene + retrain

  • Nightly: clean mislabeled rows, recompute stats, kick off a quick fine-tune if there’s enough new data.

  • Chain jobs with lockfiles so retrain only runs if preprocessing succeeded (e.g., have the first job drop a “success” marker file).

Common pitfalls (and fixes)

  • My job runs manually but fails in cron

    • Use absolute paths, don’t rely on ~ or implicit PATH.
    • Export needed env vars in the wrapper.
    • Activate venv explicitly with source /path/to/venv/bin/activate.

  • First run is slow due to model downloads

    • Set HF_HOME/TRANSFORMERS_CACHE to a persistent path; let the first run warm the cache.

  • Jobs overlap and fight for resources

    • Wrap invocation with flock -n /tmp/job.lock -c "...".

  • Cron missed runs on a laptop

    • Install anacron to catch up after suspend/offline periods.

Conclusion and next step

Cron can be your local MLOps engine: predictable, low-friction, and perfect for squeezing value out of your own hardware. Start with one job—like nightly inference—then grow into a small suite of reliable tasks.

Call to action:

  • Install and enable cron (see commands above).

  • Create the venv and wrapper script.

  • Add a simple nightly job with crontab -e.

  • Watch the logs tomorrow morning and iterate.

When you’re ready, extend with:

  • Multiple jobs with dependency markers

  • GPU-aware scheduling heuristics

  • Separate users or containers for stronger isolation

Happy automating!

Related posts