claude-home/monitoring/examples/cron-job-management.md

title	description	type	domain	tags
Cron Job Management Patterns	Cron job patterns for homelab monitoring including current schedules, HERE document best practices, resource-aware scheduling, security considerations, and debugging techniques for cron environment issues.	reference	monitoring	cron scheduling tdarr bash automation maintenance

Cron Job Management Patterns

This document outlines the cron job patterns and management strategies used in the home lab environment.

Current Cron Schedule

Overview

# Monthly maintenance
0 2 1 * * /home/cal/bin/ssh_key_maintenance.sh

# Tdarr monitoring and management
*/10 * * * * python3 /mnt/NV2/Development/claude-home/scripts/tdarr_monitor.py --server http://10.10.0.43:8265 --check nodes --detect-stuck --discord-alerts >/dev/null 2>&1
0 */6 * * * find "/mnt/NV2/tdarr-cache/nobara-pc-gpu-unmapped/temp/" -name "tdarr-workDir2-*" -type d -mmin +360 -exec rm -rf {} \; 2>/dev/null || true
0 3 * * * find "/mnt/NV2/tdarr-cache/nobara-pc-gpu-unmapped/media" -name "*.temp" -o -name "*.tdarr" -mtime +1 -delete 2>/dev/null || true

# Disabled/legacy jobs
#*/20 * * * * /mnt/NV2/Development/claude-home/scripts/monitoring/tdarr-timeout-monitor.sh

Job Categories

1. System Maintenance

SSH Key Maintenance

• Schedule: 0 2 1 * * (Monthly, 1st at 2 AM)
• Purpose: Maintain SSH key security and rotation
• Location: /home/cal/bin/ssh_key_maintenance.sh
• Priority: High (security-critical)

2. Monitoring & Alerting

Tdarr System Monitoring

• Schedule: */10 * * * * (Every 10 minutes)
• Purpose: Monitor Tdarr nodes, detect stuck jobs, send Discord alerts
• Features:
  • Stuck job detection (30-minute threshold)
  • Discord notifications with rich embeds
  • Persistent memory state tracking
• Script: /mnt/NV2/Development/claude-home/scripts/tdarr_monitor.py
• Output: Silent (>/dev/null 2>&1)

3. Cleanup & Housekeeping

Tdarr Work Directory Cleanup

• Schedule: 0 */6 * * * (Every 6 hours)
• Purpose: Remove stale Tdarr work directories
• Target: /mnt/NV2/tdarr-cache/nobara-pc-gpu-unmapped/temp/
• Pattern: tdarr-workDir2-* directories
• Age threshold: 6 hours (-mmin +360)

Failed Tdarr Job Cleanup

• Schedule: 0 3 * * * (Daily at 3 AM)
• Purpose: Remove failed transcode artifacts
• Target: /mnt/NV2/tdarr-cache/nobara-pc-gpu-unmapped/media/
• Patterns: *.temp and *.tdarr files
• Age threshold: 24 hours (-mtime +1)

Design Patterns

1. Absolute Paths

Always use absolute paths in cron jobs

# Good
*/10 * * * * python3 /full/path/to/script.py

# Bad - relative paths don't work in cron
*/10 * * * * python3 scripts/script.py

2. Error Handling

Standard error suppression pattern

command 2>/dev/null || true

• Suppresses stderr to prevent cron emails
• || true ensures job always exits successfully

3. Time-based Cleanup

Safe age thresholds for different content types

• Work directories: 6 hours (short-lived, safe for active jobs)
• Temp files: 24 hours (allows for long transcodes)
• Log files: 7-30 days (depending on importance)

4. Resource-aware Scheduling

Avoid resource conflicts

# System maintenance at low-usage times
0 2 1 * * maintenance_script.sh

# Cleanup during off-peak hours  
0 3 * * * cleanup_script.sh

# Monitoring with high frequency during active hours
*/10 * * * * monitor_script.py

Management Workflow

Adding New Cron Jobs

1. Backup current crontab

   crontab -l > /tmp/crontab_backup_$(date +%Y%m%d)
   

2. Edit safely

   crontab -l > /tmp/new_crontab
   echo "# New job description" >> /tmp/new_crontab
   echo "schedule command" >> /tmp/new_crontab
   crontab /tmp/new_crontab
   

3. Verify installation

   crontab -l
   

Proper HERE Document (EOF) Usage

When building cron files with HERE documents, use proper EOF formatting:

✅ Correct Format

cat > /tmp/new_crontab << 'EOF'
0 2 1 * * /home/cal/bin/ssh_key_maintenance.sh
# Tdarr monitoring every 10 minutes
*/10 * * * * python3 /path/to/script.py --args
EOF

❌ Common Mistakes

# BAD - Causes "EOF not found" errors
cat >> /tmp/crontab << 'EOF'
new_cron_job
EOF

# Results in malformed file with literal "EOF < /dev/null" lines

Key Rules for EOF in Cron Files

1. Use cat > not cat >> for building complete files

   # Good - overwrites file cleanly
   cat > /tmp/crontab << 'EOF'
   
   # Bad - appends and can create malformed files
   cat >> /tmp/crontab << 'EOF'
   

2. Quote the EOF delimiter to prevent variable expansion

   # Good - literal content
   cat > file << 'EOF'
   
   # Can cause issues with special characters
   cat > file << EOF
   

3. Clean up malformed files before installing

   # Remove EOF artifacts and empty lines
   head -n -1 /tmp/crontab > /tmp/clean_crontab
   
   # Or use grep to remove EOF lines
   grep -v "^EOF" /tmp/crontab > /tmp/clean_crontab
   

4. Alternative approach - direct echo method

   crontab -l > /tmp/current_crontab
   echo "# New job comment" >> /tmp/current_crontab
   echo "*/10 * * * * /path/to/command" >> /tmp/current_crontab
   crontab /tmp/current_crontab
   

Debugging EOF Issues

# Check for EOF artifacts in crontab file
cat -n /tmp/crontab | grep EOF

# Validate crontab syntax before installing
crontab -T /tmp/crontab  # Some systems support this

# Manual cleanup if needed
sed '/^EOF/d' /tmp/crontab > /tmp/clean_crontab

Testing Cron Jobs

Test command syntax first

# Test the actual command before scheduling
python3 /full/path/to/script.py --test

# Check file permissions
ls -la /path/to/script

# Verify paths exist
ls -la /target/directory/

Test with minimal frequency

# Start with 5-minute intervals for testing
*/5 * * * * /path/to/new/script.sh

# Monitor logs
tail -f /var/log/syslog | grep CRON

Monitoring Cron Jobs

Check cron logs

# System cron logs
sudo journalctl -u cron -f

# User cron logs  
grep CRON /var/log/syslog | grep $(whoami)

Verify job execution

# Check if cleanup actually ran
ls -la /target/cleanup/directory/

# Monitor script logs
tail -f /path/to/script/logs/

Security Considerations

1. Path Security

• Use absolute paths to prevent PATH manipulation
• Ensure scripts are owned by correct user
• Set appropriate permissions (750 for scripts)

2. Command Injection Prevention

# Good - quoted paths
find "/path/with spaces/" -name "pattern"

# Bad - unquoted paths vulnerable to injection
find /path/with spaces/ -name pattern

3. Resource Limits

• Prevent runaway processes with timeout
• Use ionice for I/O intensive cleanup jobs
• Consider nice for CPU-intensive tasks

Troubleshooting

Common Issues

Job not running

1. Check cron service: sudo systemctl status cron
2. Verify crontab syntax: crontab -l
3. Check file permissions and paths
4. Review cron logs for errors

Environment differences

• Cron runs with minimal environment
• Set PATH explicitly if needed
• Use absolute paths for all commands

Silent failures

• Remove 2>/dev/null temporarily for debugging
• Add logging to scripts
• Check script exit codes

Debugging Commands

# Test cron environment
* * * * * env > /tmp/cron_env.txt

# Test script in cron-like environment
env -i /bin/bash -c 'your_command_here'

# Monitor real-time execution
sudo tail -f /var/log/syslog | grep CRON

Best Practices

1. Documentation

• Comment all cron jobs with purpose and schedule
• Document in this patterns file
• Include contact info for complex jobs

2. Maintenance

• Regular review of active jobs (quarterly)
• Remove obsolete jobs promptly
• Update absolute paths when moving scripts

3. Monitoring

• Implement health checks for critical jobs
• Use Discord/email notifications for failures
• Monitor disk space usage from cleanup jobs

4. Backup Strategy

• Backup crontab before changes
• Version control cron configurations
• Document restoration procedures

Future Enhancements

Planned Additions

• Log rotation: Automated cleanup of application logs
• Health checks: System resource monitoring
• Backup verification: Automated backup integrity checks
• Certificate renewal: SSL/TLS certificate automation

Migration Considerations

• Systemd timers: Consider migration for complex scheduling
• Configuration management: Ansible or similar for multi-host
• Centralized logging: Aggregated cron job monitoring

---

Related Documentation

• Tdarr Monitoring Script
• System Maintenance
• Discord Integration