# Tdarr Discord Monitoring Setup Guide

## Overview
This guide sets up automated Discord notifications for Tdarr worker timeouts, stalls, and completions using a custom log monitoring script.

## Prerequisites
- Discord server where you want notifications
- Administrative access to create webhooks
- Tdarr server accessible via SSH
- Podman/Docker access to Tdarr node

## Setup Steps

### 1. Create Discord Webhook
1. Go to your Discord server → **Server Settings** → **Integrations** → **Webhooks**
2. Click **Create Webhook**  
3. Name it "Tdarr Monitor" and select the channel for notifications
4. Copy the **Webhook URL** (keep this secure!)

### 2. Configure Monitoring Script
Edit the script to add your webhook:
```bash
nano /mnt/NV2/Development/claude-home/scripts/monitoring/tdarr-timeout-monitor.sh
```

Update these lines:
```bash
DISCORD_WEBHOOK="https://discord.com/api/webhooks/YOUR_WEBHOOK_ID/YOUR_WEBHOOK_TOKEN"
SERVER_HOST="tdarr"  # Your SSH alias for Tdarr server
NODE_CONTAINER="tdarr-node-gpu-unmapped"  # Your node container name
```

### 3. Make Script Executable
```bash
chmod +x /mnt/NV2/Development/claude-home/scripts/monitoring/tdarr-timeout-monitor.sh
```

### 4. Test the Script
```bash
/mnt/NV2/Development/claude-home/scripts/monitoring/tdarr-timeout-monitor.sh
```

You should see a "monitoring started" message in your Discord channel.

### 5. Setup Automated Monitoring (Choose One)

#### Option A: Cron Job (Simple)
```bash
# Edit crontab
crontab -e

# Add this line to check every 5 minutes
*/5 * * * * /mnt/NV2/Development/claude-home/scripts/monitoring/tdarr-timeout-monitor.sh >/dev/null 2>&1
```

#### Option B: Systemd Service (Advanced)
Create a systemd service for more reliable monitoring:

```bash
sudo nano /etc/systemd/system/tdarr-monitor.service
```

Content:
```ini
[Unit]
Description=Tdarr Timeout Monitor
After=network.target

[Service]
Type=oneshot
User=cal
ExecStart=/mnt/NV2/Development/claude-home/scripts/monitoring/tdarr-timeout-monitor.sh
```

Create timer:
```bash
sudo nano /etc/systemd/system/tdarr-monitor.timer
```

Content:
```ini
[Unit]
Description=Run Tdarr Monitor every 5 minutes
Requires=tdarr-monitor.service

[Timer]
OnCalendar=*:0/5
Persistent=true

[Install]
WantedBy=timers.target
```

Enable and start:
```bash
sudo systemctl daemon-reload
sudo systemctl enable tdarr-monitor.timer
sudo systemctl start tdarr-monitor.timer
```

## Notification Examples

### Worker Timeout Alert
```
🎬 Tdarr Monitoring Alert
⚠️ 4 file(s) timed out in staging:
```
TV/Survivor/Season 48/Survivor (2000) - S48E04...
TV/Survivor/Season 48/Survivor (2000) - S48E11...  
TV/Survivor/Season 26/Survivor (2000) - S26E05...
```
Files were removed from staging and will retry.
```

### Worker Stall Alert
```
🎬 Tdarr Monitoring Alert
🔴 2 worker stall(s) detected:
```
Worker eager-eyas
Worker oblong-owl
```
Workers were cancelled and will restart.
```

### Success Notification (Optional)
```
🎬 Tdarr Monitoring Alert  
✅ 3 transcode(s) completed successfully in the last check period.
```

## Monitoring Features

✅ **Server Limbo Timeouts** - Files stuck in staging > timeout period  
✅ **Node Worker Stalls** - Workers that hang during transcoding  
✅ **Success Notifications** - Optional completion alerts  
✅ **Smart Timing** - Only checks every 60+ seconds to avoid spam  
✅ **Rich Discord Embeds** - Color-coded messages with timestamps  

## Customization Options

### Disable Success Messages
Edit the script and comment out this line:
```bash
# check_completions  # Comment out to disable success notifications
```

### Change Check Frequency
For cron job, modify the timing:
```bash
*/10 * * * *  # Check every 10 minutes instead of 5
```

For systemd timer, update OnCalendar:
```ini
OnCalendar=*:0/10  # Check every 10 minutes
```

### Add More Monitoring
You can extend the script to monitor:
- Disk space on cache directory
- Network connectivity to TrueNAS
- GPU utilization during transcoding
- Queue depth and processing rates

## Troubleshooting

### No Notifications Received
1. Check webhook URL is correct and accessible
2. Test webhook manually:
   ```bash
   curl -H "Content-Type: application/json" -X POST -d '{"content":"Test message"}' "YOUR_WEBHOOK_URL"
   ```
3. Check script logs: `/tmp/tdarr-monitor/`

### False Positives
- Adjust the timing logic in the script
- Filter out specific log patterns that aren't actual errors
- Tune the timeout thresholds

### Missing SSH Access
- Ensure SSH key authentication is set up for the `tdarr` server
- Test: `ssh tdarr "echo 'SSH working'"` 

## Security Notes
- Keep your Discord webhook URL private
- Consider using environment variables for sensitive data
- Restrict file permissions on the script (chmod 750)

---
*This monitoring solution provides real-time alerts for Tdarr issues without requiring external monitoring infrastructure.*