---
title: "Tdarr Monitoring Configuration"
description: "Active Discord monitoring system for Tdarr worker timeouts, staging stalls, and operational status, covering log-based polling, API-based health checks, and automated cleanup with Discord webhook alerts."
type: reference
domain: docker
tags: [tdarr, monitoring, discord, webhook, cron, systemd, api, alerting, automation]
---

# Tdarr Monitoring Configuration - Discord Integration

## Overview
This document describes the active Discord monitoring system for Tdarr worker timeouts, stalls, and operational status across the homelab environment.

## System Architecture

### Components
- **Tdarr Server**: ubuntu-ct (10.10.0.43) running `tdarr-clean` container
- **Tdarr Node**: nobara-pc (unmapped architecture) running `tdarr-node-gpu-unmapped`
- **Monitoring Script**: `/scripts/monitoring/tdarr-timeout-monitor.sh`
- **Discord Integration**: Webhook-based notifications to designated channel

### Monitoring Targets
✅ **Server Limbo Timeouts** - Files stuck in staging beyond timeout period  
✅ **Node Worker Stalls** - Workers that hang during transcoding operations  
✅ **Worker Disconnections** - Unexpected worker disconnects  
✅ **Success Notifications** - Optional completion tracking (currently disabled)

## Active Configuration

### Script Location
```
/mnt/NV2/Development/claude-home/scripts/monitoring/tdarr-timeout-monitor.sh
```

### Key Settings
- **Polling Interval**: 15 minutes (900 seconds)
- **Discord Webhook**: Configured and active
- **Server Connection**: SSH alias `tdarr` → ubuntu-ct (10.10.0.43)
- **Node Container**: `tdarr-node-gpu-unmapped` via Podman
- **Working Directory**: `/tmp/tdarr-monitor/`

### Monitored Log Patterns

#### Server Timeout Events
```bash
# Pattern: "has been in limbo for X seconds, removing from staging section"
grep -i "has been in limbo"
```
**Example Alert**:
```
⚠️ 4 file(s) timed out in staging:
TV/Survivor/Season 48/Survivor (2000) - S48E04...
TV/Survivor/Season 48/Survivor (2000) - S48E11...
TV/Survivor/Season 26/Survivor (2000) - S26E05...
TV/Survivor/Season 22/Survivor (2000) - S22E13...
Files were removed from staging and will retry.
```

#### Node Worker Events
```bash
# Patterns: "Worker X has stalled, cancelling" | "Worker X disconnected. Pruning."
grep -i "worker.*stalled\|worker.*disconnected"
```
**Example Alert**:
```
🔴 4 worker stall(s) detected:
Worker eager-eyas
Worker oblong-owl
Worker other-olm
Worker dry-dugong
Workers were cancelled and will restart.
```

## Historical Context

### Problem Timeline
1. **Initial Issue**: Multiple workers stalling due to resource competition
2. **Worker Reduction**: Reduced to 1 CPU + 1 GPU worker each
3. **Timeout Increase**: Extended staging timeout from 5 minutes to 15-20 minutes
4. **SMB Optimization**: Improved file transfer from 61.8 MB/s to 103 MB/s
5. **Monitoring Implementation**: Custom Discord alerts for proactive issue detection

### Root Causes Addressed
- **Large File Timeouts**: TV episodes (3-8GB+) exceeding 5-minute staging timeout
- **Worker Competition**: Multiple workers competing for GPU resources
- **Network Performance**: Slow SMB transfers causing download delays
- **Node Instability**: Workers hanging during complex transcoding flows

## Deployment Status

### Current State
✅ **Script Deployed**: Located and executable  
✅ **Discord Webhook**: Configured and tested  
✅ **Function Order**: Fixed (send_discord_notification defined before use)  
✅ **Polling Interval**: Set to 15 minutes  
✅ **Initial Test**: "monitoring started" message sent successfully  

### Automation Setup
**Status**: Ready for automation - choose deployment method:

#### Option A: Cron Job
```bash
# Edit user crontab
crontab -e

# Add monitoring every 15 minutes
*/15 * * * * /mnt/NV2/Development/claude-home/scripts/monitoring/tdarr-timeout-monitor.sh >/dev/null 2>&1
```

#### Option B: Systemd Service (Recommended)
```bash
# Service file
sudo tee /etc/systemd/system/tdarr-monitor.service > /dev/null <<EOF
[Unit]
Description=Tdarr Timeout Monitor
After=network.target

[Service]
Type=oneshot
User=cal
ExecStart=/mnt/NV2/Development/claude-home/scripts/monitoring/tdarr-timeout-monitor.sh
EOF

# Timer file
sudo tee /etc/systemd/system/tdarr-monitor.timer > /dev/null <<EOF
[Unit]
Description=Run Tdarr Monitor every 15 minutes
Requires=tdarr-monitor.service

[Timer]
OnCalendar=*:0/15
Persistent=true

[Install]
WantedBy=timers.target
EOF

# Enable and start
sudo systemctl daemon-reload
sudo systemctl enable tdarr-monitor.timer
sudo systemctl start tdarr-monitor.timer
```

## Discord Integration Details

### Webhook Configuration
- **Channel**: Designated homelab monitoring channel
- **Message Format**: Rich embeds with color coding
- **Alert Colors**: Red (15158332) for errors, Green (3066993) for success
- **Content**: File paths, worker names, timestamps, hostname

### Message Examples

#### Successful Start
```json
{
  "title": "🎬 Tdarr Monitoring Alert",
  "description": "Tdarr timeout monitoring started",
  "color": 3066993,
  "timestamp": "2025-08-10T14:19:55.000Z",
  "footer": {
    "text": "nobara-pc - Sun Aug 10 09:19:55 AM CDT 2025"
  }
}
```

#### Timeout Alert
```json
{
  "title": "🎬 Tdarr Monitoring Alert", 
  "description": "⚠️ **4 file(s) timed out in staging:**\n```\nTV/Survivor/Season 48/...\n```\nFiles were removed from staging and will retry.",
  "color": 15158332,
  "timestamp": "2025-08-10T14:49:00.000Z"
}
```

## Performance Impact

### System Resources
- **CPU Usage**: Minimal - quick log parsing every 15 minutes
- **Network Impact**: One SSH connection + Docker logs query per check
- **Storage**: < 1MB in `/tmp/tdarr-monitor/`
- **Execution Time**: ~2-5 seconds per check

### Benefits
- **Proactive Issue Detection**: Know about problems before manual checking
- **Historical Tracking**: Discord provides persistent log of all alerts
- **Mobile Notifications**: Get alerts on phone via Discord app
- **Reduced Manual Monitoring**: Automated awareness of system health

## Troubleshooting

### Common Issues

#### No Discord Messages
```bash
# Test webhook manually
curl -H "Content-Type: application/json" -X POST \
     -d '{"content":"Manual test message"}' \
     "https://discord.com/api/webhooks/1404105821549498398/y2Ud1RK9rzFjv58xbypUfQNe3jrL7ZUq1FkQHa4_dfOHm2ylp93z0f4tY0O8Z-vQgKhD"

# Check script execution
/mnt/NV2/Development/claude-home/scripts/monitoring/tdarr-timeout-monitor.sh
```

#### SSH Connection Issues
```bash
# Test server connection
ssh tdarr "echo 'SSH working'"

# Check SSH key authentication
ssh-add -l
```

#### Podman Container Access
```bash
# Verify container is running
podman ps | grep tdarr-node

# Test log access
podman logs --tail 5 tdarr-node-gpu-unmapped
```

#### Force Monitoring Check
```bash
# Reset timestamp to trigger immediate check
echo $(($(date +%s) - 1000)) > /tmp/tdarr-monitor/last_check.timestamp

# Run manual check
/mnt/NV2/Development/claude-home/scripts/monitoring/tdarr-timeout-monitor.sh
```

## Security Considerations

### Webhook Protection
- **URL Security**: Webhook URL contains sensitive token
- **Access Control**: Only monitoring script has access to webhook
- **Scope Limitation**: Webhook only has permission to post messages

### SSH Access
- **Key-based Authentication**: No password authentication used
- **Limited Commands**: Only Docker logs and basic system commands
- **Network Isolation**: SSH connection within trusted homelab network

### File Permissions
```bash
# Verify script permissions
ls -la /mnt/NV2/Development/claude-home/scripts/monitoring/tdarr-timeout-monitor.sh
# Should be: -rwxr--r-- (755) or more restrictive

# Secure if needed
chmod 750 /mnt/NV2/Development/claude-home/scripts/monitoring/tdarr-timeout-monitor.sh
```

## Maintenance

### Regular Checks
- **Weekly**: Verify Discord messages are still being received
- **Monthly**: Check `/tmp/tdarr-monitor/` directory size
- **Quarterly**: Review alert frequency and adjust thresholds if needed

### Log Rotation
```bash
# Clean old monitoring logs if they accumulate
find /tmp/tdarr-monitor/ -name "*.log" -mtime +7 -delete
```

### Updates
- **Script Updates**: Version control via git in homelab documentation
- **Webhook Rotation**: Update webhook URL if Discord server changes
- **Threshold Tuning**: Adjust 15-minute interval based on operational experience

## API-Based Monitoring Enhancement

### Tdarr API Monitoring Script
**Location**: `tdarr_monitor.py` - Comprehensive API-based monitoring client
**Server**: http://10.10.0.43:8265 (main Tdarr server)
**Dependencies**: Python 3 with `requests` library

#### Key Features
- **Server Health Monitoring**: Version, uptime, connectivity status
- **Queue Management**: Processing statistics, queue depth, item details  
- **Node Status Tracking**: Online/offline nodes, worker counts, active jobs
- **Library Scan Progress**: File counts, scan status, completion percentage
- **Overall Statistics**: Transcoding metrics, space saved, processing speeds
- **Comprehensive Health Checks**: Multi-component status assessment

#### API Endpoints Monitored
- `/api/v2/get-server-info` - Server version, system info, uptime
- `/api/v2/get-queue` - Current queue status and processing items
- `/api/v2/get-nodes` - Connected nodes and their status  
- `/api/v2/get-libraries` - Library scan progress and file counts
- `/api/v2/get-stats` - Overall transcoding statistics and metrics

#### Usage Examples
```bash
# Quick health check
python3 tdarr_monitor.py --server http://10.10.0.43:8265 --check health

# Queue status monitoring
python3 tdarr_monitor.py --server http://10.10.0.43:8265 --check queue --output json

# Node performance check
python3 tdarr_monitor.py --server http://10.10.0.43:8265 --check nodes --verbose

# Complete system status
python3 tdarr_monitor.py --server http://10.10.0.43:8265 --check all
```

#### Integration with Existing Discord Monitoring
```bash
#!/bin/bash
# Enhanced health monitoring with API integration
HEALTH_OUTPUT=$(python3 tdarr_monitor.py --server http://10.10.0.43:8265 --check health --output json)
STATUS=$(echo "$HEALTH_OUTPUT" | jq -r '.overall_status')

if [ "$STATUS" != "healthy" ]; then
    # Extract specific issues
    SERVER_HEALTHY=$(echo "$HEALTH_OUTPUT" | jq -r '.checks.server.healthy')
    NODE_COUNT=$(echo "$HEALTH_OUTPUT" | jq -r '.checks.nodes.online_count')
    QUEUE_HEALTHY=$(echo "$HEALTH_OUTPUT" | jq -r '.checks.queue.healthy')
    
    MESSAGE="🎬 **Tdarr API Health Alert**\n"
    MESSAGE+="Overall Status: **$STATUS**\n\n"
    
    if [ "$SERVER_HEALTHY" != "true" ]; then
        MESSAGE+="❌ Server: Offline or unreachable\n"
    fi
    
    if [ "$NODE_COUNT" == "0" ]; then
        MESSAGE+="❌ Nodes: No online transcoding nodes\n"
    fi
    
    if [ "$QUEUE_HEALTHY" != "true" ]; then
        MESSAGE+="❌ Queue: Unable to access queue data\n"
    fi
    
    MESSAGE+="\nCheck server status and node connectivity."
    
    # Send via existing Discord webhook
    send_discord_message "$MESSAGE"
fi
```

#### Cron-based API Monitoring
```bash
# API health check every 5 minutes (complement log-based monitoring)
*/5 * * * * /path/to/tdarr_monitor.py --server http://10.10.0.43:8265 --check health >> /tmp/tdarr-api-health.log 2>&1

# Full status check hourly for detailed metrics
0 * * * * /path/to/tdarr_monitor.py --server http://10.10.0.43:8265 --check all --output json > /tmp/tdarr-status.json
```

#### Gaming Scheduler Integration
```bash
# Before starting transcoding, verify server health via API
if python3 tdarr_monitor.py --server http://10.10.0.43:8265 --check health; then
    echo "Tdarr server healthy, proceeding with scheduled start"
    ./start-tdarr-gpu-podman-clean.sh
else
    echo "Tdarr server unhealthy, skipping scheduled start"
    # Send alert via existing Discord system
    send_discord_message "🎮 **Gaming Scheduler Alert**: Tdarr server unhealthy, skipping transcoding session"
fi
```

#### Monitoring Metrics Available
- **Server Metrics**: Uptime, version, system info, connectivity
- **Queue Metrics**: Total items, processing count, queued count, completed count
- **Node Metrics**: Online/offline status, worker counts, active jobs, heartbeat
- **Library Metrics**: Total files, scan progress, library count, scan status  
- **Performance Metrics**: Total transcodes, space saved, processing speed, error rates

## Integration with Other Systems

### Current Monitoring Stack
- **Log-based Monitoring**: Timeout and worker stall detection (15-minute polling)
- **API-based Monitoring**: Real-time health and performance metrics (5-minute polling)
- **Discord Integration**: Unified alerting for both monitoring methods
- **Gaming Scheduler**: API health checks before transcoding sessions

### Future Enhancements
- **Grafana Integration**: Visualize API metrics (queue depth, processing rates, node performance)
- **Prometheus Metrics**: Export API data for time-series analysis
- **Home Assistant**: Integrate server status with home automation
- **Email Backup**: Secondary notification method for critical API alerts
- **Metric Correlation**: Combine log-based alerts with API performance data

### Related Documentation
- [NAS Mount Configuration](../networking/nas-mount-configuration.md) - SMB optimization context
- [Tdarr Troubleshooting](tdarr-troubleshooting.md) - Worker timeout background  
- [SSH Key Management](../networking/ssh-key-management.md) - Server access setup
- [Tdarr Gaming Scheduler](../../scripts/tdarr/README.md) - Gaming-aware automation

---
**Status**: ✅ Log-based monitoring active, API monitoring script created  
**Last Updated**: August 12, 2025  
**Next Review**: September 12, 2025  
**Discord Channel**: Homelab monitoring alerts configured and tested  
**API Endpoint**: http://10.10.0.43:8265 (verified accessible)