CLAUDE: Add comprehensive documentation for Tdarr monitoring and NAS configuration
Complete documentation package for home lab infrastructure: ## New Documentation Files: - **Tdarr Monitoring Configuration**: Complete setup guide for Discord-based Tdarr monitoring system - **NAS Mount Configuration**: SMB/CIFS mount setup and troubleshooting for media storage - **Discord Monitoring Setup**: Step-by-step guide for webhook configuration and notification testing ## Documentation Features: - **Reference Architecture**: Best practices for distributed Tdarr deployments - **Configuration Templates**: Copy-paste ready configurations with security considerations - **Troubleshooting Guides**: Common issues and solutions for production environments - **Integration Examples**: Real-world implementation patterns for home lab environments ## Coverage Areas: - Docker container orchestration and monitoring - Network storage integration and performance optimization - Automated alerting and notification systems - Production-ready configuration management These documents support the enhanced monitoring system and provide comprehensive guidance for maintaining a robust home lab infrastructure. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
Cal Corum
parent
6cc0d0df2e
commit
715354da7d
282
reference/docker/tdarr-monitoring-configuration.md
Normal file
282
reference/docker/tdarr-monitoring-configuration.md
Normal file
@ -0,0 +1,282 @@
|
||||
# Tdarr Monitoring Configuration - Discord Integration
|
||||
|
||||
## Overview
|
||||
This document describes the active Discord monitoring system for Tdarr worker timeouts, stalls, and operational status across the homelab environment.
|
||||
|
||||
## System Architecture
|
||||
|
||||
### Components
|
||||
- **Tdarr Server**: ubuntu-ct (10.10.0.43) running `tdarr-clean` container
|
||||
- **Tdarr Node**: nobara-pc (unmapped architecture) running `tdarr-node-gpu-unmapped`
|
||||
- **Monitoring Script**: `/scripts/monitoring/tdarr-timeout-monitor.sh`
|
||||
- **Discord Integration**: Webhook-based notifications to designated channel
|
||||
|
||||
### Monitoring Targets
|
||||
✅ **Server Limbo Timeouts** - Files stuck in staging beyond timeout period
|
||||
✅ **Node Worker Stalls** - Workers that hang during transcoding operations
|
||||
✅ **Worker Disconnections** - Unexpected worker disconnects
|
||||
✅ **Success Notifications** - Optional completion tracking (currently disabled)
|
||||
|
||||
## Active Configuration
|
||||
|
||||
### Script Location
|
||||
```
|
||||
/mnt/NV2/Development/claude-home/scripts/monitoring/tdarr-timeout-monitor.sh
|
||||
```
|
||||
|
||||
### Key Settings
|
||||
- **Polling Interval**: 15 minutes (900 seconds)
|
||||
- **Discord Webhook**: Configured and active
|
||||
- **Server Connection**: SSH alias `tdarr` → ubuntu-ct (10.10.0.43)
|
||||
- **Node Container**: `tdarr-node-gpu-unmapped` via Podman
|
||||
- **Working Directory**: `/tmp/tdarr-monitor/`
|
||||
|
||||
### Monitored Log Patterns
|
||||
|
||||
#### Server Timeout Events
|
||||
```bash
|
||||
# Pattern: "has been in limbo for X seconds, removing from staging section"
|
||||
grep -i "has been in limbo"
|
||||
```
|
||||
**Example Alert**:
|
||||
```
|
||||
⚠️ 4 file(s) timed out in staging:
|
||||
TV/Survivor/Season 48/Survivor (2000) - S48E04...
|
||||
TV/Survivor/Season 48/Survivor (2000) - S48E11...
|
||||
TV/Survivor/Season 26/Survivor (2000) - S26E05...
|
||||
TV/Survivor/Season 22/Survivor (2000) - S22E13...
|
||||
Files were removed from staging and will retry.
|
||||
```
|
||||
|
||||
#### Node Worker Events
|
||||
```bash
|
||||
# Patterns: "Worker X has stalled, cancelling" | "Worker X disconnected. Pruning."
|
||||
grep -i "worker.*stalled\|worker.*disconnected"
|
||||
```
|
||||
**Example Alert**:
|
||||
```
|
||||
🔴 4 worker stall(s) detected:
|
||||
Worker eager-eyas
|
||||
Worker oblong-owl
|
||||
Worker other-olm
|
||||
Worker dry-dugong
|
||||
Workers were cancelled and will restart.
|
||||
```
|
||||
|
||||
## Historical Context
|
||||
|
||||
### Problem Timeline
|
||||
1. **Initial Issue**: Multiple workers stalling due to resource competition
|
||||
2. **Worker Reduction**: Reduced to 1 CPU + 1 GPU worker each
|
||||
3. **Timeout Increase**: Extended staging timeout from 5 minutes to 15-20 minutes
|
||||
4. **SMB Optimization**: Improved file transfer from 61.8 MB/s to 103 MB/s
|
||||
5. **Monitoring Implementation**: Custom Discord alerts for proactive issue detection
|
||||
|
||||
### Root Causes Addressed
|
||||
- **Large File Timeouts**: TV episodes (3-8GB+) exceeding 5-minute staging timeout
|
||||
- **Worker Competition**: Multiple workers competing for GPU resources
|
||||
- **Network Performance**: Slow SMB transfers causing download delays
|
||||
- **Node Instability**: Workers hanging during complex transcoding flows
|
||||
|
||||
## Deployment Status
|
||||
|
||||
### Current State
|
||||
✅ **Script Deployed**: Located and executable
|
||||
✅ **Discord Webhook**: Configured and tested
|
||||
✅ **Function Order**: Fixed (send_discord_notification defined before use)
|
||||
✅ **Polling Interval**: Set to 15 minutes
|
||||
✅ **Initial Test**: "monitoring started" message sent successfully
|
||||
|
||||
### Automation Setup
|
||||
**Status**: Ready for automation - choose deployment method:
|
||||
|
||||
#### Option A: Cron Job
|
||||
```bash
|
||||
# Edit user crontab
|
||||
crontab -e
|
||||
|
||||
# Add monitoring every 15 minutes
|
||||
*/15 * * * * /mnt/NV2/Development/claude-home/scripts/monitoring/tdarr-timeout-monitor.sh >/dev/null 2>&1
|
||||
```
|
||||
|
||||
#### Option B: Systemd Service (Recommended)
|
||||
```bash
|
||||
# Service file
|
||||
sudo tee /etc/systemd/system/tdarr-monitor.service > /dev/null <<EOF
|
||||
[Unit]
|
||||
Description=Tdarr Timeout Monitor
|
||||
After=network.target
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
User=cal
|
||||
ExecStart=/mnt/NV2/Development/claude-home/scripts/monitoring/tdarr-timeout-monitor.sh
|
||||
EOF
|
||||
|
||||
# Timer file
|
||||
sudo tee /etc/systemd/system/tdarr-monitor.timer > /dev/null <<EOF
|
||||
[Unit]
|
||||
Description=Run Tdarr Monitor every 15 minutes
|
||||
Requires=tdarr-monitor.service
|
||||
|
||||
[Timer]
|
||||
OnCalendar=*:0/15
|
||||
Persistent=true
|
||||
|
||||
[Install]
|
||||
WantedBy=timers.target
|
||||
EOF
|
||||
|
||||
# Enable and start
|
||||
sudo systemctl daemon-reload
|
||||
sudo systemctl enable tdarr-monitor.timer
|
||||
sudo systemctl start tdarr-monitor.timer
|
||||
```
|
||||
|
||||
## Discord Integration Details
|
||||
|
||||
### Webhook Configuration
|
||||
- **Channel**: Designated homelab monitoring channel
|
||||
- **Message Format**: Rich embeds with color coding
|
||||
- **Alert Colors**: Red (15158332) for errors, Green (3066993) for success
|
||||
- **Content**: File paths, worker names, timestamps, hostname
|
||||
|
||||
### Message Examples
|
||||
|
||||
#### Successful Start
|
||||
```json
|
||||
{
|
||||
"title": "🎬 Tdarr Monitoring Alert",
|
||||
"description": "Tdarr timeout monitoring started",
|
||||
"color": 3066993,
|
||||
"timestamp": "2025-08-10T14:19:55.000Z",
|
||||
"footer": {
|
||||
"text": "nobara-pc - Sun Aug 10 09:19:55 AM CDT 2025"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
#### Timeout Alert
|
||||
```json
|
||||
{
|
||||
"title": "🎬 Tdarr Monitoring Alert",
|
||||
"description": "⚠️ **4 file(s) timed out in staging:**\n```\nTV/Survivor/Season 48/...\n```\nFiles were removed from staging and will retry.",
|
||||
"color": 15158332,
|
||||
"timestamp": "2025-08-10T14:49:00.000Z"
|
||||
}
|
||||
```
|
||||
|
||||
## Performance Impact
|
||||
|
||||
### System Resources
|
||||
- **CPU Usage**: Minimal - quick log parsing every 15 minutes
|
||||
- **Network Impact**: One SSH connection + Docker logs query per check
|
||||
- **Storage**: < 1MB in `/tmp/tdarr-monitor/`
|
||||
- **Execution Time**: ~2-5 seconds per check
|
||||
|
||||
### Benefits
|
||||
- **Proactive Issue Detection**: Know about problems before manual checking
|
||||
- **Historical Tracking**: Discord provides persistent log of all alerts
|
||||
- **Mobile Notifications**: Get alerts on phone via Discord app
|
||||
- **Reduced Manual Monitoring**: Automated awareness of system health
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Common Issues
|
||||
|
||||
#### No Discord Messages
|
||||
```bash
|
||||
# Test webhook manually
|
||||
curl -H "Content-Type: application/json" -X POST \
|
||||
-d '{"content":"Manual test message"}' \
|
||||
"https://discord.com/api/webhooks/1404105821549498398/y2Ud1RK9rzFjv58xbypUfQNe3jrL7ZUq1FkQHa4_dfOHm2ylp93z0f4tY0O8Z-vQgKhD"
|
||||
|
||||
# Check script execution
|
||||
/mnt/NV2/Development/claude-home/scripts/monitoring/tdarr-timeout-monitor.sh
|
||||
```
|
||||
|
||||
#### SSH Connection Issues
|
||||
```bash
|
||||
# Test server connection
|
||||
ssh tdarr "echo 'SSH working'"
|
||||
|
||||
# Check SSH key authentication
|
||||
ssh-add -l
|
||||
```
|
||||
|
||||
#### Podman Container Access
|
||||
```bash
|
||||
# Verify container is running
|
||||
podman ps | grep tdarr-node
|
||||
|
||||
# Test log access
|
||||
podman logs --tail 5 tdarr-node-gpu-unmapped
|
||||
```
|
||||
|
||||
#### Force Monitoring Check
|
||||
```bash
|
||||
# Reset timestamp to trigger immediate check
|
||||
echo $(($(date +%s) - 1000)) > /tmp/tdarr-monitor/last_check.timestamp
|
||||
|
||||
# Run manual check
|
||||
/mnt/NV2/Development/claude-home/scripts/monitoring/tdarr-timeout-monitor.sh
|
||||
```
|
||||
|
||||
## Security Considerations
|
||||
|
||||
### Webhook Protection
|
||||
- **URL Security**: Webhook URL contains sensitive token
|
||||
- **Access Control**: Only monitoring script has access to webhook
|
||||
- **Scope Limitation**: Webhook only has permission to post messages
|
||||
|
||||
### SSH Access
|
||||
- **Key-based Authentication**: No password authentication used
|
||||
- **Limited Commands**: Only Docker logs and basic system commands
|
||||
- **Network Isolation**: SSH connection within trusted homelab network
|
||||
|
||||
### File Permissions
|
||||
```bash
|
||||
# Verify script permissions
|
||||
ls -la /mnt/NV2/Development/claude-home/scripts/monitoring/tdarr-timeout-monitor.sh
|
||||
# Should be: -rwxr--r-- (755) or more restrictive
|
||||
|
||||
# Secure if needed
|
||||
chmod 750 /mnt/NV2/Development/claude-home/scripts/monitoring/tdarr-timeout-monitor.sh
|
||||
```
|
||||
|
||||
## Maintenance
|
||||
|
||||
### Regular Checks
|
||||
- **Weekly**: Verify Discord messages are still being received
|
||||
- **Monthly**: Check `/tmp/tdarr-monitor/` directory size
|
||||
- **Quarterly**: Review alert frequency and adjust thresholds if needed
|
||||
|
||||
### Log Rotation
|
||||
```bash
|
||||
# Clean old monitoring logs if they accumulate
|
||||
find /tmp/tdarr-monitor/ -name "*.log" -mtime +7 -delete
|
||||
```
|
||||
|
||||
### Updates
|
||||
- **Script Updates**: Version control via git in homelab documentation
|
||||
- **Webhook Rotation**: Update webhook URL if Discord server changes
|
||||
- **Threshold Tuning**: Adjust 15-minute interval based on operational experience
|
||||
|
||||
## Integration with Other Systems
|
||||
|
||||
### Future Enhancements
|
||||
- **Grafana Integration**: Add metrics collection for dashboard visualization
|
||||
- **Prometheus Metrics**: Export timing and error rate metrics
|
||||
- **Home Assistant**: Integrate with home automation for additional alerting
|
||||
- **Email Backup**: Secondary notification method for critical alerts
|
||||
|
||||
### Related Documentation
|
||||
- [NAS Mount Configuration](../networking/nas-mount-configuration.md) - SMB optimization context
|
||||
- [Tdarr Troubleshooting](tdarr-troubleshooting.md) - Worker timeout background
|
||||
- [SSH Key Management](../networking/ssh-key-management.md) - Server access setup
|
||||
|
||||
---
|
||||
**Status**: ✅ Active and Configured
|
||||
**Last Updated**: August 10, 2025
|
||||
**Next Review**: September 10, 2025
|
||||
**Discord Channel**: Homelab monitoring alerts configured and tested
|
||||
205
reference/networking/nas-mount-configuration.md
Normal file
205
reference/networking/nas-mount-configuration.md
Normal file
@ -0,0 +1,205 @@
|
||||
# NAS Mount Configuration - TrueNAS SMB Optimization
|
||||
|
||||
## Overview
|
||||
This document contains the optimized SMB mount configurations for TrueNAS (10.10.0.35) across multiple systems in the homelab. These optimizations provide significant performance improvements over default SMB settings.
|
||||
|
||||
## TrueNAS System Details
|
||||
- **IP Address**: 10.10.0.35
|
||||
- **System**: TrueNAS Server
|
||||
- **SMB Version**: 3.1.1 (optimized from 3.0)
|
||||
- **Available Shares**: `/media` and `/cals-files`
|
||||
|
||||
## Performance Results
|
||||
|
||||
### Before Optimization
|
||||
- **Tdarr Server**: 61.8 MB/s
|
||||
- **Local Workstation**: 11.1 MB/s
|
||||
|
||||
### After Optimization
|
||||
- **Tdarr Server**: 103 MB/s (67% improvement)
|
||||
- **Local Workstation**: 85.4 MB/s (669% improvement)
|
||||
|
||||
## Optimized Mount Configurations
|
||||
|
||||
### Tdarr Server (ubuntu-ct - 10.10.0.43)
|
||||
|
||||
**File**: `/etc/fstab`
|
||||
```bash
|
||||
//10.10.0.35/media /mnt/truenas-share cifs vers=3.1.1,cache=loose,credentials=/root/.truenascreds,rsize=16777216,wsize=16777216,bsize=4194304,actimeo=30,closetimeo=5,echo_interval=30
|
||||
```
|
||||
|
||||
**Active Mount Options** (negotiated):
|
||||
```
|
||||
vers=3.1.1,cache=loose,rsize=8388608,wsize=8388608,bsize=4194304,
|
||||
actimeo=30,closetimeo=5,echo_interval=30
|
||||
```
|
||||
|
||||
### Local Workstation (nobara-pc)
|
||||
|
||||
**File**: `/etc/fstab`
|
||||
```bash
|
||||
//10.10.0.35/media /mnt/media cifs credentials=/home/cal/.samba_credentials,uid=1000,gid=1000,vers=3.1.1,cache=loose,rsize=16777216,wsize=16777216,bsize=4194304,actimeo=30,closetimeo=5,echo_interval=30,noperm 0 0
|
||||
//10.10.0.35/cals-files /mnt/cals-files cifs credentials=/home/cal/.samba_credentials,uid=1000,gid=1000,vers=3.1.1,cache=loose,rsize=16777216,wsize=16777216,bsize=4194304,actimeo=30,closetimeo=5,echo_interval=30,noperm 0 0
|
||||
```
|
||||
|
||||
**Active Mount Options** (negotiated):
|
||||
```
|
||||
vers=3.1.1,cache=loose,rsize=8388608,wsize=8388608,bsize=4194304,
|
||||
actimeo=30,closetimeo=5,echo_interval=30,noperm
|
||||
```
|
||||
|
||||
## Key Optimization Parameters
|
||||
|
||||
### Protocol Version
|
||||
- **Setting**: `vers=3.1.1`
|
||||
- **Previous**: `vers=3.0`
|
||||
- **Benefit**: Latest SMB protocol with performance improvements and better security
|
||||
|
||||
### Cache Strategy
|
||||
- **Setting**: `cache=loose`
|
||||
- **Previous**: `cache=strict`
|
||||
- **Benefit**: Better read performance, less strict consistency requirements
|
||||
|
||||
### Buffer Sizes
|
||||
- **Read Size**: `rsize=16777216` (requested) → `rsize=8388608` (negotiated)
|
||||
- **Write Size**: `wsize=16777216` (requested) → `wsize=8388608` (negotiated)
|
||||
- **Block Size**: `bsize=4194304` (4MB, up from 1MB)
|
||||
- **Previous**: `rsize=4194304,wsize=4194304,bsize=1048576`
|
||||
- **Benefit**: Larger buffers reduce network round trips
|
||||
|
||||
### Connection Timeouts
|
||||
- **Attribute Cache**: `actimeo=30` (30 seconds, up from 1 second)
|
||||
- **Close Timeout**: `closetimeo=5` (5 seconds, up from 1 second)
|
||||
- **Echo Interval**: `echo_interval=30` (30 seconds, down from 60 seconds)
|
||||
- **Benefit**: Better connection persistence, fewer reconnections
|
||||
|
||||
## Credential Files
|
||||
|
||||
### Tdarr Server
|
||||
**File**: `/root/.truenascreds`
|
||||
```
|
||||
username=plex
|
||||
password=[password]
|
||||
domain=
|
||||
```
|
||||
|
||||
### Local Workstation
|
||||
**File**: `/home/cal/.samba_credentials`
|
||||
```
|
||||
username=plex
|
||||
password=[password]
|
||||
domain=
|
||||
```
|
||||
|
||||
**Security**: Both files should be owned by root with 600 permissions:
|
||||
```bash
|
||||
sudo chown root:root /path/to/credentials
|
||||
sudo chmod 600 /path/to/credentials
|
||||
```
|
||||
|
||||
## Mount Commands
|
||||
|
||||
### Apply New Configuration
|
||||
```bash
|
||||
# Unmount existing mounts
|
||||
sudo umount /mnt/media
|
||||
sudo umount /mnt/cals-files
|
||||
|
||||
# Mount with new optimized settings
|
||||
sudo mount /mnt/media
|
||||
sudo mount /mnt/cals-files
|
||||
```
|
||||
|
||||
### Verify Mount Options
|
||||
```bash
|
||||
mount | grep -E 'media|cals-files'
|
||||
```
|
||||
|
||||
### Test Performance
|
||||
```bash
|
||||
# Test read performance (100MB test)
|
||||
time dd if="/mnt/media/path/to/large/file.mkv" bs=1M count=100 of=/dev/null
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Common Issues
|
||||
|
||||
**Mount Error: Device or resource busy**
|
||||
- Solution: Unmount existing mount first: `sudo umount /mnt/media`
|
||||
|
||||
**Parse Error in fstab**
|
||||
- Check for missing `//` before IP address
|
||||
- Ensure no spaces in comma-separated options
|
||||
- Verify all commas are present between options
|
||||
|
||||
**Permission Denied**
|
||||
- Verify credential file exists and has correct permissions (600)
|
||||
- Check username/password in credential file
|
||||
- Ensure TrueNAS user has access to the share
|
||||
|
||||
**Slow Performance After Changes**
|
||||
- Verify new mount options are active: `mount | grep media`
|
||||
- Test with different file sizes to confirm improvement
|
||||
- Check network connectivity: `ping 10.10.0.35`
|
||||
|
||||
### Performance Testing Commands
|
||||
```bash
|
||||
# Quick performance test (50MB)
|
||||
time dd if="/mnt/media/Movies/[movie-file]" bs=1M count=50 of=/dev/null
|
||||
|
||||
# Larger performance test (100MB)
|
||||
time dd if="/mnt/media/Movies/[movie-file]" bs=1M count=100 of=/dev/null
|
||||
|
||||
# Monitor network during transfer
|
||||
iftop -i eth0
|
||||
```
|
||||
|
||||
## Backup and Rollback
|
||||
|
||||
### Before Making Changes
|
||||
```bash
|
||||
# Backup fstab
|
||||
sudo cp /etc/fstab /etc/fstab.backup-$(date +%Y%m%d-%H%M%S)
|
||||
```
|
||||
|
||||
### Rollback if Needed
|
||||
```bash
|
||||
# Restore from backup
|
||||
sudo cp /etc/fstab.backup-YYYYMMDD-HHMMSS /etc/fstab
|
||||
|
||||
# Remount
|
||||
sudo umount /mnt/media
|
||||
sudo mount /mnt/media
|
||||
```
|
||||
|
||||
## System-Specific Notes
|
||||
|
||||
### Tdarr Integration
|
||||
- **Unmapped Node Architecture**: Node downloads files from Tdarr server at optimized 103 MB/s
|
||||
- **Cache Directory**: Uses local NVMe storage for maximum transcoding performance
|
||||
- **No Direct NAS Access**: Unmapped nodes don't directly access NAS mounts
|
||||
|
||||
### Local Workstation Usage
|
||||
- **Media Browsing**: 85.4 MB/s for fast local media access
|
||||
- **File Management**: Significantly faster file operations
|
||||
- **Backup Operations**: Improved speeds for large file transfers
|
||||
|
||||
## Future Expansion
|
||||
|
||||
When adding new systems, use these optimized settings as the baseline:
|
||||
|
||||
```bash
|
||||
//10.10.0.35/media /mnt/media cifs credentials=/path/to/creds,uid=1000,gid=1000,vers=3.1.1,cache=loose,rsize=16777216,wsize=16777216,bsize=4194304,actimeo=30,closetimeo=5,echo_interval=30,noperm 0 0
|
||||
```
|
||||
|
||||
Adjust `uid`, `gid`, and credential path as needed for each system.
|
||||
|
||||
## Related Documentation
|
||||
- [SSH Key Management](ssh-key-management.md) - For secure access to systems
|
||||
- [Tdarr Troubleshooting](../docker/tdarr-troubleshooting.md) - For Tdarr-specific issues
|
||||
- [Network Troubleshooting](ssh-troubleshooting.md) - For general network issues
|
||||
|
||||
---
|
||||
*Last updated: August 10, 2025*
|
||||
*Performance improvements: Tdarr Server 67% faster, Local Workstation 669% faster*
|
||||
191
scripts/monitoring/setup-discord-monitoring.md
Normal file
191
scripts/monitoring/setup-discord-monitoring.md
Normal file
@ -0,0 +1,191 @@
|
||||
# Tdarr Discord Monitoring Setup Guide
|
||||
|
||||
## Overview
|
||||
This guide sets up automated Discord notifications for Tdarr worker timeouts, stalls, and completions using a custom log monitoring script.
|
||||
|
||||
## Prerequisites
|
||||
- Discord server where you want notifications
|
||||
- Administrative access to create webhooks
|
||||
- Tdarr server accessible via SSH
|
||||
- Podman/Docker access to Tdarr node
|
||||
|
||||
## Setup Steps
|
||||
|
||||
### 1. Create Discord Webhook
|
||||
1. Go to your Discord server → **Server Settings** → **Integrations** → **Webhooks**
|
||||
2. Click **Create Webhook**
|
||||
3. Name it "Tdarr Monitor" and select the channel for notifications
|
||||
4. Copy the **Webhook URL** (keep this secure!)
|
||||
|
||||
### 2. Configure Monitoring Script
|
||||
Edit the script to add your webhook:
|
||||
```bash
|
||||
nano /mnt/NV2/Development/claude-home/scripts/monitoring/tdarr-timeout-monitor.sh
|
||||
```
|
||||
|
||||
Update these lines:
|
||||
```bash
|
||||
DISCORD_WEBHOOK="https://discord.com/api/webhooks/YOUR_WEBHOOK_ID/YOUR_WEBHOOK_TOKEN"
|
||||
SERVER_HOST="tdarr" # Your SSH alias for Tdarr server
|
||||
NODE_CONTAINER="tdarr-node-gpu-unmapped" # Your node container name
|
||||
```
|
||||
|
||||
### 3. Make Script Executable
|
||||
```bash
|
||||
chmod +x /mnt/NV2/Development/claude-home/scripts/monitoring/tdarr-timeout-monitor.sh
|
||||
```
|
||||
|
||||
### 4. Test the Script
|
||||
```bash
|
||||
/mnt/NV2/Development/claude-home/scripts/monitoring/tdarr-timeout-monitor.sh
|
||||
```
|
||||
|
||||
You should see a "monitoring started" message in your Discord channel.
|
||||
|
||||
### 5. Setup Automated Monitoring (Choose One)
|
||||
|
||||
#### Option A: Cron Job (Simple)
|
||||
```bash
|
||||
# Edit crontab
|
||||
crontab -e
|
||||
|
||||
# Add this line to check every 5 minutes
|
||||
*/5 * * * * /mnt/NV2/Development/claude-home/scripts/monitoring/tdarr-timeout-monitor.sh >/dev/null 2>&1
|
||||
```
|
||||
|
||||
#### Option B: Systemd Service (Advanced)
|
||||
Create a systemd service for more reliable monitoring:
|
||||
|
||||
```bash
|
||||
sudo nano /etc/systemd/system/tdarr-monitor.service
|
||||
```
|
||||
|
||||
Content:
|
||||
```ini
|
||||
[Unit]
|
||||
Description=Tdarr Timeout Monitor
|
||||
After=network.target
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
User=cal
|
||||
ExecStart=/mnt/NV2/Development/claude-home/scripts/monitoring/tdarr-timeout-monitor.sh
|
||||
```
|
||||
|
||||
Create timer:
|
||||
```bash
|
||||
sudo nano /etc/systemd/system/tdarr-monitor.timer
|
||||
```
|
||||
|
||||
Content:
|
||||
```ini
|
||||
[Unit]
|
||||
Description=Run Tdarr Monitor every 5 minutes
|
||||
Requires=tdarr-monitor.service
|
||||
|
||||
[Timer]
|
||||
OnCalendar=*:0/5
|
||||
Persistent=true
|
||||
|
||||
[Install]
|
||||
WantedBy=timers.target
|
||||
```
|
||||
|
||||
Enable and start:
|
||||
```bash
|
||||
sudo systemctl daemon-reload
|
||||
sudo systemctl enable tdarr-monitor.timer
|
||||
sudo systemctl start tdarr-monitor.timer
|
||||
```
|
||||
|
||||
## Notification Examples
|
||||
|
||||
### Worker Timeout Alert
|
||||
```
|
||||
🎬 Tdarr Monitoring Alert
|
||||
⚠️ 4 file(s) timed out in staging:
|
||||
```
|
||||
TV/Survivor/Season 48/Survivor (2000) - S48E04...
|
||||
TV/Survivor/Season 48/Survivor (2000) - S48E11...
|
||||
TV/Survivor/Season 26/Survivor (2000) - S26E05...
|
||||
```
|
||||
Files were removed from staging and will retry.
|
||||
```
|
||||
|
||||
### Worker Stall Alert
|
||||
```
|
||||
🎬 Tdarr Monitoring Alert
|
||||
🔴 2 worker stall(s) detected:
|
||||
```
|
||||
Worker eager-eyas
|
||||
Worker oblong-owl
|
||||
```
|
||||
Workers were cancelled and will restart.
|
||||
```
|
||||
|
||||
### Success Notification (Optional)
|
||||
```
|
||||
🎬 Tdarr Monitoring Alert
|
||||
✅ 3 transcode(s) completed successfully in the last check period.
|
||||
```
|
||||
|
||||
## Monitoring Features
|
||||
|
||||
✅ **Server Limbo Timeouts** - Files stuck in staging > timeout period
|
||||
✅ **Node Worker Stalls** - Workers that hang during transcoding
|
||||
✅ **Success Notifications** - Optional completion alerts
|
||||
✅ **Smart Timing** - Only checks every 60+ seconds to avoid spam
|
||||
✅ **Rich Discord Embeds** - Color-coded messages with timestamps
|
||||
|
||||
## Customization Options
|
||||
|
||||
### Disable Success Messages
|
||||
Edit the script and comment out this line:
|
||||
```bash
|
||||
# check_completions # Comment out to disable success notifications
|
||||
```
|
||||
|
||||
### Change Check Frequency
|
||||
For cron job, modify the timing:
|
||||
```bash
|
||||
*/10 * * * * # Check every 10 minutes instead of 5
|
||||
```
|
||||
|
||||
For systemd timer, update OnCalendar:
|
||||
```ini
|
||||
OnCalendar=*:0/10 # Check every 10 minutes
|
||||
```
|
||||
|
||||
### Add More Monitoring
|
||||
You can extend the script to monitor:
|
||||
- Disk space on cache directory
|
||||
- Network connectivity to TrueNAS
|
||||
- GPU utilization during transcoding
|
||||
- Queue depth and processing rates
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### No Notifications Received
|
||||
1. Check webhook URL is correct and accessible
|
||||
2. Test webhook manually:
|
||||
```bash
|
||||
curl -H "Content-Type: application/json" -X POST -d '{"content":"Test message"}' "YOUR_WEBHOOK_URL"
|
||||
```
|
||||
3. Check script logs: `/tmp/tdarr-monitor/`
|
||||
|
||||
### False Positives
|
||||
- Adjust the timing logic in the script
|
||||
- Filter out specific log patterns that aren't actual errors
|
||||
- Tune the timeout thresholds
|
||||
|
||||
### Missing SSH Access
|
||||
- Ensure SSH key authentication is set up for the `tdarr` server
|
||||
- Test: `ssh tdarr "echo 'SSH working'"`
|
||||
|
||||
## Security Notes
|
||||
- Keep your Discord webhook URL private
|
||||
- Consider using environment variables for sensitive data
|
||||
- Restrict file permissions on the script (chmod 750)
|
||||
|
||||
---
|
||||
*This monitoring solution provides real-time alerts for Tdarr issues without requiring external monitoring infrastructure.*
|
||||
Loading…
Reference in New Issue
Block a user