Add docker scripts, media-tools, VM management, and n8n workflow docs

Add CONTEXT.md for docker and VM management script directories.
Add media-tools documentation with Playwright scraping patterns.
Add Tdarr GPU monitor n8n workflow definition.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
Cal Corum 2026-02-07 22:26:10 -06:00
parent b186107b97
commit ceb4dd36a0
7 changed files with 1818 additions and 0 deletions

92
docker/scripts/CONTEXT.md Normal file
View File

@ -0,0 +1,92 @@
# Docker Scripts - Operational Context
## Script Overview
This directory is reserved for active operational scripts for Docker container management, orchestration, and automation.
**Current Status**: No operational scripts currently deployed. This structure is maintained for future Docker automation needs.
## Future Script Categories
### Planned Script Types
**Container Lifecycle Management**
- Start/stop scripts for complex multi-container setups
- Health check and restart automation
- Graceful shutdown procedures for dependent containers
**Maintenance Automation**
- Image cleanup and pruning scripts
- Volume backup and restoration
- Container log rotation and archiving
- Network cleanup and validation
**Monitoring and Alerts**
- Container health monitoring
- Resource usage tracking
- Discord/webhook notifications for container events
- Uptime and availability reporting
**Deployment Automation**
- CI/CD integration scripts
- Rolling update procedures
- Blue-green deployment automation
- Container migration tools
## Integration Points
### External Dependencies
- **Docker/Podman**: Container runtime
- **Docker Compose**: Multi-container orchestration
- **cron**: System scheduler for automation
- **Discord Webhooks**: Notification integration (when implemented)
### File System Dependencies
- **Container Volumes**: Various locations depending on service
- **Configuration Files**: Service-specific docker-compose.yml files
- **Log Files**: Container and automation logs
- **Backup Storage**: For volume snapshots and exports
### Network Dependencies
- **Docker Networks**: Bridge, host, and custom networks
- **External Services**: APIs and webhooks for monitoring
- **Registry Access**: For image pulls and pushes (when needed)
## Development Guidelines
### When Adding New Scripts
**Documentation Requirements**:
1. Add script description to this CONTEXT.md under appropriate category
2. Include usage examples and command-line options
3. Document dependencies and prerequisites
4. Specify cron schedule if automated
5. Add troubleshooting section for common issues
**Script Standards**:
```bash
#!/bin/bash
# Script name and purpose
# Dependencies: list required commands/services
# Usage: ./script.sh [options]
set -euo pipefail # Strict error handling
```
**Testing Requirements**:
- Test with both Docker and Podman where applicable
- Verify error handling and logging
- Document failure modes and recovery procedures
- Include dry-run or test mode where appropriate
## Related Documentation
- **Technology Overview**: `/docker/CONTEXT.md`
- **Troubleshooting**: `/docker/troubleshooting.md`
- **Examples**: `/docker/examples/` - Reference configurations and patterns
- **Main Instructions**: `/CLAUDE.md` - Context loading rules
## Notes
This directory structure is maintained to support future Docker automation needs while keeping operational scripts organized and documented according to the technology-first documentation pattern established in the claude-home repository.
When scripts are added, this file should be updated to include specific operational context similar to the comprehensive documentation found in `/tdarr/scripts/CONTEXT.md`.

82
media-tools/CONTEXT.md Normal file
View File

@ -0,0 +1,82 @@
# Media Tools
Tools for downloading and managing media from streaming sites.
## Overview
This directory contains utilities for:
- Extracting video URLs from streaming sites using browser automation
- Downloading videos via yt-dlp
- Managing download state for resumable operations
## Tools
### pokeflix_scraper.py
Downloads Pokemon episodes from pokeflix.tv using Playwright for browser automation.
**Location:** `scripts/pokeflix_scraper.py`
**Features:**
- Extracts episode lists from season pages
- Handles iframe-embedded video players (Streamtape, Vidoza, etc.)
- Resumable downloads with state persistence
- Configurable episode ranges
- Dry-run mode for testing
## Architecture Pattern
These tools follow a common pattern:
```
┌─────────────────┐ ┌──────────────────┐ ┌─────────────┐
│ Playwright │────▶│ Extract embed │────▶│ yt-dlp │
│ (navigate) │ │ video URLs │ │ (download) │
└─────────────────┘ └──────────────────┘ └─────────────┘
```
**Why this approach:**
1. **Playwright** handles JavaScript-heavy sites that block simple HTTP requests
2. **Iframe extraction** works around sites that use third-party video hosts
3. **yt-dlp** is the de-facto standard for video downloading with broad host support
## Dependencies
```bash
# Python packages
pip install playwright yt-dlp
# Playwright browser installation
playwright install chromium
```
## Common Patterns
### Anti-Bot Handling
- Use headed browser mode (visible window) initially
- Random delays between requests (2-5 seconds)
- Realistic viewport and user-agent settings
- Wait for `networkidle` state after navigation
### State Management
- JSON state files track downloaded episodes
- Enable `--resume` flag to skip completed downloads
- State includes error information for debugging
### Output Organization
```
{output_dir}/
├── {Season Name}/
│ ├── E01 - Episode Title.mp4
│ ├── E02 - Episode Title.mp4
│ └── download_state.json
```
## When to Use These Tools
- Downloading entire seasons of shows for offline viewing
- Archiving content before it becomes unavailable
- Building a local media library
## Legal Considerations
These tools are for personal archival use. Respect copyright laws in your jurisdiction.

View File

@ -0,0 +1,103 @@
# Media Tools Scripts
Operational scripts for media downloading and management.
## Scripts
### pokeflix_scraper.py
Downloads Pokemon episodes from pokeflix.tv.
**Dependencies:**
```bash
pip install playwright yt-dlp
playwright install chromium
```
**Quick Start:**
```bash
# Download entire season
python pokeflix_scraper.py \
--url "https://www.pokeflix.tv/browse/pokemon-indigo-league" \
--output ~/Pokemon/
# Download episodes 1-10 only
python pokeflix_scraper.py \
--url "https://www.pokeflix.tv/browse/pokemon-indigo-league" \
--output ~/Pokemon/ \
--start 1 --end 10
# Resume interrupted download
python pokeflix_scraper.py \
--url "https://www.pokeflix.tv/browse/pokemon-indigo-league" \
--output ~/Pokemon/ \
--resume
# Dry run (extract URLs, don't download)
python pokeflix_scraper.py \
--url "https://www.pokeflix.tv/browse/pokemon-indigo-league" \
--dry-run --verbose
```
**CLI Options:**
| Option | Description |
|--------|-------------|
| `--url, -u` | Season page URL (required) |
| `--output, -o` | Output directory (default: ~/Downloads/Pokemon) |
| `--start, -s` | First episode number to download |
| `--end, -e` | Last episode number to download |
| `--resume, -r` | Resume from previous state |
| `--dry-run, -n` | Extract URLs only, no download |
| `--headless` | Run browser without visible window |
| `--verbose, -v` | Enable debug logging |
**Output Structure:**
```
~/Pokemon/
├── Pokemon Indigo League/
│ ├── E01 - Pokemon I Choose You.mp4
│ ├── E02 - Pokemon Emergency.mp4
│ ├── E03 - Ash Catches a Pokemon.mp4
│ └── download_state.json
```
**State File:**
The `download_state.json` tracks progress:
```json
{
"season_url": "https://...",
"season_name": "Pokemon Indigo League",
"episodes": {
"1": {
"number": 1,
"title": "Pokemon I Choose You",
"page_url": "https://...",
"video_url": "https://...",
"downloaded": true,
"error": null
}
},
"last_updated": "2025-01-22T..."
}
```
## Adding New Scrapers
To add a scraper for a new site:
1. Copy the pattern from `pokeflix_scraper.py`
2. Modify the selectors for episode list extraction
3. Modify the iframe/video URL selectors for the new site's player
4. Test with `--dry-run` first
Key methods to customize:
- `get_season_info()` - Extract episode list from season page
- `extract_video_url()` - Get video URL from episode page
## Performance Notes
- **Non-headless mode** is recommended (default) to avoid anti-bot detection
- Random delays (2-5s) between requests prevent rate limiting
- Large seasons (80+ episodes) may take hours - use `--resume` if interrupted

View File

@ -0,0 +1,777 @@
#!/usr/bin/env python3
"""
Pokeflix Scraper - Download Pokemon episodes from pokeflix.tv
Pokeflix hosts videos directly on their CDN (v1.pkflx.com). This scraper:
1. Extracts the episode list from a season browse page
2. Visits each episode page to detect its CDN episode number
3. Downloads videos directly from the CDN via yt-dlp
Usage:
# Download entire season
python pokeflix_scraper.py --url "https://www.pokeflix.tv/browse/pokemon-indigo-league" --output ~/Pokemon/
# Download specific episode range
python pokeflix_scraper.py --url "..." --start 1 --end 10 --output ~/Pokemon/
# Resume interrupted download
python pokeflix_scraper.py --url "..." --output ~/Pokemon/ --resume
# Dry run (extract URLs only)
python pokeflix_scraper.py --url "..." --dry-run
# Choose quality
python pokeflix_scraper.py --url "..." --quality 720p --output ~/Pokemon/
Dependencies:
pip install playwright
playwright install chromium
# yt-dlp must be installed: pip install yt-dlp
Author: Cal Corum (with Jarvis assistance)
"""
import argparse
import asyncio
import json
import logging
import random
import re
import subprocess
import sys
from dataclasses import dataclass, field, asdict
from datetime import datetime
from pathlib import Path
from typing import Optional
try:
from playwright.async_api import async_playwright, Page, Browser
except ImportError:
print("ERROR: playwright not installed. Run: pip install playwright && playwright install chromium")
sys.exit(1)
# ============================================================================
# Data Classes
# ============================================================================
@dataclass
class Episode:
"""Represents a single episode with its metadata and download status."""
cdn_number: int # The actual episode number on the CDN
title: str
page_url: str
slug: str # URL slug e.g., "01-pokemon-i-choose-you"
video_url: Optional[str] = None
downloaded: bool = False
error: Optional[str] = None
@property
def filename(self) -> str:
"""Generate safe filename for the episode."""
safe_title = re.sub(r'[<>:"/\\|?*]', '', self.title)
safe_title = safe_title.strip()
return f"E{self.cdn_number:02d} - {safe_title}.mp4"
@dataclass
class Season:
"""Represents a season/series with all its episodes."""
name: str
url: str
cdn_slug: str # e.g., "01-indigo-league" - used for CDN URLs
episodes: list[Episode] = field(default_factory=list)
@property
def safe_name(self) -> str:
"""Generate safe directory name for the season."""
safe = re.sub(r'[<>:"/\\|?*]', '', self.name)
return safe.strip()
@dataclass
class DownloadState:
"""Persistent state for resumable downloads."""
season_url: str
season_name: str
cdn_slug: str
episodes: dict[int, dict] = field(default_factory=dict) # cdn_number -> episode dict
episode_urls: list[str] = field(default_factory=list) # All episode page URLs
last_updated: str = ""
def save(self, path: Path) -> None:
"""Save state to JSON file."""
self.last_updated = datetime.now().isoformat()
with open(path, 'w') as f:
json.dump(asdict(self), f, indent=2)
@classmethod
def load(cls, path: Path) -> Optional['DownloadState']:
"""Load state from JSON file."""
if not path.exists():
return None
with open(path) as f:
data = json.load(f)
return cls(**data)
# ============================================================================
# Logging Setup
# ============================================================================
def setup_logging(verbose: bool = False) -> logging.Logger:
"""Configure logging with console output."""
logger = logging.getLogger('pokeflix_scraper')
logger.setLevel(logging.DEBUG if verbose else logging.INFO)
if not logger.handlers:
console = logging.StreamHandler()
console.setLevel(logging.DEBUG if verbose else logging.INFO)
console.setFormatter(logging.Formatter(
'%(asctime)s [%(levelname)s] %(message)s',
datefmt='%H:%M:%S'
))
logger.addHandler(console)
return logger
# ============================================================================
# Scraper Class
# ============================================================================
class PokeflixScraper:
"""
Scrapes pokeflix.tv for video URLs and downloads them.
Pokeflix hosts videos on their CDN with URLs like:
https://v1.pkflx.com/hls/{season-slug}/{ep-num}/{ep-num}_{quality}.mp4
The episode number must be detected by visiting each episode page,
as the browse page URL slugs don't contain episode numbers.
"""
BASE_URL = "https://www.pokeflix.tv"
CDN_URL = "https://v1.pkflx.com/hls"
# Map browse page URL slugs to CDN slugs
SEASON_SLUG_MAP = {
'pokemon-indigo-league': '01-indigo-league',
'pokemon-adventures-in-the-orange-islands': '02-orange-islands',
'pokemon-the-johto-journeys': '03-johto-journeys',
'pokemon-johto-league-champions': '04-johto-league-champions',
'pokemon-master-quest': '05-master-quest',
'pokemon-advanced': '06-advanced',
'pokemon-advanced-challenge': '07-advanced-challenge',
'pokemon-advanced-battle': '08-advanced-battle',
'pokemon-battle-frontier': '09-battle-frontier',
'pokemon-diamond-and-pearl': '10-diamond-and-pearl',
'pokemon-dp-battle-dimension': '11-battle-dimension',
'pokemon-dp-galactic-battles': '12-galactic-battles',
'pokemon-dp-sinnoh-league-victors': '13-sinnoh-league-victors',
'pokemon-black-white': '14-black-and-white',
'pokemon-bw-rival-destinies': '15-rival-destinies',
'pokemon-bw-adventures-in-unova': '16-adventures-in-unova',
'pokemon-xy': '17-xy',
'pokemon-xy-kalos-quest': '18-kalos-quest',
'pokemon-xyz': '19-xyz',
'pokemon-sun-moon': '20-sun-and-moon',
'pokemon-sun-moon-ultra-adventures': '21-ultra-adventures',
'pokemon-sun-moon-ultra-legends': '22-ultra-legends',
'pokemon-journeys': '23-journeys',
'pokemon-master-journeys': '24-master-journeys',
'pokemon-ultimate-journeys': '25-ultimate-journeys',
'pokemon-horizons': '26-horizons',
}
def __init__(
self,
output_dir: Path,
headless: bool = False,
dry_run: bool = False,
verbose: bool = False,
quality: str = "1080p"
):
self.output_dir = output_dir
self.headless = headless
self.dry_run = dry_run
self.quality = quality
self.logger = setup_logging(verbose)
self.browser: Optional[Browser] = None
self._context = None
async def __aenter__(self):
"""Async context manager entry - launch browser."""
playwright = await async_playwright().start()
self.browser = await playwright.chromium.launch(
headless=self.headless,
args=['--disable-blink-features=AutomationControlled']
)
self._playwright = playwright
# Create a persistent context to reuse
self._context = await self.browser.new_context(
viewport={'width': 1920, 'height': 1080},
user_agent='Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36'
)
return self
async def __aexit__(self, exc_type, exc_val, exc_tb):
"""Async context manager exit - close browser."""
if self._context:
await self._context.close()
if self.browser:
await self.browser.close()
await self._playwright.stop()
async def _new_page(self) -> Page:
"""Create a new page using the shared context."""
return await self._context.new_page()
async def _random_delay(self, min_sec: float = 1.0, max_sec: float = 3.0):
"""Random delay to avoid detection."""
delay = random.uniform(min_sec, max_sec)
await asyncio.sleep(delay)
async def _wait_for_cloudflare(self, page: Page, timeout: int = 60):
"""Wait for Cloudflare challenge to be solved by user."""
try:
# Check if we're on a Cloudflare challenge page
is_cf = await page.query_selector('#challenge-running, .cf-browser-verification, [id*="challenge"]')
if is_cf:
self.logger.warning("Cloudflare challenge detected - please solve it in the browser window")
self.logger.info("Waiting up to 60 seconds for challenge completion...")
# Wait for the challenge to be solved (URL changes or challenge element disappears)
for _ in range(timeout):
await asyncio.sleep(1)
is_cf = await page.query_selector('#challenge-running, .cf-browser-verification, [id*="challenge"]')
if not is_cf:
self.logger.info("Cloudflare challenge completed!")
await asyncio.sleep(2) # Wait for page to fully load
return True
self.logger.error("Cloudflare challenge timeout - please try again")
return False
except Exception:
pass
return True
def _get_cdn_slug(self, browse_url: str) -> Optional[str]:
"""Extract CDN slug from browse page URL."""
match = re.search(r'/browse/([^/]+)', browse_url)
if match:
page_slug = match.group(1)
if page_slug in self.SEASON_SLUG_MAP:
return self.SEASON_SLUG_MAP[page_slug]
self.logger.warning(f"Unknown season slug: {page_slug}, will try to detect from page")
return None
def _construct_video_url(self, cdn_slug: str, ep_num: int) -> str:
"""Construct direct CDN video URL."""
return f"{self.CDN_URL}/{cdn_slug}/{ep_num:02d}/{ep_num:02d}_{self.quality}.mp4"
def _slug_to_title(self, slug: str) -> str:
"""Convert URL slug to human-readable title."""
# Remove season prefix like "01-"
title_slug = re.sub(r'^\d+-', '', slug)
# Convert to title case
title = title_slug.replace('-', ' ').title()
# Clean up common words
title = re.sub(r'\bPokemon\b', 'Pokémon', title)
return title
async def get_episode_list(self, season_url: str) -> tuple[str, str, list[tuple[str, str]]]:
"""
Get the list of episode URLs from a season browse page.
Returns:
Tuple of (season_name, cdn_slug, list of (page_url, slug) tuples)
"""
self.logger.info(f"Fetching season page: {season_url}")
cdn_slug = self._get_cdn_slug(season_url)
page = await self._new_page()
try:
await page.goto(season_url, wait_until='networkidle', timeout=60000)
await self._wait_for_cloudflare(page)
await self._random_delay(2, 4)
# Extract season title
title_elem = await page.query_selector('h1, .season-title, .series-title')
if not title_elem:
title_elem = await page.query_selector('title')
season_name = await title_elem.inner_text() if title_elem else "Unknown Season"
season_name = season_name.replace('Pokéflix - Watch ', '').replace(' for free online!', '').strip()
self.logger.info(f"Season: {season_name}")
# Find all episode links with /v/ pattern
links = await page.query_selector_all('a[href^="/v/"]')
self.logger.info(f"Found {len(links)} episode links")
# If we don't have CDN slug, detect it from first episode
if not cdn_slug and links:
first_href = await links[0].get_attribute('href')
cdn_slug = await self._detect_cdn_slug(first_href)
if not cdn_slug:
self.logger.error("Could not determine CDN slug for this season")
return season_name, "unknown", []
self.logger.info(f"CDN slug: {cdn_slug}")
# Collect all episode URLs
episode_data = []
seen_urls = set()
for link in links:
href = await link.get_attribute('href')
if not href or href in seen_urls:
continue
seen_urls.add(href)
# Extract slug from URL
slug_match = re.search(r'/v/(.+)', href)
if slug_match:
slug = slug_match.group(1)
full_url = self.BASE_URL + href
episode_data.append((full_url, slug))
return season_name, cdn_slug, episode_data
finally:
await page.close()
async def _detect_cdn_slug(self, episode_href: str) -> Optional[str]:
"""Visit an episode page to detect the CDN slug from network requests."""
self.logger.info("Detecting CDN slug from episode page...")
detected_slug = None
async def capture_request(request):
nonlocal detected_slug
if 'v1.pkflx.com/hls/' in request.url:
match = re.search(r'hls/([^/]+)/', request.url)
if match:
detected_slug = match.group(1)
page = await self._new_page()
page.on('request', capture_request)
try:
await page.goto(self.BASE_URL + episode_href, timeout=60000)
await self._wait_for_cloudflare(page)
await asyncio.sleep(5)
return detected_slug
finally:
await page.close()
async def get_episode_cdn_number(self, page_url: str) -> Optional[int]:
"""
Visit an episode page and detect its CDN episode number.
Returns:
The episode number used in CDN URLs, or None if not detected
"""
detected_num = None
async def capture_request(request):
nonlocal detected_num
if 'v1.pkflx.com/hls/' in request.url:
match = re.search(r'/(\d+)/\d+_', request.url)
if match:
detected_num = int(match.group(1))
page = await self._new_page()
page.on('request', capture_request)
try:
await page.goto(page_url, timeout=60000)
await self._wait_for_cloudflare(page)
# Wait for initial load
await asyncio.sleep(2)
# Try to trigger video playback by clicking play button or video area
play_selectors = [
'button[aria-label*="play" i]',
'.play-button',
'[class*="play"]',
'video',
'.video-player',
'.player',
'#player',
]
for selector in play_selectors:
try:
elem = await page.query_selector(selector)
if elem:
await elem.click()
await asyncio.sleep(0.5)
if detected_num:
break
except Exception:
pass
# Wait for video requests after click attempts
for _ in range(10): # Wait up to 5 seconds
if detected_num:
break
await asyncio.sleep(0.5)
# If still not detected, try looking in page source
if not detected_num:
content = await page.content()
match = re.search(r'v1\.pkflx\.com/hls/[^/]+/(\d+)/', content)
if match:
detected_num = int(match.group(1))
return detected_num
finally:
await page.close()
def download_video(self, video_url: str, output_path: Path) -> bool:
"""
Download video using yt-dlp.
Args:
video_url: Direct CDN URL to the video
output_path: Full path for output file
Returns:
True if download succeeded
"""
if self.dry_run:
self.logger.info(f" [DRY RUN] Would download: {video_url}")
self.logger.info(f" To: {output_path}")
return True
self.logger.info(f" Downloading: {output_path.name}")
cmd = [
'yt-dlp',
'--no-warnings',
'-o', str(output_path),
'--no-part',
video_url
]
try:
result = subprocess.run(
cmd,
capture_output=True,
text=True,
timeout=1800
)
if result.returncode == 0:
self.logger.info(f" Download complete!")
return True
else:
self.logger.error(f" yt-dlp error: {result.stderr}")
return False
except subprocess.TimeoutExpired:
self.logger.error(" Download timed out after 30 minutes")
return False
except FileNotFoundError:
self.logger.error(" yt-dlp not found. Install with: pip install yt-dlp")
return False
async def download_direct(
self,
season_url: str,
start_ep: int,
end_ep: int,
resume: bool = False
) -> None:
"""
Direct download mode - download episodes by number without visiting pages.
This is faster and more reliable when you know the episode range.
"""
# Get CDN slug from URL
cdn_slug = self._get_cdn_slug(season_url)
if not cdn_slug:
self.logger.error("Unknown season - direct mode requires a known season URL")
self.logger.info("Known seasons: " + ", ".join(self.SEASON_SLUG_MAP.keys()))
return
# In direct mode, output_dir is the final destination (no subfolder created)
season_dir = self.output_dir
season_dir.mkdir(parents=True, exist_ok=True)
self.logger.info(f"Direct download mode: Episodes {start_ep}-{end_ep}")
self.logger.info(f"CDN slug: {cdn_slug}")
self.logger.info(f"Quality: {self.quality}")
self.logger.info(f"Output: {season_dir}")
downloaded = 0
skipped = 0
failed = 0
for ep_num in range(start_ep, end_ep + 1):
output_path = season_dir / f"E{ep_num:02d}.mp4"
# Check if already exists
if output_path.exists() and resume:
self.logger.info(f"E{ep_num:02d}: Skipping (file exists)")
skipped += 1
continue
video_url = self._construct_video_url(cdn_slug, ep_num)
self.logger.info(f"E{ep_num:02d}: Downloading...")
success = self.download_video(video_url, output_path)
if success:
downloaded += 1
else:
failed += 1
if not self.dry_run:
await self._random_delay(0.5, 1.5)
self.logger.info(f"\nComplete! Downloaded: {downloaded}, Skipped: {skipped}, Failed: {failed}")
async def download_season(
self,
season_url: str,
start_ep: Optional[int] = None,
end_ep: Optional[int] = None,
resume: bool = False,
direct: bool = False
) -> None:
"""
Download all episodes from a season.
Args:
season_url: URL to the season browse page
start_ep: First episode number to download (inclusive)
end_ep: Last episode number to download (inclusive)
resume: Whether to resume from previous state
direct: If True, skip page visits and download by episode number
"""
# Direct mode - just download by episode number
if direct:
if start_ep is None or end_ep is None:
self.logger.error("Direct mode requires --start and --end episode numbers")
return
await self.download_direct(season_url, start_ep, end_ep, resume)
return
# Get episode list
season_name, cdn_slug, episode_data = await self.get_episode_list(season_url)
if not episode_data:
self.logger.error("No episodes found!")
return
# Create output directory
season_dir = self.output_dir / re.sub(r'[<>:"/\\|?*]', '', season_name).strip()
season_dir.mkdir(parents=True, exist_ok=True)
# State file for resume
state_path = season_dir / 'download_state.json'
state = None
if resume:
state = DownloadState.load(state_path)
if state:
self.logger.info(f"Resuming from previous state ({state.last_updated})")
if not state:
state = DownloadState(
season_url=season_url,
season_name=season_name,
cdn_slug=cdn_slug,
episode_urls=[url for url, _ in episode_data]
)
self.logger.info(f"Processing {len(episode_data)} episodes (quality: {self.quality})")
# Process each episode
downloaded_count = 0
skipped_count = 0
failed_count = 0
for page_url, slug in episode_data:
title = self._slug_to_title(slug)
# Check if we already have this episode in state (by URL)
existing_ep = None
for ep_data in state.episodes.values():
if ep_data.get('page_url') == page_url:
existing_ep = ep_data
break
if existing_ep and existing_ep.get('downloaded') and resume:
self.logger.info(f"Skipping: {title} (already downloaded)")
skipped_count += 1
continue
# Get the CDN episode number by visiting the page
self.logger.info(f"Checking: {title}")
cdn_num = await self.get_episode_cdn_number(page_url)
if cdn_num is None:
self.logger.error(f" Could not detect episode number, skipping")
failed_count += 1
continue
# Check if within requested range
if start_ep is not None and cdn_num < start_ep:
self.logger.info(f" Episode {cdn_num} before start range, skipping")
continue
if end_ep is not None and cdn_num > end_ep:
self.logger.info(f" Episode {cdn_num} after end range, skipping")
continue
# Check if file already exists
output_path = season_dir / f"E{cdn_num:02d} - {title}.mp4"
if output_path.exists() and resume:
self.logger.info(f" File exists, skipping")
state.episodes[str(cdn_num)] = {
'cdn_number': cdn_num,
'title': title,
'page_url': page_url,
'slug': slug,
'video_url': self._construct_video_url(cdn_slug, cdn_num),
'downloaded': True,
'error': None
}
state.save(state_path)
skipped_count += 1
continue
# Construct video URL and download
video_url = self._construct_video_url(cdn_slug, cdn_num)
self.logger.info(f" Episode {cdn_num}: {title}")
success = self.download_video(video_url, output_path)
# Save state
state.episodes[str(cdn_num)] = {
'cdn_number': cdn_num,
'title': title,
'page_url': page_url,
'slug': slug,
'video_url': video_url,
'downloaded': success,
'error': None if success else "Download failed"
}
state.save(state_path)
if success:
downloaded_count += 1
else:
failed_count += 1
# Delay between episodes
if not self.dry_run:
await self._random_delay(1, 2)
# Summary
self.logger.info(f"\nComplete!")
self.logger.info(f" Downloaded: {downloaded_count}")
self.logger.info(f" Skipped: {skipped_count}")
self.logger.info(f" Failed: {failed_count}")
self.logger.info(f" Output: {season_dir}")
# ============================================================================
# CLI
# ============================================================================
def main():
parser = argparse.ArgumentParser(
description='Download Pokemon episodes from pokeflix.tv',
formatter_class=argparse.RawDescriptionHelpFormatter,
epilog="""
Examples:
%(prog)s --url "https://www.pokeflix.tv/browse/pokemon-indigo-league" --output ~/Pokemon/
%(prog)s --url "..." --start 1 --end 10 --output ~/Pokemon/
%(prog)s --url "..." --output ~/Pokemon/ --resume
%(prog)s --url "..." --quality 720p --output ~/Pokemon/
%(prog)s --url "..." --dry-run
"""
)
parser.add_argument(
'--url', '-u',
required=True,
help='URL of the season/series page'
)
parser.add_argument(
'--output', '-o',
type=Path,
default=Path.home() / 'Downloads' / 'Pokemon',
help='Output directory (default: ~/Downloads/Pokemon)'
)
parser.add_argument(
'--start', '-s',
type=int,
help='Start episode number (CDN number)'
)
parser.add_argument(
'--end', '-e',
type=int,
help='End episode number (CDN number)'
)
parser.add_argument(
'--quality', '-q',
choices=['1080p', '720p', '360p'],
default='1080p',
help='Video quality (default: 1080p)'
)
parser.add_argument(
'--resume', '-r',
action='store_true',
help='Resume from previous download state'
)
parser.add_argument(
'--dry-run', '-n',
action='store_true',
help='Extract URLs only, do not download'
)
parser.add_argument(
'--headless',
action='store_true',
help='Run browser in headless mode (may trigger anti-bot)'
)
parser.add_argument(
'--verbose', '-v',
action='store_true',
help='Enable verbose logging'
)
parser.add_argument(
'--direct',
action='store_true',
help='Direct download mode - skip page visits, just download episode range by number'
)
args = parser.parse_args()
async def run():
async with PokeflixScraper(
output_dir=args.output,
headless=args.headless,
dry_run=args.dry_run,
verbose=args.verbose,
quality=args.quality
) as scraper:
await scraper.download_season(
season_url=args.url,
start_ep=args.start,
end_ep=args.end,
resume=args.resume,
direct=args.direct
)
asyncio.run(run())
if __name__ == '__main__':
main()

View File

@ -0,0 +1,195 @@
# Media Tools Troubleshooting
## Common Issues
### Playwright Issues
#### "playwright not installed" Error
```
ERROR: playwright not installed. Run: pip install playwright && playwright install chromium
```
**Solution:**
```bash
pip install playwright
playwright install chromium
```
#### Browser Launch Fails
```
Error: Executable doesn't exist at /home/user/.cache/ms-playwright/chromium-xxx/chrome-linux/chrome
```
**Solution:**
```bash
playwright install chromium
```
#### Timeout Errors
```
TimeoutError: Timeout 30000ms exceeded
```
**Causes:**
- Slow network connection
- Site is blocking automated access
- Page structure has changed
**Solutions:**
1. Increase timeout in script
2. Try without `--headless` flag
3. Check if site is up manually
---
### yt-dlp Issues
#### "yt-dlp not found" Error
```
yt-dlp not found. Install with: pip install yt-dlp
```
**Solution:**
```bash
pip install yt-dlp
```
#### Download Fails for Specific Host
```
ERROR: Unsupported URL: https://somehost.com/...
```
**Solution:**
```bash
# Update yt-dlp to latest version
pip install -U yt-dlp
```
If still failing, the host may be unsupported. Check [yt-dlp supported sites](https://github.com/yt-dlp/yt-dlp/blob/master/supportedsites.md).
#### Slow Downloads
**Causes:**
- Video host throttling
- Network issues
**Solutions:**
- Downloads are typically limited by the source server
- Try at different times of day
---
### Scraping Issues
#### No Episodes Found
```
No episodes found!
```
**Causes:**
- Site structure has changed
- Page requires authentication
- Cloudflare protection triggered
**Solutions:**
1. Run without `--headless` to see what's happening
2. Check if the URL is correct and accessible manually
3. Site may have updated their HTML structure - check selectors in script
#### Video URL Not Found
```
No video URL found for episode X
```
**Causes:**
- Video is on an unsupported host
- Page uses non-standard embedding method
- Anti-bot protection on video player
**Solutions:**
1. Run with `--verbose` to see what URLs are being tried
2. Open episode manually and check Network tab for video requests
3. May need to add new iframe selectors for the specific host
#### 403 Forbidden on Site
**Cause:** Site is blocking automated requests
**Solutions:**
1. Ensure you're NOT using `--headless`
2. Increase random delays
3. Clear browser cache/cookies (restart script)
4. Try from a different IP
---
### Resume Issues
#### Resume Not Working
```
# Should skip downloaded episodes but re-downloads them
```
**Check:**
1. Ensure `download_state.json` exists in output directory
2. Verify the `--resume` flag is being used
3. Check that episode numbers match between runs
#### Corrupt State File
```
JSONDecodeError: ...
```
**Solution:**
```bash
# Delete the state file to start fresh
rm /path/to/season/download_state.json
```
---
## Debug Mode
Run with verbose output:
```bash
python pokeflix_scraper.py --url "..." --output ~/Pokemon/ --verbose
```
Run dry-run to test URL extraction:
```bash
python pokeflix_scraper.py --url "..." --dry-run --verbose
```
Watch the browser (non-headless):
```bash
python pokeflix_scraper.py --url "..." --output ~/Pokemon/
# (headless is off by default)
```
---
## Manual Workarounds
### If Automated Extraction Fails
1. **Browser DevTools method:**
- Open episode in browser
- F12 → Network tab → filter "m3u8" or "mp4"
- Play video, copy the stream URL
- Download manually: `yt-dlp "URL"`
2. **Check iframe manually:**
- Right-click video player → Inspect
- Find `<iframe>` element
- Copy `src` attribute
- Use that URL with yt-dlp
### Known Video Hosts
These hosts are typically supported by yt-dlp:
- Streamtape
- Vidoza
- Mp4upload
- Doodstream
- Filemoon
- Voe.sx
If the video is on an unsupported host, check if there's an alternative server/quality option on the episode page.

View File

@ -0,0 +1,184 @@
{
"name": "Tdarr GPU Monitor",
"nodes": [
{
"parameters": {
"rule": {
"interval": [
{
"field": "minutes",
"minutesInterval": 5
}
]
}
},
"id": "schedule-trigger",
"name": "Every 5 Minutes",
"type": "n8n-nodes-base.scheduleTrigger",
"typeVersion": 1.2,
"position": [0, 0]
},
{
"parameters": {
"command": "ssh -i /root/.ssh/n8n_to_claude -o BatchMode=yes -o ConnectTimeout=10 cal@10.10.0.226 'docker exec tdarr-node nvidia-smi --query-gpu=name --format=csv,noheader 2>&1'"
},
"id": "check-gpu",
"name": "Check GPU Access",
"type": "n8n-nodes-base.executeCommand",
"typeVersion": 1,
"position": [220, 0]
},
{
"parameters": {
"conditions": {
"options": {
"caseSensitive": false,
"leftValue": "",
"typeValidation": "loose"
},
"conditions": [
{
"id": "gpu-error-check",
"leftValue": "={{ $json.stdout + $json.stderr }}",
"rightValue": "NVML|CUDA_ERROR|Unknown Error|no CUDA-capable|Failed to initialize",
"operator": {
"type": "string",
"operation": "regex"
}
}
],
"combinator": "or"
}
},
"id": "check-error",
"name": "GPU Error Detected?",
"type": "n8n-nodes-base.if",
"typeVersion": 2.2,
"position": [440, 0]
},
{
"parameters": {
"command": "ssh -i /root/.ssh/n8n_to_claude -o BatchMode=yes -o ConnectTimeout=10 cal@10.10.0.226 'cd /home/cal/docker/tdarr && docker compose restart tdarr-node 2>&1'"
},
"id": "restart-container",
"name": "Restart tdarr-node",
"type": "n8n-nodes-base.executeCommand",
"typeVersion": 1,
"position": [660, -100]
},
{
"parameters": {
"command": "ssh -i /root/.ssh/n8n_to_claude -o BatchMode=yes -o ConnectTimeout=30 cal@10.10.0.226 'sleep 5 && docker exec tdarr-node nvidia-smi --query-gpu=name --format=csv,noheader 2>&1'"
},
"id": "verify-gpu",
"name": "Verify GPU Restored",
"type": "n8n-nodes-base.executeCommand",
"typeVersion": 1,
"position": [880, -100]
},
{
"parameters": {
"method": "POST",
"url": "https://discord.com/api/webhooks/1451783909409816763/O9PMDiNt6ZIWRf8HKocIZ_E4vMGV_lEwq50aAiZ9HVFR2UGwO6J1N9_wOm82p0MetIqT",
"sendBody": true,
"specifyBody": "json",
"jsonBody": "={\n \"embeds\": [{\n \"title\": \"Tdarr GPU Recovery\",\n \"description\": \"GPU access was lost and container was automatically restarted.\",\n \"color\": {{ $('Verify GPU Restored').item.json.stdout.includes('GTX') || $('Verify GPU Restored').item.json.stdout.includes('NVIDIA') ? 3066993 : 15158332 }},\n \"fields\": [\n {\n \"name\": \"Original Error\",\n \"value\": \"```{{ $('Check GPU Access').item.json.stdout.substring(0, 200) }}{{ $('Check GPU Access').item.json.stderr.substring(0, 200) }}```\",\n \"inline\": false\n },\n {\n \"name\": \"Recovery Status\",\n \"value\": \"{{ $('Verify GPU Restored').item.json.stdout.includes('GTX') || $('Verify GPU Restored').item.json.stdout.includes('NVIDIA') ? '✅ GPU access restored' : '❌ GPU still not accessible - manual intervention needed' }}\",\n \"inline\": false\n },\n {\n \"name\": \"Post-Restart GPU\",\n \"value\": \"```{{ $('Verify GPU Restored').item.json.stdout.substring(0, 200) }}```\",\n \"inline\": false\n }\n ],\n \"timestamp\": \"{{ new Date().toISOString() }}\",\n \"footer\": {\n \"text\": \"Tdarr GPU Monitor | ubuntu-manticore\"\n }\n }]\n}",
"options": {}
},
"id": "discord-notify",
"name": "Discord Notification",
"type": "n8n-nodes-base.httpRequest",
"typeVersion": 4.2,
"position": [1100, -100]
},
{
"parameters": {},
"id": "no-action",
"name": "GPU OK - No Action",
"type": "n8n-nodes-base.noOp",
"typeVersion": 1,
"position": [660, 100]
}
],
"connections": {
"Every 5 Minutes": {
"main": [
[
{
"node": "Check GPU Access",
"type": "main",
"index": 0
}
]
]
},
"Check GPU Access": {
"main": [
[
{
"node": "GPU Error Detected?",
"type": "main",
"index": 0
}
]
]
},
"GPU Error Detected?": {
"main": [
[
{
"node": "Restart tdarr-node",
"type": "main",
"index": 0
}
],
[
{
"node": "GPU OK - No Action",
"type": "main",
"index": 0
}
]
]
},
"Restart tdarr-node": {
"main": [
[
{
"node": "Verify GPU Restored",
"type": "main",
"index": 0
}
]
]
},
"Verify GPU Restored": {
"main": [
[
{
"node": "Discord Notification",
"type": "main",
"index": 0
}
]
]
}
},
"settings": {
"executionOrder": "v1"
},
"staticData": null,
"tags": [
{
"name": "homelab"
},
{
"name": "monitoring"
},
{
"name": "tdarr"
}
],
"triggerCount": 1,
"pinData": {}
}

View File

@ -0,0 +1,385 @@
# VM Management Scripts - Operational Context
## Script Overview
This directory contains active operational scripts for VM provisioning, LXC container creation, Docker configuration in containers, and system migration.
## Core Scripts
### VM Post-Installation Provisioning
**Script**: `vm-post-install.sh`
**Purpose**: Automated provisioning of existing VMs with security hardening, SSH keys, and Docker
**Key Features**:
- System updates and essential packages installation
- SSH key deployment (primary + emergency keys)
- SSH security hardening (disable password authentication)
- Docker and Docker Compose installation
- User environment setup with bash aliases
- Automatic security updates configuration
**Usage**:
```bash
./vm-post-install.sh <vm-ip> [ssh-user]
# Example
./vm-post-install.sh 10.10.0.100 cal
```
**Requirements**:
- Target VM must have SSH access enabled initially
- Homelab SSH keys must exist: `~/.ssh/homelab_rsa` and `~/.ssh/emergency_homelab_rsa`
- Initial connection may require password authentication (disabled after provisioning)
**Post-Provision Verification**:
```bash
# Test SSH with key
ssh cal@<vm-ip>
# Verify Docker
docker --version
docker run --rm hello-world
# Check security
sudo sshd -T | grep -E "(passwordauth|pubkeyauth)"
```
### Cloud-Init Automated Provisioning
**File**: `cloud-init-user-data.yaml`
**Purpose**: Fully automated VM provisioning from first boot using Proxmox cloud-init
**Features**:
- User creation with sudo privileges
- SSH keys pre-installed (no password auth needed)
- Automatic package updates
- Docker and Docker Compose installation
- Security hardening from first boot
- Useful bash aliases and environment setup
- Welcome message with system status
**Usage in Proxmox**:
1. Create new VM with cloud-init support
2. Go to Cloud-Init tab in VM settings
3. Copy contents of `cloud-init-user-data.yaml`
4. Paste into "User Data" field
5. Start VM - fully provisioned automatically
**Benefits**:
- Zero-touch provisioning
- Consistent configuration across all VMs
- No password authentication ever enabled
- Faster deployment than post-install script
### Docker AppArmor Fix for LXC
**Script**: `fix-docker-apparmor.sh`
**Purpose**: Add AppArmor=unconfined to docker-compose.yml files for LXC compatibility
**Why Needed**: Docker containers inside LXC containers require AppArmor to be disabled. Without this fix, containers may fail to start or have permission issues.
**Usage**:
```bash
./fix-docker-apparmor.sh <LXC_IP> [COMPOSE_DIR]
# Example - use default directory (/home/cal/container-data)
./fix-docker-apparmor.sh 10.10.0.214
# Example - specify custom directory
./fix-docker-apparmor.sh 10.10.0.214 /home/cal/docker
```
**What It Does**:
1. SSHs into the LXC container
2. Finds all docker-compose.yml files in specified directory
3. Adds `security_opt: ["apparmor=unconfined"]` to each service
4. Creates `.bak` backups of original files before modification
**Safety Features**:
- Creates backups before modifications
- Color-coded output for easy monitoring
- Error handling with detailed logging
- Validates SSH connectivity before proceeding
### LXC Container Creation with Docker
**Script**: `lxc-docker-create.sh`
**Purpose**: Create new LXC containers pre-configured for Docker hosting
**Key Features**:
- Automated LXC container creation in Proxmox
- Docker and Docker Compose pre-installed
- AppArmor configuration for container compatibility
- Network configuration
- Security settings optimized for Docker hosting
**Usage**:
```bash
./lxc-docker-create.sh [options]
```
**Common Use Cases**:
- Creating Docker hosts for specific services (n8n, gitea, etc.)
- Rapid deployment of containerized applications
- Consistent LXC configuration across infrastructure
### LXC Migration Guide
**Document**: `LXC-MIGRATION-GUIDE.md`
**Purpose**: Step-by-step procedures for migrating LXC containers between hosts
**Covers**:
- Pre-migration planning and backups
- LXC configuration export/import
- Storage migration strategies
- Network reconfiguration
- Post-migration validation
- Rollback procedures
**When to Use**:
- Moving containers to new Proxmox host
- Hardware upgrades
- Load balancing across nodes
- Disaster recovery scenarios
## Operational Patterns
### VM Provisioning Workflow
**Option 1: New VMs (Preferred)**
```bash
# 1. Create VM in Proxmox with cloud-init support
# 2. Copy cloud-init-user-data.yaml to User Data field
# 3. Start VM
# 4. Verify provisioning completed:
ssh cal@<vm-ip>
docker --version
```
**Option 2: Existing VMs**
```bash
# 1. Ensure VM has SSH enabled and accessible
# 2. Run post-install script:
./vm-post-install.sh 10.10.0.100 cal
# 3. Verify provisioning:
ssh cal@10.10.0.100
docker --version
```
### LXC Docker Deployment Workflow
**Creating New LXC for Docker**:
```bash
# 1. Create LXC container
./lxc-docker-create.sh --id 220 --hostname docker-app --ip 10.10.0.220
# 2. Deploy docker-compose configurations
scp -r ./app-config/ cal@10.10.0.220:/home/cal/container-data/
# 3. Fix AppArmor compatibility
./fix-docker-apparmor.sh 10.10.0.220
# 4. Start containers
ssh cal@10.10.0.220 "cd /home/cal/container-data/app-config && docker compose up -d"
```
**Existing LXC with Docker Issues**:
```bash
# If containers failing to start in LXC:
./fix-docker-apparmor.sh <LXC_IP>
# Restart affected containers
ssh cal@<LXC_IP> "cd /home/cal/container-data && docker compose restart"
```
### SSH Key Integration
**Both provisioning methods use**:
- **Primary Key**: `~/.ssh/homelab_rsa` - Daily use authentication
- **Emergency Key**: `~/.ssh/emergency_homelab_rsa` - Backup access
**Security Configuration**:
- Password authentication completely disabled after provisioning
- Only key-based SSH access allowed
- Emergency keys provide backup access if primary key fails
- Automatic security updates enabled
**Key Management Integration**:
- Keys managed by `/networking/scripts/ssh_key_maintenance.sh`
- Monthly backups of all SSH keys
- Rotation recommendations for keys > 365 days old
## Configuration Dependencies
### Required Local Files
- `~/.ssh/homelab_rsa` - Primary SSH private key
- `~/.ssh/homelab_rsa.pub` - Primary SSH public key
- `~/.ssh/emergency_homelab_rsa` - Emergency SSH private key
- `~/.ssh/emergency_homelab_rsa.pub` - Emergency SSH public key
### Target VM Requirements
- **For post-install script**: SSH enabled, initial authentication method available
- **For cloud-init**: Proxmox cloud-init support, fresh VM
- **For LXC**: Proxmox host with LXC support
### Network Requirements
- VMs/LXCs on 10.10.0.0/24 network (homelab subnet)
- SSH access (port 22) to target systems
- Internet access on target systems for package installation
## Troubleshooting Context
### Common Issues
**1. vm-post-install.sh Connection Failures**
```bash
# Verify VM is accessible
ping <vm-ip>
nc -z <vm-ip> 22
# Check SSH service on target
ssh <vm-ip> "systemctl status sshd"
# Verify SSH keys exist locally
ls -la ~/.ssh/homelab_rsa*
```
**2. Cloud-Init Not Working**
```bash
# On Proxmox host, check cloud-init support
qm cloudinit dump <vmid>
# On VM, check cloud-init logs
sudo cloud-init status --long
sudo cat /var/log/cloud-init.log
```
**3. Docker Containers Fail in LXC**
```bash
# Symptom: Containers won't start, permission errors
# Solution: Run AppArmor fix
./fix-docker-apparmor.sh <LXC_IP>
# Verify security_opt was added
ssh cal@<LXC_IP> "grep -r 'security_opt' ~/container-data/"
# Check Docker logs
ssh cal@<LXC_IP> "docker compose logs"
```
**4. SSH Key Authentication Fails After Provisioning**
```bash
# Verify key permissions
ls -la ~/.ssh/homelab_rsa
chmod 600 ~/.ssh/homelab_rsa
# Check authorized_keys on target
ssh <vm-ip> "cat ~/.ssh/authorized_keys"
# Test with verbose output
ssh -v cal@<vm-ip>
```
**5. Docker Installation Issues**
```bash
# Check internet connectivity on VM
ssh <vm-ip> "ping -c 3 8.8.8.8"
# Verify Docker GPG key
ssh <vm-ip> "apt-key list | grep -i docker"
# Check Docker service
ssh <vm-ip> "systemctl status docker"
# Manual Docker install if needed
ssh <vm-ip> "curl -fsSL https://get.docker.com | sh"
```
### Diagnostic Commands
```bash
# Post-provisioning validation
ssh cal@<vm-ip> "groups" # Should include: sudo docker
ssh cal@<vm-ip> "docker run --rm hello-world"
ssh cal@<vm-ip> "sudo sshd -T | grep passwordauth" # Should be "no"
# Cloud-init status check
ssh cal@<vm-ip> "cloud-init status"
ssh cal@<vm-ip> "cloud-init query -f '{{ds.meta_data.hostname}}'"
# Docker in LXC verification
ssh cal@<LXC_IP> "docker info | grep -i apparmor"
ssh cal@<LXC_IP> "docker compose config" # Validate compose files
# SSH key connectivity test
ssh -o ConnectTimeout=5 cal@<vm-ip> "echo 'SSH OK'"
```
## Integration Points
### External Dependencies
- **Proxmox VE**: For VM/LXC creation and cloud-init support
- **SSH**: For remote provisioning and management
- **Docker**: Installed on target systems
- **Cloud-init**: For automated VM provisioning
- **AppArmor**: Security framework (configured for LXC compatibility)
### File System Dependencies
- **Script Directory**: `/mnt/NV2/Development/claude-home/vm-management/scripts/`
- **SSH Keys**: `~/.ssh/homelab_rsa*`, `~/.ssh/emergency_homelab_rsa*`
- **LXC Compose Directories**: Typically `/home/cal/container-data/` on target
- **Backup Files**: `.bak` files created by AppArmor fix script
### Network Dependencies
- **Management Network**: 10.10.0.0/24 subnet
- **Internet Access**: Required for package installation
- **Proxmox API**: For LXC creation operations
- **DNS**: For hostname resolution
## Security Considerations
### SSH Security
- Password authentication disabled after provisioning
- Only key-based authentication allowed
- Emergency keys provide backup access
- Root login disabled
### Docker Security
- User in docker group (no sudo needed for docker commands)
- AppArmor unconfined in LXC (required for functionality)
- Containers run as non-root when possible
- Network isolation via Docker networks
### VM/LXC Security
- Automatic security updates enabled
- Minimal package installation (only essentials)
- Firewall configuration recommended post-provisioning
- Regular key rotation via SSH key maintenance
## Performance Considerations
### Cloud-Init vs Post-Install
- **Cloud-init**: Faster (0-touch), no manual SSH needed, better for multiple VMs
- **Post-install**: More flexible, works with existing VMs, easier debugging
### LXC vs VM
- **LXC**: Lower overhead, faster startup, shared kernel
- **VM**: Better isolation, GPU passthrough support, different kernels possible
### Docker in LXC
- **Performance**: Near-native, minimal overhead with AppArmor disabled
- **I/O**: Use local storage for best performance, NFS for shared data
- **Networking**: Bridge mode for simplicity, macvlan for direct network access
## Related Documentation
- **Technology Overview**: `/vm-management/CONTEXT.md`
- **Troubleshooting**: `/vm-management/troubleshooting.md`
- **Examples**: `/vm-management/examples/` - Configuration templates
- **SSH Management**: `/networking/scripts/ssh_key_maintenance.sh`
- **Docker Patterns**: `/docker/CONTEXT.md`
- **Main Instructions**: `/CLAUDE.md` - Context loading rules
## Notes
These scripts form the foundation of the homelab VM and LXC provisioning strategy. They ensure consistent configuration, security hardening, and Docker compatibility across all virtualized infrastructure.
The cloud-init approach is preferred for new deployments due to zero-touch provisioning, while the post-install script provides flexibility for existing systems or troubleshooting scenarios.
AppArmor configuration is critical for Docker-in-LXC deployments and should be applied to all LXC containers running Docker to prevent container startup failures.