claude-home/networking/troubleshooting.md

title	description	type	domain	tags
Networking Troubleshooting Guide	Comprehensive troubleshooting for SSH, DNS, reverse proxy, SSL, CIFS/NFS mounts, Pi-hole HA, iOS DNS bypass, UniFi firewall rules, and emergency recovery procedures.	troubleshooting	networking	ssh dns pihole ssl cifs nfs firewall unifi ios nginx troubleshooting

Networking Infrastructure Troubleshooting Guide

SSH Connection Issues

SSH Authentication Failures

Symptoms: Permission denied, connection refused, timeout Diagnosis:

# Verbose SSH debugging
ssh -vvv user@host

# Test different authentication methods
ssh -o PasswordAuthentication=no user@host
ssh -o PubkeyAuthentication=yes user@host

# Check local key files
ls -la ~/.ssh/
ssh-keygen -lf ~/.ssh/homelab_rsa.pub

Solutions:

# Re-deploy SSH keys
ssh-copy-id -i ~/.ssh/homelab_rsa.pub user@host
ssh-copy-id -i ~/.ssh/emergency_homelab_rsa.pub user@host

# Fix key permissions
chmod 600 ~/.ssh/homelab_rsa
chmod 644 ~/.ssh/homelab_rsa.pub
chmod 700 ~/.ssh

# Verify remote authorized_keys
ssh user@host 'chmod 700 ~/.ssh && chmod 600 ~/.ssh/authorized_keys'

SSH Service Issues

Symptoms: Connection refused, service not running Diagnosis:

# Check SSH service status
systemctl status sshd
ss -tlnp | grep :22

# Test port connectivity
nc -zv host 22
nmap -p 22 host

Solutions:

# Restart SSH service
sudo systemctl restart sshd
sudo systemctl enable sshd

# Check firewall
sudo ufw status
sudo ufw allow ssh

# Verify SSH configuration
sudo sshd -T | grep -E "(passwordauth|pubkeyauth|permitroot)"

Network Connectivity Problems

Basic Network Troubleshooting

Symptoms: Cannot reach hosts, timeouts, routing issues Diagnosis:

# Basic connectivity tests
ping host
traceroute host
mtr host

# Check local network configuration
ip addr show
ip route show
cat /etc/resolv.conf

Solutions:

# Restart networking
sudo systemctl restart networking
sudo netplan apply  # Ubuntu

# Reset network interface
sudo ip link set eth0 down
sudo ip link set eth0 up

# Check default gateway
sudo ip route add default via 10.10.0.1

DNS Resolution Issues

Symptoms: Cannot resolve hostnames, slow resolution Diagnosis:

# Test DNS resolution
nslookup google.com
dig google.com
host google.com

# Check DNS servers
systemd-resolve --status
cat /etc/resolv.conf

Solutions:

# Temporary DNS fix
echo "nameserver 8.8.8.8" | sudo tee /etc/resolv.conf

# Restart DNS services
sudo systemctl restart systemd-resolved

# Flush DNS cache
sudo systemd-resolve --flush-caches

UniFi Firewall Blocking DNS to New Networks

Symptoms: New network/VLAN has "no internet access" - devices connect to WiFi but cannot browse or resolve domain names. Ping to IP addresses (8.8.8.8) works, but DNS resolution fails.

Root Cause: Firewall rules blocking traffic from DNS servers (Pi-holes in "Servers" network group) to new networks. Rules like "Servers to WiFi" or "Servers to Home" with DROP action block ALL traffic including DNS responses on port 53.

Diagnosis:

# From affected device on new network:

# Test if routing works (should succeed)
ping 8.8.8.8
traceroute 8.8.8.8

# Test if DNS resolution works (will fail)
nslookup google.com

# Test DNS servers directly (will timeout or fail)
nslookup google.com 10.10.0.16
nslookup google.com 10.10.0.226

# Test public DNS (should work)
nslookup google.com 8.8.8.8

# Check DHCP-assigned DNS servers
# Windows:
ipconfig /all | findstr DNS

# Linux/macOS:
cat /etc/resolv.conf

If routing works but DNS fails, the issue is firewall blocking DNS traffic, not network configuration.

Solutions:

Step 1: Identify Blocking Rules

• In UniFi: Settings → Firewall & Security → Traffic Rules → LAN In
• Look for DROP rules with:
  • Source: Servers (or network group containing Pi-holes)
  • Destination: Your new network (e.g., "Home WiFi", "Home Network")
  • Examples: "Servers to WiFi", "Servers to Home"

Step 2: Create DNS Allow Rules (BEFORE Drop Rules)

Create new rules positioned ABOVE the drop rules:

Name: Allow DNS - Servers to [Network Name]
Action: Accept
Rule Applied: Before Predefined Rules
Type: LAN In
Protocol: TCP and UDP
Source:
  - Network/Group: Servers (or specific Pi-hole IPs: 10.10.0.16, 10.10.0.226)
  - Port: Any
Destination:
  - Network: [Your new network - e.g., Home WiFi]
  - Port: 53 (DNS)

Repeat for each network that needs DNS access from servers.

Step 3: Verify Rule Order

CRITICAL: Firewall rules process top-to-bottom, first match wins!

Correct order:

✅ Allow DNS - Servers to Home Network (Accept, Port 53)
✅ Allow DNS - Servers to Home WiFi (Accept, Port 53)
❌ Servers to Home (Drop, All ports)
❌ Servers to WiFi (Drop, All ports)

Step 4: Re-enable Drop Rules

Once DNS allow rules are in place and positioned correctly, re-enable the drop rules.

Verification:

# From device on new network:

# DNS should work
nslookup google.com

# Browsing should work
ping google.com

# Other server traffic should still be blocked (expected)
ping 10.10.0.16  # Should fail or timeout
ssh 10.10.0.16   # Should be blocked

Real-World Example: New "Home WiFi" network (10.1.0.0/24, VLAN 2)

• Problem: Devices connected but couldn't browse web
• Diagnosis: traceroute 8.8.8.8 worked (16ms), but nslookup google.com failed
• Cause: Firewall rule "Servers to WiFi" (rule 20004) blocked Pi-hole DNS responses
• Solution: Added "Allow DNS - Servers to Home WiFi" rule (Accept, port 53) above drop rule
• Result: DNS resolution works, other server traffic remains properly blocked

Reverse Proxy and Load Balancer Issues

Nginx Configuration Problems

Symptoms: 502 Bad Gateway, 503 Service Unavailable, SSL errors Diagnosis:

# Check Nginx status and logs
systemctl status nginx
sudo tail -f /var/log/nginx/error.log
sudo tail -f /var/log/nginx/access.log

# Test Nginx configuration
sudo nginx -t
sudo nginx -T  # Show full configuration

Solutions:

# Reload Nginx configuration
sudo nginx -s reload

# Check upstream servers
curl -I http://backend-server:port
telnet backend-server port

# Fix common configuration issues
sudo nano /etc/nginx/sites-available/default
# Check proxy_pass URLs, upstream definitions

SSL/TLS Certificate Issues

Symptoms: Certificate warnings, expired certificates, connection errors Diagnosis:

# Check certificate validity
openssl s_client -connect host:443 -servername host
openssl x509 -in /etc/ssl/certs/cert.pem -text -noout

# Check certificate expiry
openssl x509 -in /etc/ssl/certs/cert.pem -noout -dates

Solutions:

# Renew Let's Encrypt certificates
sudo certbot renew --dry-run
sudo certbot renew --force-renewal

# Generate self-signed certificate
sudo openssl req -x509 -nodes -days 365 -newkey rsa:2048 \
    -keyout /etc/ssl/private/selfsigned.key \
    -out /etc/ssl/certs/selfsigned.crt

Intermittent SSL Errors (ERR_SSL_UNRECOGNIZED_NAME_ALERT)

Symptoms: SSL errors that work sometimes but fail other times, ERR_SSL_UNRECOGNIZED_NAME_ALERT in browser, connection works from internal network intermittently

Root Cause: IPv6/IPv4 DNS conflicts where public DNS returns Cloudflare IPv6 addresses while local DNS (Pi-hole) only overrides IPv4. Modern systems prefer IPv6, causing intermittent failures when IPv6 connection attempts fail.

Diagnosis:

# Check for multiple DNS records (IPv4 + IPv6)
nslookup domain.example.com 10.10.0.16
dig domain.example.com @10.10.0.16

# Compare with public DNS
host domain.example.com 8.8.8.8

# Test IPv6 vs IPv4 connectivity
curl -6 -I https://domain.example.com  # IPv6 (may fail)
curl -4 -I https://domain.example.com  # IPv4 (should work)

# Check if system has IPv6 connectivity
ip -6 addr show | grep global

Example Problem:

# Local Pi-hole returns:
domain.example.com → 10.10.0.16 (IPv4 internal NPM)

# Public DNS also returns:
domain.example.com → 2606:4700:... (Cloudflare IPv6)

# System tries IPv6 first → fails
# Sometimes falls back to IPv4 → works
# Result: Intermittent SSL errors

Solutions:

Option 1: Add IPv6 Local DNS Override (Recommended)

# Add non-routable IPv6 address to Pi-hole custom.list
ssh pihole "docker exec pihole bash -c 'echo \"fe80::1 domain.example.com\" >> /etc/pihole/custom.list'"

# Restart Pi-hole DNS
ssh pihole "docker exec pihole pihole restartdns"

# Verify fix
nslookup domain.example.com 10.10.0.16
# Should show: 10.10.0.16 (IPv4) and fe80::1 (IPv6 link-local)

Option 2: Remove Cloudflare DNS Records (If public access not needed)

# In Cloudflare dashboard:
# - Turn off orange cloud (proxy) for the domain
# - Or delete A/AAAA records entirely

# This removes Cloudflare IPs from public DNS

Option 3: Disable IPv6 on Client (Temporary testing)

# Disable IPv6 temporarily to confirm diagnosis
sudo sysctl -w net.ipv6.conf.all.disable_ipv6=1

# Test domain - should work consistently now

# Re-enable when done testing
sudo sysctl -w net.ipv6.conf.all.disable_ipv6=0

Verification:

# After applying fix, verify consistent resolution
for i in {1..10}; do
    echo "Test $i:"
    curl -I https://domain.example.com 2>&1 | grep -E "(HTTP|SSL|certificate)"
    sleep 1
done

# All attempts should succeed consistently

Real-World Example: git.manticorum.com

• Problem: Intermittent SSL errors from internal network (10.0.0.0/24)
• Diagnosis: Pi-hole had IPv4 override (10.10.0.16) but public DNS returned Cloudflare IPv6
• Solution: Added fe80::1 git.manticorum.com to Pi-hole custom.list
• Result: Consistent successful connections, always routes to internal NPM

iOS DNS Bypass Issues (Encrypted DNS)

Symptoms: iOS device gets 403 errors when accessing internal services, NPM logs show external public IP as source instead of local 10.x.x.x IP, even with correct Pi-hole DNS configuration

Root Cause: iOS devices can use encrypted DNS (DNS-over-HTTPS or DNS-over-TLS) that bypasses traditional DNS servers, even when correctly configured. This causes the device to resolve to public/Cloudflare IPs instead of local overrides, routing traffic through the public internet and triggering ACL denials.

Diagnosis:

# Check NPM access logs for the service
ssh 10.10.0.16 "docker exec nginx-proxy-manager_app_1 tail -50 /data/logs/proxy-host-*_access.log | grep 403"

# Look for external IPs in logs instead of local 10.x.x.x:
# BAD:  [Client 73.36.102.55] - - 403 (external IP, blocked by ACL)
# GOOD: [Client 10.0.0.207] - 200 200 (local IP, allowed)

# Verify iOS device is on local network
# On iOS: Settings → Wi-Fi → (i) → IP Address
# Should show 10.0.0.x or 10.10.0.x

# Verify Pi-hole DNS is configured
# On iOS: Settings → Wi-Fi → (i) → DNS
# Should show 10.10.0.16

# Test if DNS is actually being used
nslookup domain.example.com 10.10.0.16  # Shows what Pi-hole returns
# Then check what iOS actually resolves (if possible via network sniffer)

Example Problem:

# iOS device configuration:
IP Address: 10.0.0.207 (correct, on local network)
DNS: 10.10.0.16 (correct, Pi-hole configured)
Cellular Data: OFF

# But NPM logs show:
[Client 73.36.102.55] - - 403  # Coming from ISP public IP!

# Why: iOS is using encrypted DNS, bypassing Pi-hole
# Result: Resolves to Cloudflare IP, routes through public internet,
#         NPM sees external IP, ACL blocks with 403

Solutions:

Option 1: Add Public IP to NPM Access Rules (Quickest, recommended for mobile devices)

# Find which config file contains your domain
ssh 10.10.0.16 "docker exec nginx-proxy-manager_app_1 sh -c 'grep -l domain.example.com /data/nginx/proxy_host/*.conf'"
# Example output: /data/nginx/proxy_host/19.conf

# Add public IP to access rules (replace YOUR_PUBLIC_IP and config number)
ssh 10.10.0.16 "docker exec nginx-proxy-manager_app_1 sed -i '/allow 10.10.0.0\/24;/a \    \n    allow YOUR_PUBLIC_IP;' /data/nginx/proxy_host/19.conf"

# Verify the change
ssh 10.10.0.16 "docker exec nginx-proxy-manager_app_1 cat /data/nginx/proxy_host/19.conf" | grep -A 8 "Access Rules"

# Test and reload nginx
ssh 10.10.0.16 "docker exec nginx-proxy-manager_app_1 nginx -t"
ssh 10.10.0.16 "docker exec nginx-proxy-manager_app_1 nginx -s reload"

Option 2: Reset iOS Network Settings (Nuclear option, clears DNS cache/profiles)

iOS: Settings → General → Transfer or Reset iPhone → Reset → Reset Network Settings
WARNING: This removes all saved WiFi passwords and network configurations

Option 3: Check for DNS Configuration Profiles

iOS: Settings → General → VPN & Device Management
- Look for any DNS or Configuration Profiles
- Remove any third-party DNS profiles (AdGuard, NextDNS, etc.)

Option 4: Disable Private Relay and IP Tracking (Usually already tried)

iOS: Settings → [Your Name] → iCloud → Private Relay → OFF
iOS: Settings → Wi-Fi → (i) → Limit IP Address Tracking → OFF

Option 5: Check Browser DNS Settings (If using Brave or Firefox)

Brave: Settings → Brave Shields & Privacy → Use secure DNS → OFF
Firefox: Settings → DNS over HTTPS → OFF

Verification:

# After applying fix, check NPM logs while accessing from iOS
ssh 10.10.0.16 "docker exec nginx-proxy-manager_app_1 tail -f /data/logs/proxy-host-*_access.log"

# With Option 1 (added public IP): Should see 200 status with external IP
# With Option 2-5 (fixed DNS): Should see 200 status with local 10.x.x.x IP

Important Notes:

• Option 1 is recommended for mobile devices as iOS encrypted DNS behavior is inconsistent
• Public IP workaround requires updating if ISP changes your IP (rare for residential)
• Manual nginx config changes (Option 1) will be overwritten if you edit the proxy host in NPM UI
• To make permanent, either use NPM UI to add the IP, or re-apply after UI changes
• This issue can affect any iOS device (iPhone, iPad) and some Android devices with encrypted DNS

Real-World Example: git.manticorum.com iOS Access

• Problem: iPhone showing 403 errors, desktop working fine on same network
• iOS Config: IP 10.0.0.207, DNS 10.10.0.16, Cellular OFF (all correct)
• NPM Logs: iPhone requests showing as [Client 73.36.102.55] (ISP public IP)
• Diagnosis: iOS using encrypted DNS, bypassing Pi-hole, routing through Cloudflare
• Solution: Added allow 73.36.102.55; to NPM proxy_host/19.conf ACL rules
• Result: Immediate access, user able to log in to Gitea successfully

Network Storage Issues

CIFS/SMB Mount Problems

Symptoms: Mount failures, connection timeouts, permission errors Diagnosis:

# Test SMB connectivity
smbclient -L //nas-server -U username
testparm  # Test Samba configuration

# Check mount status
mount | grep cifs
df -h | grep cifs

Solutions:

# Remount with verbose logging
sudo mount -t cifs //server/share /mnt/point -o username=user,password=pass,vers=3.0

# Fix mount options in /etc/fstab
//server/share /mnt/point cifs credentials=/etc/cifs/credentials,uid=1000,gid=1000,iocharset=utf8,file_mode=0644,dir_mode=0755,cache=strict,_netdev 0 0

# Test credentials
sudo cat /etc/cifs/credentials
# Should contain: username=, password=, domain=

NFS Mount Issues

Symptoms: Stale file handles, mount hangs, permission denied Diagnosis:

# Check NFS services
systemctl status nfs-client.target
showmount -e nfs-server

# Test NFS connectivity
rpcinfo -p nfs-server

Solutions:

# Restart NFS services
sudo systemctl restart nfs-client.target

# Remount NFS shares
sudo umount /mnt/nfs-share
sudo mount -t nfs server:/path /mnt/nfs-share

# Fix stale file handles
sudo umount -f /mnt/nfs-share
sudo mount /mnt/nfs-share

Firewall and Security Issues

Port Access Problems

Symptoms: Connection refused, filtered ports, blocked services Diagnosis:

# Check firewall status
sudo ufw status verbose
sudo iptables -L -n -v

# Test port accessibility
nc -zv host port
nmap -p port host

Solutions:

# Open required ports
sudo ufw allow ssh
sudo ufw allow 80/tcp
sudo ufw allow 443/tcp
sudo ufw allow from 10.10.0.0/24

# Reset firewall if needed
sudo ufw --force reset
sudo ufw enable

Network Security Issues

Symptoms: Unauthorized access, suspicious traffic, security alerts Diagnosis:

# Check active connections
ss -tuln
netstat -tuln

# Review logs for security events
sudo tail -f /var/log/auth.log
sudo tail -f /var/log/syslog | grep -i security

Solutions:

# Block suspicious IPs
sudo ufw deny from suspicious-ip

# Update SSH security
sudo nano /etc/ssh/sshd_config
# Set: PasswordAuthentication no, PermitRootLogin no
sudo systemctl restart sshd

Pi-hole High Availability Troubleshooting

Pi-hole Not Responding to DNS Queries

Symptoms: DNS resolution failures, clients cannot resolve domains, Pi-hole web UI inaccessible Diagnosis:

# Test DNS response from both Pi-holes
dig @10.10.0.16 google.com
dig @10.10.0.226 google.com

# Check Pi-hole container status
ssh npm-pihole "docker ps | grep pihole"
ssh ubuntu-manticore "docker ps | grep pihole"

# Check Pi-hole logs
ssh npm-pihole "docker logs pihole --tail 50"
ssh ubuntu-manticore "docker logs pihole --tail 50"

# Test port 53 is listening
ssh ubuntu-manticore "netstat -tulpn | grep :53"
ssh ubuntu-manticore "ss -tulpn | grep :53"

Solutions:

# Restart Pi-hole containers
ssh npm-pihole "docker restart pihole"
ssh ubuntu-manticore "cd ~/docker/pihole && docker compose restart"

# Check for port conflicts
ssh ubuntu-manticore "lsof -i :53"

# If systemd-resolved is conflicting, disable it
ssh ubuntu-manticore "sudo systemctl stop systemd-resolved"
ssh ubuntu-manticore "sudo systemctl disable systemd-resolved"

# Rebuild Pi-hole container
ssh ubuntu-manticore "cd ~/docker/pihole && docker compose down && docker compose up -d"

DNS Failover Not Working

Symptoms: DNS stops working when primary Pi-hole fails, clients not using secondary DNS Diagnosis:

# Check UniFi DHCP DNS configuration
# Via UniFi UI: Settings → Networks → LAN → DHCP
# DNS Server 1: 10.10.0.16
# DNS Server 2: 10.10.0.226

# Check client DNS configuration
# Windows:
ipconfig /all | findstr /i "DNS"

# Linux/macOS:
cat /etc/resolv.conf

# Check if secondary Pi-hole is reachable
ping -c 4 10.10.0.226
dig @10.10.0.226 google.com

# Test failover manually
ssh npm-pihole "docker stop pihole"
dig google.com  # Should still work via secondary
ssh npm-pihole "docker start pihole"

Solutions:

# Force DHCP lease renewal to get updated DNS servers
# Windows:
ipconfig /release && ipconfig /renew

# Linux:
sudo dhclient -r && sudo dhclient

# macOS/iOS:
# Disconnect and reconnect to WiFi

# Verify UniFi DHCP settings are correct
# Both DNS servers must be configured in UniFi controller

# Check client respects both DNS servers
# Some clients may cache failed DNS responses
# Flush DNS cache:
# Windows: ipconfig /flushdns
# macOS: sudo dscacheutil -flushcache
# Linux: sudo systemd-resolve --flush-caches

Orbital Sync Not Syncing

Symptoms: Blocklists/whitelists differ between Pi-holes, custom DNS entries missing on secondary Diagnosis:

# Check Orbital Sync container status
ssh ubuntu-manticore "docker ps | grep orbital-sync"

# Check Orbital Sync logs
ssh ubuntu-manticore "docker logs orbital-sync --tail 100"

# Look for sync errors in logs
ssh ubuntu-manticore "docker logs orbital-sync 2>&1 | grep -i error"

# Verify API tokens are correct
ssh ubuntu-manticore "cat ~/docker/orbital-sync/.env"

# Test API access manually
ssh npm-pihole "docker exec pihole pihole -a -p"  # Get API token
curl -H "Authorization: Token YOUR_TOKEN" http://10.10.0.16/admin/api.php?status

# Compare blocklist counts between Pi-holes
ssh npm-pihole "docker exec pihole pihole -g -l"
ssh ubuntu-manticore "docker exec pihole pihole -g -l"

Solutions:

# Regenerate API tokens
# Primary Pi-hole: http://10.10.0.16/admin → Settings → API → Generate New Token
# Secondary Pi-hole: http://10.10.0.226:8053/admin → Settings → API → Generate New Token

# Update Orbital Sync .env file
ssh ubuntu-manticore "nano ~/docker/orbital-sync/.env"
# Update PRIMARY_HOST_PASSWORD and SECONDARY_HOST_PASSWORD

# Restart Orbital Sync
ssh ubuntu-manticore "cd ~/docker/orbital-sync && docker compose restart"

# Force immediate sync by restarting
ssh ubuntu-manticore "cd ~/docker/orbital-sync && docker compose down && docker compose up -d"

# Monitor sync in real-time
ssh ubuntu-manticore "docker logs orbital-sync -f"

# If all else fails, manually sync via Teleporter
# Primary: Settings → Teleporter → Backup
# Secondary: Settings → Teleporter → Restore (upload backup file)

NPM DNS Sync Failing

Symptoms: NPM proxy hosts missing from Pi-hole custom.list, new domains not resolving Diagnosis:

# Check NPM sync script status
ssh npm-pihole "cat /var/log/cron.log | grep npm-pihole-sync"

# Run sync script manually to see errors
ssh npm-pihole "/home/cal/scripts/npm-pihole-sync.sh"

# Check script can access both Pi-holes
ssh npm-pihole "docker exec pihole cat /etc/pihole/custom.list | grep git.manticorum.com"
ssh npm-pihole "ssh ubuntu-manticore 'docker exec pihole cat /etc/pihole/custom.list | grep git.manticorum.com'"

# Verify SSH connectivity to ubuntu-manticore
ssh npm-pihole "ssh ubuntu-manticore 'echo SSH OK'"

Solutions:

# Fix SSH key authentication (if needed)
ssh npm-pihole "ssh-copy-id ubuntu-manticore"

# Test script with dry-run
ssh npm-pihole "/home/cal/scripts/npm-pihole-sync.sh --dry-run"

# Run script manually to sync immediately
ssh npm-pihole "/home/cal/scripts/npm-pihole-sync.sh"

# Verify cron job is configured
ssh npm-pihole "crontab -l | grep npm-pihole-sync"

# If cron job missing, add it
ssh npm-pihole "crontab -e"
# Add: 0 * * * * /home/cal/scripts/npm-pihole-sync.sh >> /var/log/npm-pihole-sync.log 2>&1

# Check script logs
ssh npm-pihole "tail -50 /var/log/npm-pihole-sync.log"

Secondary Pi-hole Performance Issues

Symptoms: ubuntu-manticore slow, high CPU/RAM usage, Pi-hole affecting Jellyfin/Tdarr Diagnosis:

# Check resource usage
ssh ubuntu-manticore "docker stats --no-stream"

# Pi-hole should use <1% CPU and ~150MB RAM
# If higher, investigate:
ssh ubuntu-manticore "docker logs pihole --tail 100"

# Check for excessive queries
ssh ubuntu-manticore "docker exec pihole pihole -c -e"

# Check for DNS loops or misconfiguration
ssh ubuntu-manticore "docker exec pihole pihole -t"  # Tail pihole.log

Solutions:

# Restart Pi-hole if resource usage is high
ssh ubuntu-manticore "docker restart pihole"

# Check for DNS query loops
# Look for same domain being queried repeatedly
ssh ubuntu-manticore "docker exec pihole pihole -t | grep -A 5 'query\[A\]'"

# Adjust Pi-hole cache settings if needed
ssh ubuntu-manticore "docker exec pihole bash -c 'echo \"cache-size=10000\" >> /etc/dnsmasq.d/99-custom.conf'"
ssh ubuntu-manticore "docker restart pihole"

# If Jellyfin/Tdarr are affected, verify Pi-hole is using minimal resources
# Resource limits can be added to docker-compose.yml:
ssh ubuntu-manticore "nano ~/docker/pihole/docker-compose.yml"
# Add under pihole service:
#   deploy:
#     resources:
#       limits:
#         cpus: '0.5'
#         memory: 256M

iOS Devices Still Getting 403 Errors (Post-HA Deployment)

Symptoms: After deploying dual Pi-hole setup, iOS devices still bypass DNS and get 403 errors on internal services Diagnosis:

# Verify UniFi DHCP has BOTH Pi-holes configured, NO public DNS
# UniFi UI: Settings → Networks → LAN → DHCP → Name Server
# DNS1: 10.10.0.16
# DNS2: 10.10.0.226
# Public DNS (1.1.1.1, 8.8.8.8): REMOVED

# Check iOS DNS settings
# iOS: Settings → WiFi → (i) → DNS
# Should show: 10.10.0.16

# Force iOS DHCP renewal
# iOS: Settings → WiFi → Forget Network → Reconnect

# Check NPM logs for request source
ssh npm-pihole "docker exec nginx-proxy-manager_app_1 tail -50 /data/logs/proxy-host-*_access.log | grep 403"

# Verify both Pi-holes have custom DNS entries
ssh npm-pihole "docker exec pihole cat /etc/pihole/custom.list | grep git.manticorum.com"
ssh ubuntu-manticore "docker exec pihole cat /etc/pihole/custom.list | grep git.manticorum.com"

Solutions:

# Solution 1: Verify public DNS is removed from UniFi DHCP
# If public DNS (1.1.1.1) is still configured, iOS will prefer it
# Remove ALL public DNS servers from UniFi DHCP configuration

# Solution 2: Force iOS to renew DHCP lease
# iOS: Settings → WiFi → Forget Network
# Then reconnect to WiFi
# This forces device to get new DNS servers from DHCP

# Solution 3: Disable iOS encrypted DNS if still active
# iOS: Settings → [Your Name] → iCloud → Private Relay → OFF
# iOS: Check for DNS profiles: Settings → General → VPN & Device Management

# Solution 4: If encrypted DNS persists, add public IP to NPM ACL (fallback)
# See "iOS DNS Bypass Issues" section above for detailed steps

# Solution 5: Test with different iOS device to isolate issue
# If other iOS devices work, issue is device-specific configuration

# Verification after fix
ssh npm-pihole "docker exec nginx-proxy-manager_app_1 tail -f /data/logs/proxy-host-*_access.log"
# Access git.manticorum.com from iOS
# Should see: [Client 10.0.0.x] - - 200 (local IP)

Both Pi-holes Failing Simultaneously

Symptoms: Complete DNS failure across network, all devices cannot resolve domains Diagnosis:

# Check both Pi-hole containers
ssh npm-pihole "docker ps -a | grep pihole"
ssh ubuntu-manticore "docker ps -a | grep pihole"

# Check both hosts are reachable
ping -c 4 10.10.0.16
ping -c 4 10.10.0.226

# Check Docker daemon on both hosts
ssh npm-pihole "systemctl status docker"
ssh ubuntu-manticore "systemctl status docker"

# Test emergency DNS (bypassing Pi-hole)
dig @8.8.8.8 google.com

Solutions:

# Emergency: Temporarily use public DNS
# UniFi UI: Settings → Networks → LAN → DHCP → Name Server
# DNS1: 8.8.8.8 (Google DNS - temporary)
# DNS2: 1.1.1.1 (Cloudflare - temporary)

# Restart both Pi-holes
ssh npm-pihole "docker restart pihole"
ssh ubuntu-manticore "docker restart pihole"

# If Docker daemon issues:
ssh npm-pihole "sudo systemctl restart docker"
ssh ubuntu-manticore "sudo systemctl restart docker"

# Rebuild both Pi-holes if corruption suspected
ssh npm-pihole "cd ~/pihole && docker compose down && docker compose up -d"
ssh ubuntu-manticore "cd ~/docker/pihole && docker compose down && docker compose up -d"

# After Pi-holes are restored, revert UniFi DHCP to Pi-holes
# UniFi UI: Settings → Networks → LAN → DHCP → Name Server
# DNS1: 10.10.0.16
# DNS2: 10.10.0.226

Query Load Not Balanced Between Pi-holes

Symptoms: Primary Pi-hole getting most queries, secondary rarely used Diagnosis:

# Check query counts on both Pi-holes
# Primary: http://10.10.0.16/admin → Dashboard → Total Queries
# Secondary: http://10.10.0.226:8053/admin → Dashboard → Total Queries

# This is NORMAL behavior - clients prefer DNS1 by default
# Secondary is for failover, not load balancing

# To verify failover works:
ssh npm-pihole "docker stop pihole"
# Wait 30 seconds
# Check secondary query count - should increase
ssh npm-pihole "docker start pihole"

Solutions:

# No action needed - this is expected behavior
# DNS failover is for redundancy, not load distribution

# If you want true load balancing (advanced):
# Option 1: Configure some devices to prefer DNS2
# Manually set DNS on specific devices to 10.10.0.226, 10.10.0.16

# Option 2: Implement DNS round-robin (requires custom DHCP)
# Not recommended for homelab - adds complexity

# Option 3: Accept default behavior (recommended)
# Primary handles most traffic, secondary provides failover
# This is industry standard DNS HA behavior

Pi-hole Blocklist Blocking Legitimate Apps

Facebook Blocklist Breaking Messenger Kids (2026-03-05)

Symptoms: iPad could not connect to Facebook Messenger Kids. App would not load or send/receive messages. Disconnecting iPad from WiFi (using cellular) restored functionality.

Root Cause: The anudeepND/blacklist/master/facebook.txt blocklist was subscribed in Pi-hole, which blocked all core Facebook domains needed by Messenger Kids.

Blocked Domains (from pihole.log):

Domain	Purpose
edge-mqtt.facebook.com	MQTT real-time message transport
graph.facebook.com	Facebook Graph API (login, contacts, profiles)
graph-fallback.facebook.com	Graph API fallback (blocked via CNAME chain)
www.facebook.com	Core Facebook domain

Allowed Domains (not on the blocklist, resolved fine):

• dgw.c10r.facebook.com - Data gateway
• mqtt.fallback.c10r.facebook.com - MQTT fallback
• chat-e2ee.c10r.facebook.com - E2E encrypted chat

Diagnosis:

# Find blocked domains for a specific client IP
ssh pihole "docker exec pihole grep 'CLIENT_IP' /var/log/pihole/pihole.log | grep 'gravity blocked'"

# Check which blocklist contains a domain
ssh pihole "docker exec pihole pihole -q edge-mqtt.facebook.com"
# Output: https://raw.githubusercontent.com/anudeepND/blacklist/master/facebook.txt (block)

Resolution: Removed the Facebook blocklist from primary Pi-hole (secondary didn't have it). The blocklist contained ~3,997 Facebook domains.

Pi-hole v6 API - Deleting a Blocklist:

# Authenticate and get session ID
SID=$(curl -s -X POST 'http://PIHOLE_IP:PORT/api/auth' \
  -H 'Content-Type: application/json' \
  -d '{"password":"APP_PASSWORD"}' \
  | python3 -c 'import sys,json; print(json.load(sys.stdin)["session"]["sid"])')

# DELETE uses the URL-encoded list ADDRESS as path parameter (NOT numeric ID)
# The ?type=block parameter is REQUIRED
curl -s -X DELETE \
  "http://PIHOLE_IP:PORT/api/lists/URL_ENCODED_LIST_ADDRESS?type=block" \
  -H "X-FTL-SID: $SID"
# Success returns HTTP 204 No Content

# Update gravity after removal
ssh pihole "docker exec pihole pihole -g"

# Verify domain is no longer blocked
ssh pihole "docker exec pihole pihole -q edge-mqtt.facebook.com"

Important Pi-hole v6 API Notes:

• List endpoints use the URL-encoded blocklist address as path param, not numeric IDs
• ?type=block query parameter is mandatory for DELETE operations
• Numeric ID DELETE returns 200 with {"took": ...} but DOES NOT actually delete (silent failure)
• Successful address-based DELETE returns HTTP 204 (no body)
• Must run pihole -g (gravity update) after deletion for changes to take effect

Future Improvement (TODO): Implement Pi-hole v6 group/client-based approach:

• Create a group for the iPad that bypasses the Facebook blocklist
• Re-add the Facebook blocklist assigned to the default group only
• Assign the iPad's IP to a "Kids Devices" client group that excludes the Facebook list
• This would maintain Facebook blocking for other devices while allowing Messenger Kids
• See: Pi-hole v6 Admin -> Groups/Clients for per-device blocklist management

Service Discovery and DNS Issues

Local DNS Problems

Symptoms: Services unreachable by hostname, DNS timeouts Diagnosis:

# Test local DNS resolution
nslookup service.homelab.local
dig @10.10.0.16 service.homelab.local

# Check DNS server status
systemctl status bind9  # or named

Solutions:

# Add to /etc/hosts as temporary fix
echo "10.10.0.100 service.homelab.local" | sudo tee -a /etc/hosts

# Restart DNS services
sudo systemctl restart bind9
sudo systemctl restart systemd-resolved

Container Networking Issues

Symptoms: Containers cannot communicate, service discovery fails Diagnosis:

# Check Docker networks
docker network ls
docker network inspect bridge

# Test container connectivity
docker exec container1 ping container2
docker exec container1 nslookup container2

Solutions:

# Create custom network
docker network create --driver bridge app-network
docker run --network app-network container

# Fix DNS in containers
docker run --dns 8.8.8.8 container

Performance Issues

Network Latency Problems

Symptoms: Slow response times, timeouts, poor performance Diagnosis:

# Measure network latency
ping -c 100 host
mtr --report host

# Check network interface stats
ip -s link show
cat /proc/net/dev

Solutions:

# Optimize network settings
echo 'net.core.rmem_max = 134217728' | sudo tee -a /etc/sysctl.conf
echo 'net.core.wmem_max = 134217728' | sudo tee -a /etc/sysctl.conf
sudo sysctl -p

# Check for network congestion
iftop
nethogs

Bandwidth Issues

Symptoms: Slow transfers, network congestion, dropped packets Diagnosis:

# Test bandwidth
iperf3 -s  # Server
iperf3 -c server-ip  # Client

# Check interface utilization
vnstat -i eth0

Solutions:

# Implement QoS if needed
sudo tc qdisc add dev eth0 root fq_codel

# Optimize buffer sizes
sudo ethtool -G eth0 rx 4096 tx 4096

Emergency Recovery Procedures

Network Emergency Recovery

Complete network failure recovery:

# Reset all network configuration
sudo systemctl stop networking
sudo ip addr flush eth0
sudo ip route flush table main
sudo systemctl start networking

# Manual network configuration
sudo ip addr add 10.10.0.100/24 dev eth0
sudo ip route add default via 10.10.0.1
echo "nameserver 8.8.8.8" | sudo tee /etc/resolv.conf

SSH Emergency Access

When locked out of systems:

# Use emergency SSH key
ssh -i ~/.ssh/emergency_homelab_rsa user@host

# Via console access (if available)
# Use hypervisor console or physical access

# Reset SSH to allow password auth temporarily
sudo sed -i 's/PasswordAuthentication no/PasswordAuthentication yes/' /etc/ssh/sshd_config
sudo systemctl restart sshd

Service Recovery

Critical service restoration:

# Restart all network services
sudo systemctl restart networking
sudo systemctl restart nginx
sudo systemctl restart sshd

# Emergency firewall disable
sudo ufw disable  # CAUTION: Only for troubleshooting

# Service-specific recovery
sudo systemctl restart docker
sudo systemctl restart systemd-resolved

Monitoring and Prevention

Network Health Monitoring

#!/bin/bash
# network-monitor.sh
CRITICAL_HOSTS="10.10.0.1 10.10.0.16 nas.homelab.local"
CRITICAL_SERVICES="https://homelab.local http://proxmox.homelab.local:8006"

for host in $CRITICAL_HOSTS; do
    if ! ping -c1 -W5 $host >/dev/null 2>&1; then
        echo "ALERT: $host unreachable" | logger -t network-monitor
    fi
done

for service in $CRITICAL_SERVICES; do
    if ! curl -sSf --max-time 10 "$service" >/dev/null 2>&1; then
        echo "ALERT: $service unavailable" | logger -t network-monitor
    fi
done

Automated Recovery Scripts

#!/bin/bash
# network-recovery.sh
if ! ping -c1 8.8.8.8 >/dev/null 2>&1; then
    echo "Network down, attempting recovery..."
    sudo systemctl restart networking
    sleep 10
    if ping -c1 8.8.8.8 >/dev/null 2>&1; then
        echo "Network recovered"
    else
        echo "Manual intervention required"
    fi
fi

Quick Reference Commands

Network Diagnostics

# Connectivity tests
ping host
traceroute host
mtr host
nc -zv host port

# Service checks
systemctl status networking
systemctl status nginx
systemctl status sshd

# Network configuration
ip addr show
ip route show
ss -tuln

Emergency Commands

# Network restart
sudo systemctl restart networking

# SSH emergency access
ssh -i ~/.ssh/emergency_homelab_rsa user@host

# Firewall quick disable (emergency only)
sudo ufw disable

# DNS quick fix
echo "nameserver 8.8.8.8" | sudo tee /etc/resolv.conf

This troubleshooting guide provides comprehensive solutions for common networking issues in home lab environments.