Add NPM + Pi-hole DNS sync automation and documentation
## Changes - Created comprehensive NPM + Pi-hole setup documentation - Added DNS sync script to automatically sync NPM proxy hosts to Pi-hole - Updated hosts.yml with npm-pihole host entry ## Features - Automatic local DNS resolution for homelab services - Fixes 403 errors with "Internal Only" access lists - Hourly cron sync keeps Pi-hole updated with NPM changes - Cloudflare real IP configuration for proper IP detection ## Files Added - server-configs/networking/nginx-proxy-manager-pihole.md - server-configs/networking/scripts/npm-pihole-sync.sh ## Files Modified - server-configs/hosts.yml (added npm-pihole host) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
This commit is contained in:
Cal Corum
parent
7eadacc6db
commit
2017b56985
@ -110,6 +110,7 @@ hosts:
|
||||
services:
|
||||
- n8n
|
||||
- omni-tools
|
||||
- termix
|
||||
|
||||
# Foundry VTT LXC (Proxmox)
|
||||
foundry-lxc:
|
||||
@ -187,6 +188,23 @@ hosts:
|
||||
- major-domo-dev
|
||||
- strat-gameplay-webapp
|
||||
|
||||
# NPM + Pi-hole Host
|
||||
npm-pihole:
|
||||
type: docker
|
||||
ssh_alias: npm-pihole
|
||||
ip: 10.10.0.16
|
||||
user: cal
|
||||
description: "Nginx Proxy Manager + Pi-hole DNS server"
|
||||
config_paths:
|
||||
scripts: /home/cal/scripts
|
||||
services:
|
||||
- nginx-proxy-manager
|
||||
- pihole
|
||||
documentation: server-configs/networking/nginx-proxy-manager-pihole.md
|
||||
automation:
|
||||
dns_sync: /home/cal/scripts/npm-pihole-sync.sh
|
||||
cron: "0 * * * *" # Hourly
|
||||
|
||||
# Akamai Cloud Server
|
||||
akamai:
|
||||
type: docker
|
||||
|
||||
302
server-configs/networking/nginx-proxy-manager-pihole.md
Normal file
302
server-configs/networking/nginx-proxy-manager-pihole.md
Normal file
@ -0,0 +1,302 @@
|
||||
# Nginx Proxy Manager + Pi-hole Setup
|
||||
|
||||
**Host**: 10.10.0.16
|
||||
**Services**: Nginx Proxy Manager, Pi-hole DNS
|
||||
|
||||
This host runs both NPM (reverse proxy) and Pi-hole (DNS server) as Docker containers.
|
||||
|
||||
## Quick Info
|
||||
|
||||
| Property | Value |
|
||||
|----------|-------|
|
||||
| **IP** | 10.10.0.16 |
|
||||
| **User** | cal |
|
||||
| **NPM Container** | nginx-proxy-manager_app_1 |
|
||||
| **Pi-hole Container** | pihole |
|
||||
| **Pi-hole Web** | http://10.10.0.16/admin |
|
||||
|
||||
## Services
|
||||
|
||||
### Nginx Proxy Manager
|
||||
- **Container**: nginx-proxy-manager_app_1
|
||||
- **Web UI**: http://10.10.0.16:81
|
||||
- **Ports**: 80, 443, 81
|
||||
- **Database**: SQLite at `/data/database.sqlite`
|
||||
- **Config**: `/data/nginx/`
|
||||
- **Logs**: `/data/logs/`
|
||||
|
||||
### Pi-hole
|
||||
- **Container**: pihole
|
||||
- **Web UI**: http://10.10.0.16/admin
|
||||
- **Ports**: 53 (DNS), 80 (Web)
|
||||
- **Config**: `/etc/pihole/`
|
||||
- **Custom DNS**: `/etc/pihole/custom.list`
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
┌─────────────┐
|
||||
│ Clients │
|
||||
│ (Homelab) │
|
||||
└──────┬──────┘
|
||||
│ DNS Query: termix.manticorum.com?
|
||||
▼
|
||||
┌─────────────────────┐
|
||||
│ Pi-hole (10.10.0.16)│
|
||||
│ Returns: 10.10.0.16 │ ← Local DNS override
|
||||
└──────┬──────────────┘
|
||||
│ HTTPS Request
|
||||
▼
|
||||
┌──────────────────────────┐
|
||||
│ NPM (10.10.0.16) │
|
||||
│ - Checks access list │ ← Sees real client IP (10.0.0.206)
|
||||
│ - Terminates SSL │
|
||||
│ - Proxies to backend │
|
||||
└──────┬───────────────────┘
|
||||
│ HTTP Proxy
|
||||
▼
|
||||
┌──────────────────────┐
|
||||
│ Backend Service │
|
||||
│ (e.g., Termix) │
|
||||
│ 10.10.0.210:8180 │
|
||||
└──────────────────────┘
|
||||
```
|
||||
|
||||
## DNS Sync Automation
|
||||
|
||||
### Problem Solved
|
||||
When accessing homelab services by domain name (e.g., `termix.manticorum.com`), DNS would resolve to Cloudflare IPs, causing traffic to hairpin through the internet. This broke IP-based access lists since NPM would see your public IP instead of your internal IP.
|
||||
|
||||
### Solution
|
||||
Automatically sync all NPM proxy hosts to Pi-hole's local DNS, pointing them to NPM's internal IP (10.10.0.16).
|
||||
|
||||
### Sync Script
|
||||
|
||||
**Location**: `/home/cal/scripts/npm-pihole-sync.sh` (on 10.10.0.16)
|
||||
**Repository**: `server-configs/networking/scripts/npm-pihole-sync.sh`
|
||||
|
||||
The script:
|
||||
1. Queries NPM's SQLite database for all enabled proxy hosts
|
||||
2. Extracts domain names
|
||||
3. Creates Pi-hole custom DNS entries pointing to NPM (10.10.0.16)
|
||||
4. Reloads Pi-hole DNS
|
||||
|
||||
**Usage:**
|
||||
```bash
|
||||
# Dry run (preview changes)
|
||||
ssh cal@10.10.0.16 /home/cal/scripts/npm-pihole-sync.sh --dry-run
|
||||
|
||||
# Apply changes
|
||||
ssh cal@10.10.0.16 /home/cal/scripts/npm-pihole-sync.sh
|
||||
```
|
||||
|
||||
**Automation:**
|
||||
- Runs hourly via cron: `0 * * * * /home/cal/scripts/npm-pihole-sync.sh >> /home/cal/logs/npm-pihole-sync.log 2>&1`
|
||||
- Logs: `/home/cal/logs/npm-pihole-sync.log`
|
||||
|
||||
**Deployment:**
|
||||
```bash
|
||||
# Copy script to server
|
||||
scp server-configs/networking/scripts/npm-pihole-sync.sh cal@10.10.0.16:/home/cal/scripts/
|
||||
|
||||
# Make executable
|
||||
ssh cal@10.10.0.16 "chmod +x /home/cal/scripts/npm-pihole-sync.sh"
|
||||
|
||||
# Test
|
||||
ssh cal@10.10.0.16 "/home/cal/scripts/npm-pihole-sync.sh --dry-run"
|
||||
|
||||
# Apply
|
||||
ssh cal@10.10.0.16 "/home/cal/scripts/npm-pihole-sync.sh"
|
||||
```
|
||||
|
||||
## Cloudflare Real IP Configuration
|
||||
|
||||
To support external access through Cloudflare while maintaining proper IP detection for access lists, NPM is configured to trust Cloudflare's IP ranges and read the real client IP from the `CF-Connecting-IP` header.
|
||||
|
||||
**Configuration** (in `/etc/nginx/nginx.conf` inside NPM container):
|
||||
|
||||
```nginx
|
||||
# Trust internal networks
|
||||
set_real_ip_from 10.0.0.0/8;
|
||||
set_real_ip_from 172.16.0.0/12;
|
||||
set_real_ip_from 192.168.0.0/16;
|
||||
|
||||
# Trust Cloudflare IPs
|
||||
set_real_ip_from 173.245.48.0/20;
|
||||
set_real_ip_from 103.21.244.0/22;
|
||||
set_real_ip_from 103.22.200.0/22;
|
||||
set_real_ip_from 103.31.4.0/22;
|
||||
set_real_ip_from 141.101.64.0/18;
|
||||
set_real_ip_from 108.162.192.0/18;
|
||||
set_real_ip_from 190.93.240.0/20;
|
||||
set_real_ip_from 188.114.96.0/20;
|
||||
set_real_ip_from 197.234.240.0/22;
|
||||
set_real_ip_from 198.41.128.0/17;
|
||||
set_real_ip_from 162.158.0.0/15;
|
||||
set_real_ip_from 104.16.0.0/13;
|
||||
set_real_ip_from 104.24.0.0/14;
|
||||
set_real_ip_from 172.64.0.0/13;
|
||||
set_real_ip_from 131.0.72.0/22;
|
||||
|
||||
# Read real IP from Cloudflare header
|
||||
real_ip_header CF-Connecting-IP;
|
||||
real_ip_recursive on;
|
||||
```
|
||||
|
||||
This ensures:
|
||||
- **Local access**: Client IP seen correctly (e.g., 10.0.0.206)
|
||||
- **External access via Cloudflare**: Real public IP seen (not Cloudflare's proxy IP)
|
||||
|
||||
## Access Lists
|
||||
|
||||
NPM supports IP-based access restrictions. Common patterns:
|
||||
|
||||
### Internal Only
|
||||
Allows access only from local networks:
|
||||
```
|
||||
allow 10.0.0.0/24;
|
||||
allow 10.10.0.0/24;
|
||||
deny all;
|
||||
```
|
||||
|
||||
Use this for:
|
||||
- Internal services (Termix, n8n, OmniTools)
|
||||
- Admin interfaces
|
||||
- Development environments
|
||||
|
||||
### Cloudflare Only
|
||||
Trusts Cloudflare's authenticated access (Cloudflare Access/Zero Trust):
|
||||
- No IP restrictions needed
|
||||
- Cloudflare handles authentication
|
||||
- Use for public services behind Cloudflare Access
|
||||
|
||||
## Management
|
||||
|
||||
### Access Containers
|
||||
```bash
|
||||
# SSH to host
|
||||
ssh cal@10.10.0.16
|
||||
|
||||
# Access NPM container
|
||||
docker exec -it nginx-proxy-manager_app_1 bash
|
||||
|
||||
# Access Pi-hole container
|
||||
docker exec -it pihole bash
|
||||
```
|
||||
|
||||
### NPM Commands
|
||||
```bash
|
||||
# View proxy host configs
|
||||
docker exec nginx-proxy-manager_app_1 ls /data/nginx/proxy_host/
|
||||
|
||||
# View specific proxy host
|
||||
docker exec nginx-proxy-manager_app_1 cat /data/nginx/proxy_host/21.conf
|
||||
|
||||
# Test nginx config
|
||||
docker exec nginx-proxy-manager_app_1 nginx -t
|
||||
|
||||
# Reload nginx
|
||||
docker exec nginx-proxy-manager_app_1 nginx -s reload
|
||||
|
||||
# View access logs
|
||||
docker exec nginx-proxy-manager_app_1 tail -f /data/logs/proxy-host-21_access.log
|
||||
```
|
||||
|
||||
### Pi-hole Commands
|
||||
```bash
|
||||
# View custom DNS
|
||||
docker exec pihole cat /etc/pihole/custom.list
|
||||
|
||||
# Restart DNS
|
||||
docker exec pihole pihole restartdns reload
|
||||
|
||||
# View logs
|
||||
docker logs pihole -f
|
||||
|
||||
# Pi-hole status
|
||||
docker exec pihole pihole status
|
||||
```
|
||||
|
||||
### Query NPM Database
|
||||
```bash
|
||||
# List all proxy hosts
|
||||
docker exec nginx-proxy-manager_app_1 python3 -c "
|
||||
import sqlite3
|
||||
conn = sqlite3.connect('/data/database.sqlite')
|
||||
cursor = conn.cursor()
|
||||
cursor.execute('SELECT id, domain_names, forward_host, forward_port FROM proxy_host')
|
||||
for row in cursor.fetchall():
|
||||
print(row)
|
||||
conn.close()
|
||||
"
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### 403 Forbidden Errors
|
||||
|
||||
**Symptom**: Getting 403 errors when accessing services with "Internal Only" access list.
|
||||
|
||||
**Causes**:
|
||||
1. **Hairpinning through Cloudflare**: DNS resolving to Cloudflare IPs instead of local
|
||||
- **Fix**: Ensure Pi-hole has local DNS override (run sync script)
|
||||
- **Check**: `dig +short service.manticorum.com @10.10.0.16` should return `10.10.0.16`
|
||||
|
||||
2. **Access list doesn't include your subnet**: Your IP not in allowed ranges
|
||||
- **Fix**: Add your subnet to access list (e.g., `10.0.0.0/24`)
|
||||
- **Check**: View access logs to see what IP NPM is seeing
|
||||
|
||||
3. **Real IP not detected**: NPM seeing proxy/Cloudflare IP instead of client IP
|
||||
- **Fix**: Ensure Cloudflare IPs are in `set_real_ip_from` and `real_ip_header CF-Connecting-IP` is set
|
||||
- **Check**: Access logs should show your real IP, not Cloudflare's
|
||||
|
||||
**Debug Steps**:
|
||||
```bash
|
||||
# Check what IP NPM sees
|
||||
docker exec nginx-proxy-manager_app_1 tail -20 /data/logs/proxy-host-21_access.log
|
||||
|
||||
# Check DNS resolution
|
||||
dig +short termix.manticorum.com @10.10.0.16
|
||||
|
||||
# Check access list in proxy host config
|
||||
docker exec nginx-proxy-manager_app_1 cat /data/nginx/proxy_host/21.conf | grep -A 10 "Access Rules"
|
||||
```
|
||||
|
||||
### DNS Not Resolving Locally
|
||||
|
||||
**Symptom**: Domains still resolve to Cloudflare IPs.
|
||||
|
||||
**Causes**:
|
||||
1. DNS sync not run or failed
|
||||
2. Desktop not using Pi-hole for DNS
|
||||
3. DNS cache on desktop
|
||||
|
||||
**Fix**:
|
||||
```bash
|
||||
# Re-run sync
|
||||
ssh cal@10.10.0.16 /home/cal/scripts/npm-pihole-sync.sh
|
||||
|
||||
# Check Pi-hole has the record
|
||||
ssh cal@10.10.0.16 "docker exec pihole cat /etc/pihole/custom.list | grep termix"
|
||||
|
||||
# Verify DNS server on desktop
|
||||
cat /etc/resolv.conf # Should show 10.10.0.16
|
||||
|
||||
# Flush DNS cache on desktop
|
||||
sudo resolvectl flush-caches # SystemD
|
||||
# or
|
||||
sudo systemd-resolve --flush-caches
|
||||
```
|
||||
|
||||
## Related Documentation
|
||||
|
||||
- [Gitea Server](../gitea/README.md) - Uses NPM proxy
|
||||
- [n8n Automation](../../productivity/n8n/CONTEXT.md) - Uses NPM proxy
|
||||
- [Akamai NPM](../akamai/docker-compose/nginx-proxy-manager/) - Public-facing NPM instance
|
||||
|
||||
## Deployment Date
|
||||
|
||||
**Created**: 2026-02-04
|
||||
**By**: Claude Code
|
||||
**NPM Version**: (check with `docker exec nginx-proxy-manager_app_1 cat /app/package.json | grep version`)
|
||||
**Pi-hole Version**: (check with `docker exec pihole pihole -v`)
|
||||
74
server-configs/networking/scripts/npm-pihole-sync.sh
Executable file
74
server-configs/networking/scripts/npm-pihole-sync.sh
Executable file
@ -0,0 +1,74 @@
|
||||
#!/bin/bash
|
||||
# NPM to Pi-hole DNS Sync
|
||||
# Syncs Nginx Proxy Manager proxy hosts to Pi-hole local DNS
|
||||
# All domains point to NPM's IP, not the forward destination
|
||||
|
||||
set -e
|
||||
|
||||
DRY_RUN=false
|
||||
if [[ "$1" == "--dry-run" ]]; then
|
||||
DRY_RUN=true
|
||||
fi
|
||||
|
||||
# NPM's IP address (where all domains should point)
|
||||
NPM_IP="10.10.0.16"
|
||||
|
||||
echo "NPM → Pi-hole DNS Sync"
|
||||
echo "============================================================"
|
||||
|
||||
# Query NPM database for all enabled proxy hosts
|
||||
DOMAINS=$(docker exec nginx-proxy-manager_app_1 python3 -c '
|
||||
import sqlite3
|
||||
import json
|
||||
|
||||
conn = sqlite3.connect("/data/database.sqlite")
|
||||
cursor = conn.cursor()
|
||||
cursor.execute("SELECT domain_names FROM proxy_host WHERE enabled = 1")
|
||||
|
||||
domains = []
|
||||
for (domain_names,) in cursor.fetchall():
|
||||
for domain in json.loads(domain_names or "[]"):
|
||||
domains.append(domain)
|
||||
|
||||
for domain in sorted(domains):
|
||||
print(domain)
|
||||
|
||||
conn.close()
|
||||
')
|
||||
|
||||
# Count records
|
||||
RECORD_COUNT=$(echo "$DOMAINS" | wc -l)
|
||||
echo "Found $RECORD_COUNT enabled proxy hosts"
|
||||
echo ""
|
||||
echo "All domains will point to NPM at: $NPM_IP"
|
||||
echo ""
|
||||
echo "Domains to sync:"
|
||||
echo "$DOMAINS" | awk -v ip="$NPM_IP" '{printf " %-15s %s\n", ip, $0}'
|
||||
|
||||
if [ "$DRY_RUN" = true ]; then
|
||||
echo ""
|
||||
echo "[DRY RUN] Not applying changes"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# Build new custom.list
|
||||
NEW_DNS="# Pi-hole Local DNS Records
|
||||
# Auto-synced from Nginx Proxy Manager
|
||||
# All domains point to NPM at $NPM_IP
|
||||
|
||||
"
|
||||
|
||||
while IFS= read -r domain; do
|
||||
NEW_DNS+="$NPM_IP $domain"$'\n'
|
||||
done <<< "$DOMAINS"
|
||||
|
||||
# Write to Pi-hole
|
||||
echo "$NEW_DNS" | docker exec -i pihole tee /etc/pihole/custom.list > /dev/null
|
||||
|
||||
# Reload Pi-hole DNS
|
||||
docker exec pihole pihole restartdns reload > /dev/null
|
||||
|
||||
echo ""
|
||||
echo "✓ Updated $RECORD_COUNT DNS records in Pi-hole"
|
||||
echo "✓ All domains now point to NPM at $NPM_IP"
|
||||
echo "✓ Reloaded Pi-hole DNS"
|
||||
Loading…
Reference in New Issue
Block a user