claude-home/server-configs/networking/nginx-proxy-manager-pihole.md

# Nginx Proxy Manager + Pi-hole Setup

**Primary Host**: 10.10.0.16 (npm-pihole)
**Secondary Host**: 10.10.0.226 (ubuntu-manticore)
**Services**: Nginx Proxy Manager, Dual Pi-hole DNS (HA)

This deployment uses dual Pi-hole instances across separate physical hosts for high availability DNS, with NPM on the primary host handling reverse proxy duties.

## Quick Info

### Primary Host (npm-pihole)
| Property | Value |
|----------|-------|
| **IP** | 10.10.0.16 |
| **User** | cal |
| **NPM Container** | nginx-proxy-manager_app_1 |
| **Pi-hole Container** | pihole |
| **Pi-hole Web** | http://10.10.0.16/admin |
| **Role** | Primary DNS + Reverse Proxy |

### Secondary Host (ubuntu-manticore)
| Property | Value |
|----------|-------|
| **IP** | 10.10.0.226 |
| **User** | cal |
| **Pi-hole Container** | pihole |
| **Orbital Sync Container** | orbital-sync |
| **Pi-hole Web** | http://10.10.0.226:8053/admin |
| **Role** | Secondary DNS (failover) |

## Services

### Nginx Proxy Manager
- **Container**: nginx-proxy-manager_app_1
- **Web UI**: http://10.10.0.16:81
- **Ports**: 80, 443, 81
- **Database**: SQLite at `/data/database.sqlite`
- **Config**: `/data/nginx/`
- **Logs**: `/data/logs/`

### Pi-hole
- **Container**: pihole
- **Web UI**: http://10.10.0.16/admin
- **Ports**: 53 (DNS), 80 (Web)
- **Config**: `/etc/pihole/`
- **Custom DNS**: `/etc/pihole/custom.list`

## Architecture

### High Availability DNS Architecture

```
┌─────────────────────────────────────────────────────────────┐
│                     UniFi DHCP Server                        │
│            DNS1: 10.10.0.16  DNS2: 10.10.0.226              │
└────────────┬────────────────────────────┬───────────────────┘
             │                            │
             ▼                            ▼
    ┌────────────────┐          ┌────────────────┐
    │  npm-pihole    │          │ ubuntu-        │
    │  10.10.0.16    │◄────────►│ manticore      │
    │                │  Orbital │ 10.10.0.226    │
    │  - NPM         │   Sync   │                │
    │  - Pi-hole 1   │          │ - Jellyfin     │
    │  (Primary)     │          │ - Tdarr        │
    └────────────────┘          │ - Pi-hole 2    │
             ▲                  │  (Secondary)   │
             │                  └────────────────┘
             │
    ┌────────────────┐
    │  NPM DNS Sync  │
    │  (hourly cron) │
    │                │
    │  Syncs proxy   │
    │  hosts to both │
    │  Pi-holes      │
    └────────────────┘
```

### Request Flow

```
┌─────────────┐
│   Clients   │
│ (Homelab)   │
└──────┬──────┘
       │ DNS Query: termix.manticorum.com?
       ▼
┌──────────────────────┐
│  Pi-hole (Primary)   │  ◄── Clients prefer DNS1
│  10.10.0.16          │  ◄── Failover to DNS2 (10.10.0.226) if down
│  Returns: 10.10.0.16 │  ← Local DNS override
└──────┬───────────────┘
       │ HTTPS Request
       ▼
┌──────────────────────────┐
│  NPM (10.10.0.16)        │
│  - Checks access list    │  ← Sees real client IP (10.0.0.206)
│  - Terminates SSL        │
│  - Proxies to backend    │
└──────┬───────────────────┘
       │ HTTP Proxy
       ▼
┌──────────────────────┐
│  Backend Service     │
│  (e.g., Termix)      │
│  10.10.0.210:8180    │
└──────────────────────┘
```

**Benefits**:
- ✅ True high availability: DNS survives single host failure
- ✅ No public DNS fallback: All devices consistently use Pi-hole
- ✅ Automatic synchronization: Blocklists and custom DNS entries sync every 5 minutes
- ✅ Physical separation: Primary (LXC) and secondary (physical server) on different hosts
- ✅ Fixes iOS 403 errors: No more encrypted DNS bypass to public IPs

## DNS Synchronization

### NPM → Pi-hole DNS Sync

#### Problem Solved
When accessing homelab services by domain name (e.g., `termix.manticorum.com`), DNS would resolve to Cloudflare IPs, causing traffic to hairpin through the internet. This broke IP-based access lists since NPM would see your public IP instead of your internal IP.

#### Solution
Automatically sync all NPM proxy hosts to **both Pi-holes'** local DNS, pointing them to NPM's internal IP (10.10.0.16).

#### Sync Script

**Location**: `/home/cal/scripts/npm-pihole-sync.sh` (on 10.10.0.16)
**Repository**: `server-configs/networking/scripts/npm-pihole-sync.sh`

The script:
1. Queries NPM's SQLite database for all enabled proxy hosts
2. Extracts domain names
3. Creates Pi-hole custom DNS entries pointing to NPM (10.10.0.16)
4. Syncs to **primary Pi-hole** (local Docker container)
5. Syncs to **secondary Pi-hole** (via SSH to ubuntu-manticore)
6. Reloads DNS on both Pi-holes

**Usage:**
```bash
# Dry run (preview changes)
ssh cal@10.10.0.16 /home/cal/scripts/npm-pihole-sync.sh --dry-run

# Apply changes to both Pi-holes
ssh cal@10.10.0.16 /home/cal/scripts/npm-pihole-sync.sh
```

**Automation:**
- Runs hourly via cron: `0 * * * * /home/cal/scripts/npm-pihole-sync.sh >> /home/cal/logs/npm-pihole-sync.log 2>&1`
- Logs: `/home/cal/logs/npm-pihole-sync.log`

**Deployment:**
```bash
# Copy script to server
scp server-configs/networking/scripts/npm-pihole-sync.sh cal@10.10.0.16:/home/cal/scripts/

# Make executable
ssh cal@10.10.0.16 "chmod +x /home/cal/scripts/npm-pihole-sync.sh"

# Test (shows what would be synced to both Pi-holes)
ssh cal@10.10.0.16 "/home/cal/scripts/npm-pihole-sync.sh --dry-run"

# Apply
ssh cal@10.10.0.16 "/home/cal/scripts/npm-pihole-sync.sh"

# Verify both Pi-holes have the entries
ssh cal@10.10.0.16 "docker exec pihole cat /etc/pihole/custom.list | grep termix"
ssh ubuntu-manticore "docker exec pihole cat /etc/pihole/custom.list | grep termix"
```

### Pi-hole → Pi-hole Sync (Orbital Sync)

#### Purpose
Synchronizes blocklists, whitelists, custom DNS entries, and all Pi-hole configuration from primary to secondary Pi-hole for consistent ad blocking and DNS behavior.

#### Technology
- **Orbital Sync**: Modern replacement for deprecated Gravity Sync
- **Method**: Uses Pi-hole's official Teleporter API (backup/restore)
- **Interval**: Every 5 minutes
- **Location**: ubuntu-manticore (co-located with secondary Pi-hole)

#### What Gets Synced
- ✅ Blocklists (adlists)
- ✅ Whitelists
- ✅ Regex blacklists/whitelists
- ✅ Custom DNS entries (local DNS records)
- ✅ CNAME records
- ✅ Client groups
- ✅ Audit log
- ❌ DHCP leases (not using Pi-hole for DHCP)

#### Management
```bash
# Check Orbital Sync status
ssh ubuntu-manticore "docker ps | grep orbital-sync"

# View sync logs
ssh ubuntu-manticore "docker logs orbital-sync --tail 50"

# Force immediate sync (restart container)
ssh ubuntu-manticore "cd ~/docker/orbital-sync && docker compose restart"

# Monitor sync in real-time
ssh ubuntu-manticore "docker logs orbital-sync -f"
```

#### Configuration
**Location**: `~/docker/orbital-sync/.env` on ubuntu-manticore

```bash
PRIMARY_HOST_PASSWORD=<api_token_from_primary_pihole>
SECONDARY_HOST_PASSWORD=<api_token_from_secondary_pihole>
```

**To regenerate app passwords (Pi-hole v6):**
1. Primary: http://10.10.0.16:81/admin → Settings → Web Interface / API → Configure app password
2. Secondary: http://10.10.0.226:8053/admin → Settings → Web Interface / API → Configure app password
3. Update `~/.claude/secrets/pihole1_app_password` and `pihole2_app_password`
4. Update `.env` file on ubuntu-manticore
5. Restart orbital-sync: `cd ~/docker/orbital-sync && docker compose restart`

## Cloudflare Real IP Configuration

To support external access through Cloudflare while maintaining proper IP detection for access lists, NPM is configured to trust Cloudflare's IP ranges and read the real client IP from the `CF-Connecting-IP` header.

**Configuration** (in `/etc/nginx/nginx.conf` inside NPM container):

```nginx
# Trust internal networks
set_real_ip_from 10.0.0.0/8;
set_real_ip_from 172.16.0.0/12;
set_real_ip_from 192.168.0.0/16;

# Trust Cloudflare IPs
set_real_ip_from 173.245.48.0/20;
set_real_ip_from 103.21.244.0/22;
set_real_ip_from 103.22.200.0/22;
set_real_ip_from 103.31.4.0/22;
set_real_ip_from 141.101.64.0/18;
set_real_ip_from 108.162.192.0/18;
set_real_ip_from 190.93.240.0/20;
set_real_ip_from 188.114.96.0/20;
set_real_ip_from 197.234.240.0/22;
set_real_ip_from 198.41.128.0/17;
set_real_ip_from 162.158.0.0/15;
set_real_ip_from 104.16.0.0/13;
set_real_ip_from 104.24.0.0/14;
set_real_ip_from 172.64.0.0/13;
set_real_ip_from 131.0.72.0/22;

# Read real IP from Cloudflare header
real_ip_header CF-Connecting-IP;
real_ip_recursive on;
```

This ensures:
- **Local access**: Client IP seen correctly (e.g., 10.0.0.206)
- **External access via Cloudflare**: Real public IP seen (not Cloudflare's proxy IP)

## Access Lists

NPM supports IP-based access restrictions. Common patterns:

### Internal Only
Allows access only from local networks:
```
allow 10.0.0.0/24;
allow 10.10.0.0/24;
deny all;
```

Use this for:
- Internal services (Termix, n8n, OmniTools)
- Admin interfaces
- Development environments

### Cloudflare Only
Trusts Cloudflare's authenticated access (Cloudflare Access/Zero Trust):
- No IP restrictions needed
- Cloudflare handles authentication
- Use for public services behind Cloudflare Access

## Management

### Access Containers
```bash
# SSH to host
ssh cal@10.10.0.16

# Access NPM container
docker exec -it nginx-proxy-manager_app_1 bash

# Access Pi-hole container
docker exec -it pihole bash
```

### NPM Commands
```bash
# View proxy host configs
docker exec nginx-proxy-manager_app_1 ls /data/nginx/proxy_host/

# View specific proxy host
docker exec nginx-proxy-manager_app_1 cat /data/nginx/proxy_host/21.conf

# Test nginx config
docker exec nginx-proxy-manager_app_1 nginx -t

# Reload nginx
docker exec nginx-proxy-manager_app_1 nginx -s reload

# View access logs
docker exec nginx-proxy-manager_app_1 tail -f /data/logs/proxy-host-21_access.log
```

### Pi-hole Commands
```bash
# View custom DNS
docker exec pihole cat /etc/pihole/custom.list

# Restart DNS
docker exec pihole pihole restartdns reload

# View logs
docker logs pihole -f

# Pi-hole status
docker exec pihole pihole status
```

### Query NPM Database
```bash
# List all proxy hosts
docker exec nginx-proxy-manager_app_1 python3 -c "
import sqlite3
conn = sqlite3.connect('/data/database.sqlite')
cursor = conn.cursor()
cursor.execute('SELECT id, domain_names, forward_host, forward_port FROM proxy_host')
for row in cursor.fetchall():
    print(row)
conn.close()
"
```

## Troubleshooting

### 403 Forbidden Errors

**Symptom**: Getting 403 errors when accessing services with "Internal Only" access list.

**Causes**:
1. **Hairpinning through Cloudflare**: DNS resolving to Cloudflare IPs instead of local
   - **Fix**: Ensure Pi-hole has local DNS override (run sync script)
   - **Check**: `dig +short service.manticorum.com @10.10.0.16` should return `10.10.0.16`

2. **Access list doesn't include your subnet**: Your IP not in allowed ranges
   - **Fix**: Add your subnet to access list (e.g., `10.0.0.0/24`)
   - **Check**: View access logs to see what IP NPM is seeing

3. **Real IP not detected**: NPM seeing proxy/Cloudflare IP instead of client IP
   - **Fix**: Ensure Cloudflare IPs are in `set_real_ip_from` and `real_ip_header CF-Connecting-IP` is set
   - **Check**: Access logs should show your real IP, not Cloudflare's

**Debug Steps**:
```bash
# Check what IP NPM sees
docker exec nginx-proxy-manager_app_1 tail -20 /data/logs/proxy-host-21_access.log

# Check DNS resolution
dig +short termix.manticorum.com @10.10.0.16

# Check access list in proxy host config
docker exec nginx-proxy-manager_app_1 cat /data/nginx/proxy_host/21.conf | grep -A 10 "Access Rules"
```

### DNS Not Resolving Locally

**Symptom**: Domains still resolve to Cloudflare IPs.

**Causes**:
1. DNS sync not run or failed
2. Desktop not using Pi-hole for DNS
3. DNS cache on desktop

**Fix**:
```bash
# Re-run sync
ssh cal@10.10.0.16 /home/cal/scripts/npm-pihole-sync.sh

# Check Pi-hole has the record
ssh cal@10.10.0.16 "docker exec pihole cat /etc/pihole/custom.list | grep termix"

# Verify DNS server on desktop
cat /etc/resolv.conf  # Should show 10.10.0.16

# Flush DNS cache on desktop
sudo resolvectl flush-caches  # SystemD
# or
sudo systemd-resolve --flush-caches
```

## Related Documentation

- [Gitea Server](../gitea/README.md) - Uses NPM proxy
- [n8n Automation](../../productivity/n8n/CONTEXT.md) - Uses NPM proxy
- [Akamai NPM](../akamai/docker-compose/nginx-proxy-manager/) - Public-facing NPM instance

## Deployment Date

**Created**: 2026-02-04
**By**: Claude Code
**NPM Version**: (check with `docker exec nginx-proxy-manager_app_1 cat /app/package.json | grep version`)
**Pi-hole Version**: (check with `docker exec pihole pihole -v`)