# Docker Container Troubleshooting Guide

## Container Startup Issues

### Container Won't Start
**Check container logs first**:
```bash
# Docker
docker logs <container_name>
docker logs --tail 50 -f <container_name>

# Podman  
podman logs <container_name>
podman logs --tail 50 -f <container_name>
```

### Common Startup Failures

#### Port Conflicts
**Symptoms**: `bind: address already in use` error
**Solution**:
```bash
# Find conflicting process
sudo netstat -tulpn | grep <port>
docker ps | grep <port>

# Change port mapping
docker run -p 8081:8080 myapp  # Use different host port
```

#### Permission Errors
**Symptoms**: `permission denied` when accessing files/volumes
**Solutions**:
```bash
# Check file ownership
ls -la /host/volume/path

# Fix ownership (match container user)
sudo chown -R 1000:1000 /host/volume/path

# Use correct UID/GID in container
docker run -e PUID=1000 -e PGID=1000 myapp
```

#### Missing Environment Variables
**Symptoms**: Application fails with configuration errors
**Diagnostic**:
```bash
# Check container environment
docker exec -it <container> env
docker exec -it <container> printenv

# Verify required variables are set
docker inspect <container> | grep -A 20 "Env"
```

#### Resource Constraints
**Symptoms**: Container killed or OOM errors
**Solutions**:
```bash
# Check resource usage
docker stats <container>

# Increase memory limit
docker run -m 4g myapp

# Check system resources
free -h
df -h
```

### Debug Running Containers
```bash
# Access container shell
docker exec -it <container> /bin/bash
docker exec -it <container> /bin/sh  # if bash not available

# Check container processes
docker exec <container> ps aux

# Check container filesystem
docker exec <container> ls -la /app
```

## Build Issues

### Build Failures
**Clear build cache when encountering issues**:
```bash
# Docker
docker system prune -a
docker builder prune

# Podman
podman system prune -a
podman image prune -a
```

### Verbose Build Output
```bash
# Docker
docker build --progress=plain --no-cache .

# Podman  
podman build --layers=false .
```

### Common Build Problems

#### COPY/ADD Errors
**Issue**: Files not found during build
**Solutions**:
```dockerfile
# Check .dockerignore file
# Verify file paths relative to build context
COPY ./src /app/src  # ✅ Correct
COPY /absolute/path /app  # ❌ Wrong - no absolute paths
```

#### Package Installation Failures
**Issue**: apt/yum/dnf package installation fails
**Solutions**:
```dockerfile
# Update package lists first
RUN apt-get update && apt-get install -y package-name

# Combine RUN commands to reduce layers
RUN apt-get update && \
    apt-get install -y package1 package2 && \
    apt-get clean && \
    rm -rf /var/lib/apt/lists/*
```

#### Network Issues During Build
**Issue**: Cannot reach package repositories
**Solutions**:
```bash
# Check DNS resolution
docker build --network host .

# Use custom DNS
docker build --dns 8.8.8.8 .
```

## GPU Container Issues

### NVIDIA GPU Support Problems

#### Docker Desktop vs Podman on Fedora/Nobara
**Issue**: Docker Desktop has GPU compatibility issues on Fedora-based systems
**Symptoms**:
- `CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected`
- `unknown or invalid runtime name: nvidia`
- Device nodes exist but CUDA fails to initialize

**Solution**: Use Podman instead of Docker on Fedora systems
```bash
# Verify host GPU works
nvidia-smi

# Test with Podman (recommended)
podman run --rm --device nvidia.com/gpu=all ubuntu:20.04 nvidia-smi

# Test with Docker (may fail on Fedora)
docker run --rm --gpus all ubuntu:20.04 nvidia-smi
```

#### GPU Container Configuration
**Working Podman GPU template**:
```bash
podman run -d --name gpu-container \
    --device nvidia.com/gpu=all \
    --restart unless-stopped \
    -e NVIDIA_DRIVER_CAPABILITIES=all \
    -e NVIDIA_VISIBLE_DEVICES=all \
    myapp:latest
```

**Working Docker GPU template**:
```bash
docker run -d --name gpu-container \
    --gpus all \
    --restart unless-stopped \
    -e NVIDIA_DRIVER_CAPABILITIES=all \
    -e NVIDIA_VISIBLE_DEVICES=all \
    myapp:latest
```

#### GPU Troubleshooting Steps
1. **Verify Host GPU Access**:
   ```bash
   nvidia-smi                    # Should show GPU info
   lsmod | grep nvidia          # Should show nvidia modules
   ls -la /dev/nvidia*          # Should show device files
   ```

2. **Check NVIDIA Container Toolkit**:
   ```bash
   rpm -qa | grep nvidia-container-toolkit  # Fedora/RHEL
   dpkg -l | grep nvidia-container-toolkit  # Ubuntu/Debian
   nvidia-ctk --version
   ```

3. **Test GPU in Container**:
   ```bash
   # Should show GPU information
   podman exec gpu-container nvidia-smi
   
   # Test CUDA functionality
   podman exec gpu-container nvidia-ml-py
   ```

#### Platform-Specific GPU Notes
**Fedora/Nobara/RHEL**:
- ✅ Podman: Works out-of-the-box with GPU support
- ❌ Docker Desktop: Known GPU integration issues
- Solution: Use Podman for GPU workloads

**Ubuntu/Debian**:
- ✅ Docker: Generally works well with proper NVIDIA toolkit setup
- ✅ Podman: Also works well
- Solution: Either runtime typically works

## Performance Issues

### Resource Monitoring
**Real-time resource usage**:
```bash
# Overall container stats
docker stats
podman stats

# Inside container analysis
docker exec <container> top
docker exec <container> free -h
docker exec <container> df -h

# Network usage
docker exec <container> netstat -i
```

### Image Size Optimization
**Analyze image layers**:
```bash
# Check image sizes
docker images --format "table {{.Repository}}\t{{.Tag}}\t{{.Size}}"

# Analyze layer history
docker history <image>

# Find large files in container
docker exec <container> du -sh /* | sort -hr
```

**Optimization strategies**:
```dockerfile
# Use multi-stage builds
FROM node:18 AS builder
# ... build steps ...

FROM node:18-alpine AS production
COPY --from=builder /app/dist /app
# Smaller final image

# Combine RUN commands
RUN apt-get update && \
    apt-get install -y package && \
    apt-get clean && \
    rm -rf /var/lib/apt/lists/*

# Use .dockerignore
# .dockerignore
node_modules
.git
*.log
```

### Storage Performance Issues
**Slow volume performance**:
```bash
# Test volume I/O performance
docker exec <container> dd if=/dev/zero of=/volume/test bs=1M count=1000

# Check volume mount options
docker inspect <container> | grep -A 10 "Mounts"

# Consider using tmpfs for temporary data
docker run --tmpfs /tmp myapp
```

## Network Debugging

### Network Connectivity Issues
**Inspect network configuration**:
```bash
# List networks
docker network ls
podman network ls

# Inspect specific network
docker network inspect <network_name>

# Check container networking
docker exec <container> ip addr show
docker exec <container> ip route show
```

### Service Discovery Problems
**Test connectivity between containers**:
```bash
# Test by container name (same network)
docker exec container1 ping container2

# Test by IP address
docker exec container1 ping 172.17.0.3

# Check DNS resolution
docker exec container1 nslookup container2
```

### Port Binding Issues
**Verify port mappings**:
```bash
# Check exposed ports
docker port <container>

# Test external connectivity
curl localhost:8080

# Check if port is bound to all interfaces
netstat -tulpn | grep :8080
```

## Emergency Recovery

### Complete Container Reset
**Remove all containers and start fresh**:
```bash
# Stop all containers
docker stop $(docker ps -q)
podman stop --all

# Remove all containers
docker container prune -f
podman container prune -f

# Remove all images
docker image prune -a -f
podman image prune -a -f

# Remove all volumes (CAUTION: data loss)
docker volume prune -f
podman volume prune -f

# Complete system cleanup
docker system prune -a --volumes -f
podman system prune -a --volumes -f
```

### Container Recovery
**Recover from corrupted container**:
```bash
# Create backup of container data
docker cp <container>:/important/data ./backup/

# Export container filesystem
docker export <container> > container-backup.tar

# Import and restart
docker import container-backup.tar new-image:latest
docker run -d --name new-container new-image:latest
```

### Data Recovery
**Recover data from volumes**:
```bash
# List volumes
docker volume ls

# Inspect volume location
docker volume inspect <volume_name>

# Access volume data directly
sudo ls -la /var/lib/docker/volumes/<volume_name>/_data

# Mount volume to temporary container
docker run --rm -v <volume_name>:/data alpine ls -la /data
```

## Health Check Issues

### Container Health Checks
**Implement health checks**:
```dockerfile
# Dockerfile health check
HEALTHCHECK --interval=30s --timeout=3s --start-period=5s --retries=3 \
    CMD curl -f http://localhost:3000/health || exit 1
```

**Debug health check failures**:
```bash
# Check health status
docker inspect <container> | grep -A 10 Health

# Manual health check test
docker exec <container> curl -f http://localhost:3000/health

# Check health check logs
docker events --filter container=<container>
```

## Log Analysis

### Log Management
**View and manage container logs**:
```bash
# View recent logs
docker logs --tail 100 <container>

# Follow logs in real-time
docker logs -f <container>

# Logs with timestamps
docker logs -t <container>

# Search logs for errors
docker logs <container> 2>&1 | grep ERROR
```

### Log Rotation Issues
**Configure log rotation to prevent disk filling**:
```bash
# Run with log size limits
docker run --log-opt max-size=10m --log-opt max-file=3 myapp

# Check log file sizes
sudo du -sh /var/lib/docker/containers/*/
```

## Platform-Specific Issues

### Fedora/Nobara/RHEL Systems
- **GPU Support**: Use Podman instead of Docker Desktop
- **SELinux**: May require container contexts (`-Z` flag)
- **Firewall**: Configure firewalld for container networking

### Ubuntu/Debian Systems  
- **AppArmor**: May restrict container operations
- **Snap Docker**: May have permission issues vs native package

### General Linux Issues
- **cgroups v2**: Some older containers need cgroups v1
- **User namespaces**: May cause UID/GID mapping issues
- **systemd**: Integration differences between Docker/Podman

## Prevention Best Practices

1. **Resource Limits**: Always set memory and CPU limits
2. **Health Checks**: Implement application health monitoring
3. **Log Rotation**: Configure to prevent disk space issues
4. **Security Scanning**: Regular vulnerability scans
5. **Backup Strategy**: Regular data and configuration backups
6. **Testing**: Test containers in staging before production
7. **Documentation**: Document container configurations and dependencies

This troubleshooting guide covers the most common Docker and Podman container issues encountered in home lab and production environments.