major-domo-database/CUSTOM_COMMANDS_MIGRATION.md

# Custom Commands Migration Documentation

## Overview

This document provides complete instructions for migrating existing custom commands from the old `sba_is_fun.db` database to the new custom_commands schema in the Major Domo system. This migration has been developed and tested successfully on local containers and is ready for production deployment.

## Background

### What We're Migrating
- **Source**: `/mnt/NV2/Development/major-domo/sba_is_fun.db` - Contains 140 custom commands and 30 creators
- **Target**: Production `sba_master.db` with new custom_commands schema
- **Scope**: Complete migration of commands, creators, usage statistics, and metadata

### Project Context
The Major Domo system consists of:
- **Discord Bot** - Uses custom commands for user interactions
- **Database API** - FastAPI backend with new custom_commands endpoints
- **Website** - Vue.js frontend (future integration)

## Database Schema Changes

### Old Schema (sba_is_fun.db)
```sql
-- creator table
CREATE TABLE creator (
    id INTEGER PRIMARY KEY,
    name VARCHAR(255) NOT NULL,
    discordid INTEGER NOT NULL
);

-- command table  
CREATE TABLE command (
    id INTEGER PRIMARY KEY,
    name VARCHAR(255) NOT NULL,
    message VARCHAR(255) NOT NULL,
    creator_id INTEGER NOT NULL,
    createtime DATETIME NOT NULL,
    last_used DATETIME NULL,
    sent_warns DATETIME NULL
);
```

### New Schema (sba_master.db)
```sql
-- custom_command_creators table
CREATE TABLE custom_command_creators (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    discord_id INTEGER UNIQUE NOT NULL,
    username TEXT(32) NOT NULL,
    display_name TEXT(32),
    created_at TIMESTAMP NOT NULL,
    total_commands INTEGER DEFAULT 0,
    active_commands INTEGER DEFAULT 0
);

-- custom_commands table
CREATE TABLE custom_commands (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    name TEXT(32) UNIQUE NOT NULL,
    content TEXT NOT NULL,
    creator_id INTEGER NOT NULL,
    created_at TIMESTAMP NOT NULL,
    updated_at TIMESTAMP,
    last_used TIMESTAMP,
    use_count INTEGER DEFAULT 0,
    warning_sent BOOLEAN DEFAULT 0,
    is_active BOOLEAN DEFAULT 1,
    tags TEXT,
    FOREIGN KEY (creator_id) REFERENCES custom_command_creators (id)
);
```

## Code Changes Made

### 1. Database Schema Setup
Tables were created in production using SSH:
```bash
ssh sba-database "docker exec sba_database python -c \"
import sqlite3
conn = sqlite3.connect('/usr/src/app/storage/sba_master.db')
cursor = conn.cursor()
# [Table creation SQL - see above schemas]
\""
```

### 2. FastAPI Integration
**File**: `/mnt/NV2/Development/major-domo/database/app/main.py`

**Changes Made**:
- Line 15: Added `custom_commands` to router imports
- Line 71: Added `app.include_router(custom_commands.router)`

**Fixed Issues**:
- Route ordering: Moved `/{command_id}` route after specific string routes
- SQL result handling: Converted Peewee tuple results to dictionaries
- Peewee model: Removed invalid `max_length` from `TextField` in `db_engine.py:2222`

### 3. Custom Commands Module
**File**: `/mnt/NV2/Development/major-domo/database/app/routers_v3/custom_commands.py`

**Available Endpoints**:
- `GET /api/v3/custom_commands` - List commands with pagination/filtering
- `GET /api/v3/custom_commands/stats` - Command statistics
- `GET /api/v3/custom_commands/autocomplete` - Command name autocomplete  
- `GET /api/v3/custom_commands/by_name/{name}` - Get command by name
- `GET /api/v3/custom_commands/{id}` - Get command by ID
- `POST /api/v3/custom_commands/creators` - Create creator (auth required)
- `POST /api/v3/custom_commands` - Create command (auth required)
- `PATCH /api/v3/custom_commands/{id}` - Update command (auth required)
- `PATCH /api/v3/custom_commands/by_name/{name}/execute` - Execute command (auth required)
- `DELETE /api/v3/custom_commands/{id}` - Delete command (auth required)

## Migration Script

### Location
`/mnt/NV2/Development/major-domo/database/migrate_custom_commands.py`

### Key Features
1. **Validation**: Checks source and target database schemas
2. **Data Mapping**: Maps old schema fields to new schema
3. **Conflict Resolution**: Handles existing data gracefully
4. **Statistics Updates**: Maintains creator command counts
5. **Logging**: Comprehensive logging with timestamps
6. **Dry Run Mode**: Test migrations without making changes
7. **Grace Period**: Updates `last_used` dates to prevent immediate deletion
8. **Migration Tags**: Marks migrated commands with `["migrated"]` tag

### Usage
```bash
# Dry run (recommended first)
python migrate_custom_commands.py \
  --source /path/to/sba_is_fun.db \
  --target /path/to/sba_master.db \
  --dry-run

# Actual migration
python migrate_custom_commands.py \
  --source /path/to/sba_is_fun.db \
  --target /path/to/sba_master.db
```

### Data Transformations
- `creator.name` → `custom_command_creators.username`
- `creator.discordid` → `custom_command_creators.discord_id`
- `command.message` → `custom_commands.content`
- `command.createtime` → `custom_commands.created_at`
- `command.sent_warns` → `custom_commands.warning_sent` (boolean)
- Commands older than 60 days get `last_used` updated to migration date
- All migrated commands get `tags = ["migrated"]`

## Testing Results

### Local Testing Environment
- **Container**: `test-sba-database-v3:local` on port 8001
- **Database**: SQLite with volume mounts
- **API**: http://localhost:8001/api/v3/custom_commands

### Migration Test Results
✅ **Source Data**: 30 creators, 140 commands  
✅ **Migration Success**: 30 creators → 31 total, 140 commands → 141 total  
✅ **Data Integrity**: All relationships preserved  
✅ **API Functionality**: All endpoints working  
✅ **Sample Commands**: `yeli`, `thedata`, `momma`, `charlieblackmon` verified  

### API Test Results
✅ **GET /custom_commands** - Lists all commands with pagination  
✅ **GET /custom_commands/stats** - Shows correct counts and statistics  
✅ **GET /custom_commands/by_name/yeli** - Returns correct command data  
✅ **GET /custom_commands/autocomplete?partial_name=mom** - Returns `["momma"]`  
✅ **Authentication** - Properly blocks unauthorized requests  

## Production Deployment Status

### Completed Steps
1. ✅ **Database Tables Created** - Production tables exist and are functional
2. ✅ **API Endpoints Working** - All custom_commands endpoints operational
3. ✅ **Code Deployed** - FastAPI integration complete
4. ✅ **Migration Script Ready** - Tested and validated locally

### Ready for Production Migration
- [x] Source database accessible: `/mnt/NV2/Development/major-domo/sba_is_fun.db`
- [x] Target database ready: Production `sba_master.db` 
- [x] Migration script tested: `migrate_custom_commands.py`
- [x] API endpoints verified: https://sba.manticorum.com/api/v3/custom_commands
- [x] Error handling confirmed: Proper 404s, authentication, validation

## Production Migration Checklist

### Pre-Migration
- [ ] Copy source database to production server
- [ ] Copy migration script to production server  
- [ ] Verify production API is accessible
- [ ] Run migration in dry-run mode first

### Migration Commands
```bash
# 1. Copy files to production
scp /mnt/NV2/Development/major-domo/sba_is_fun.db sba-database:/tmp/
scp migrate_custom_commands.py sba-database:/tmp/

# 2. SSH to production and copy to container
ssh sba-database
docker cp /tmp/sba_is_fun.db sba_database:/usr/src/app/storage/
docker cp /tmp/migrate_custom_commands.py sba_database:/usr/src/app/storage/

# 3. Run dry-run migration
docker exec sba_database python /usr/src/app/storage/migrate_custom_commands.py \
  --source /usr/src/app/storage/sba_is_fun.db \
  --target /usr/src/app/storage/sba_master.db \
  --dry-run

# 4. Run actual migration
docker exec sba_database python /usr/src/app/storage/migrate_custom_commands.py \
  --source /usr/src/app/storage/sba_is_fun.db \
  --target /usr/src/app/storage/sba_master.db
```

### Post-Migration Validation
- [ ] Check API stats: https://sba.manticorum.com/api/v3/custom_commands/stats
- [ ] Test command lookup: https://sba.manticorum.com/api/v3/custom_commands/by_name/yeli
- [ ] Verify autocomplete: https://sba.manticorum.com/api/v3/custom_commands/autocomplete
- [ ] Check creator statistics in database
- [ ] Test Discord bot integration (if applicable)

## Known Issues & Considerations

### Grace Period Implementation
- Commands not used in 60+ days get `last_used` updated to migration date
- Prevents mass deletion of historical commands
- Gives commands a grace period to be discovered/used again

### Duplicate Handling
- Migration script checks for existing creators by `discord_id`
- Migration script checks for existing commands by `name`
- Existing entries are skipped with warnings logged

### Performance
- 140 commands migrate in ~1 second
- No performance impact on API during migration
- Indexes created for optimal query performance

### Rollback Plan
If migration issues occur:
1. Delete migrated entries: `DELETE FROM custom_commands WHERE tags LIKE '%migrated%'`
2. Delete migrated creators: `DELETE FROM custom_command_creators WHERE id > [original_max_id]`
3. Re-run migration after fixing issues

## Contact Information

**Development Team**: Claude Code AI Assistant  
**Migration Date**: August 17, 2025  
**Production URL**: https://sba.manticorum.com/api/v3/custom_commands  
**Local Test URL**: http://localhost:8001/api/v3/custom_commands  

## File Locations

- **Migration Script**: `/mnt/NV2/Development/major-domo/database/migrate_custom_commands.py`
- **Source Database**: `/mnt/NV2/Development/major-domo/sba_is_fun.db`
- **Custom Commands Module**: `/mnt/NV2/Development/major-domo/database/app/routers_v3/custom_commands.py`
- **Main API File**: `/mnt/NV2/Development/major-domo/database/app/main.py`
- **Database Models**: `/mnt/NV2/Development/major-domo/database/app/db_engine.py`

---

**Status**: Ready for production migration. All testing completed successfully.