major-domo-database/CUSTOM_COMMANDS_MIGRATION.md

Custom Commands Migration Documentation

Overview

This document provides complete instructions for migrating existing custom commands from the old sba_is_fun.db database to the new custom_commands schema in the Major Domo system. This migration has been developed and tested successfully on local containers and is ready for production deployment.

Background

What We're Migrating

• Source: /mnt/NV2/Development/major-domo/sba_is_fun.db - Contains 140 custom commands and 30 creators
• Target: Production sba_master.db with new custom_commands schema
• Scope: Complete migration of commands, creators, usage statistics, and metadata

Project Context

The Major Domo system consists of:

• Discord Bot - Uses custom commands for user interactions
• Database API - FastAPI backend with new custom_commands endpoints
• Website - Vue.js frontend (future integration)

Database Schema Changes

Old Schema (sba_is_fun.db)

-- creator table
CREATE TABLE creator (
    id INTEGER PRIMARY KEY,
    name VARCHAR(255) NOT NULL,
    discordid INTEGER NOT NULL
);

-- command table  
CREATE TABLE command (
    id INTEGER PRIMARY KEY,
    name VARCHAR(255) NOT NULL,
    message VARCHAR(255) NOT NULL,
    creator_id INTEGER NOT NULL,
    createtime DATETIME NOT NULL,
    last_used DATETIME NULL,
    sent_warns DATETIME NULL
);

New Schema (sba_master.db)

-- custom_command_creators table
CREATE TABLE custom_command_creators (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    discord_id INTEGER UNIQUE NOT NULL,
    username TEXT(32) NOT NULL,
    display_name TEXT(32),
    created_at TIMESTAMP NOT NULL,
    total_commands INTEGER DEFAULT 0,
    active_commands INTEGER DEFAULT 0
);

-- custom_commands table
CREATE TABLE custom_commands (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    name TEXT(32) UNIQUE NOT NULL,
    content TEXT NOT NULL,
    creator_id INTEGER NOT NULL,
    created_at TIMESTAMP NOT NULL,
    updated_at TIMESTAMP,
    last_used TIMESTAMP,
    use_count INTEGER DEFAULT 0,
    warning_sent BOOLEAN DEFAULT 0,
    is_active BOOLEAN DEFAULT 1,
    tags TEXT,
    FOREIGN KEY (creator_id) REFERENCES custom_command_creators (id)
);

Code Changes Made

1. Database Schema Setup

Tables were created in production using SSH:

ssh sba-database "docker exec sba_database python -c \"
import sqlite3
conn = sqlite3.connect('/usr/src/app/storage/sba_master.db')
cursor = conn.cursor()
# [Table creation SQL - see above schemas]
\""

2. FastAPI Integration

File: /mnt/NV2/Development/major-domo/database/app/main.py

Changes Made:

• Line 15: Added custom_commands to router imports
• Line 71: Added app.include_router(custom_commands.router)

Fixed Issues:

• Route ordering: Moved /{command_id} route after specific string routes
• SQL result handling: Converted Peewee tuple results to dictionaries
• Peewee model: Removed invalid max_length from TextField in db_engine.py:2222

3. Custom Commands Module

File: /mnt/NV2/Development/major-domo/database/app/routers_v3/custom_commands.py

Available Endpoints:

• GET /api/v3/custom_commands - List commands with pagination/filtering
• GET /api/v3/custom_commands/stats - Command statistics
• GET /api/v3/custom_commands/autocomplete - Command name autocomplete
• GET /api/v3/custom_commands/by_name/{name} - Get command by name
• GET /api/v3/custom_commands/{id} - Get command by ID
• POST /api/v3/custom_commands/creators - Create creator (auth required)
• POST /api/v3/custom_commands - Create command (auth required)
• PATCH /api/v3/custom_commands/{id} - Update command (auth required)
• PATCH /api/v3/custom_commands/by_name/{name}/execute - Execute command (auth required)
• DELETE /api/v3/custom_commands/{id} - Delete command (auth required)

Migration Script

Location

/mnt/NV2/Development/major-domo/database/migrate_custom_commands.py

Key Features

1. Validation: Checks source and target database schemas
2. Data Mapping: Maps old schema fields to new schema
3. Conflict Resolution: Handles existing data gracefully
4. Statistics Updates: Maintains creator command counts
5. Logging: Comprehensive logging with timestamps
6. Dry Run Mode: Test migrations without making changes
7. Grace Period: Updates last_used dates to prevent immediate deletion
8. Migration Tags: Marks migrated commands with ["migrated"] tag

Usage

# Dry run (recommended first)
python migrate_custom_commands.py \
  --source /path/to/sba_is_fun.db \
  --target /path/to/sba_master.db \
  --dry-run

# Actual migration
python migrate_custom_commands.py \
  --source /path/to/sba_is_fun.db \
  --target /path/to/sba_master.db

Data Transformations

• creator.name → custom_command_creators.username
• creator.discordid → custom_command_creators.discord_id
• command.message → custom_commands.content
• command.createtime → custom_commands.created_at
• command.sent_warns → custom_commands.warning_sent (boolean)
• Commands older than 60 days get last_used updated to migration date
• All migrated commands get tags = ["migrated"]

Testing Results

Local Testing Environment

• Container: test-sba-database-v3:local on port 8001
• Database: SQLite with volume mounts
• API: http://localhost:8001/api/v3/custom_commands

Migration Test Results

✅ Source Data: 30 creators, 140 commands
✅ Migration Success: 30 creators → 31 total, 140 commands → 141 total
✅ Data Integrity: All relationships preserved
✅ API Functionality: All endpoints working
✅ Sample Commands: yeli, thedata, momma, charlieblackmon verified

API Test Results

✅ GET /custom_commands - Lists all commands with pagination
✅ GET /custom_commands/stats - Shows correct counts and statistics
✅ GET /custom_commands/by_name/yeli - Returns correct command data
✅ GET /custom_commands/autocomplete?partial_name=mom - Returns ["momma"]
✅ Authentication - Properly blocks unauthorized requests

Production Deployment Status

Completed Steps

1. ✅ Database Tables Created - Production tables exist and are functional
2. ✅ API Endpoints Working - All custom_commands endpoints operational
3. ✅ Code Deployed - FastAPI integration complete
4. ✅ Migration Script Ready - Tested and validated locally

Ready for Production Migration

• Source database accessible: /mnt/NV2/Development/major-domo/sba_is_fun.db
• Target database ready: Production sba_master.db
• Migration script tested: migrate_custom_commands.py
• API endpoints verified: https://sba.manticorum.com/api/v3/custom_commands
• Error handling confirmed: Proper 404s, authentication, validation

Production Migration Checklist

Pre-Migration

• Copy source database to production server
• Copy migration script to production server
• Verify production API is accessible
• Run migration in dry-run mode first

Migration Commands

# 1. Copy files to production
scp /mnt/NV2/Development/major-domo/sba_is_fun.db sba-database:/tmp/
scp migrate_custom_commands.py sba-database:/tmp/

# 2. SSH to production and copy to container
ssh sba-database
docker cp /tmp/sba_is_fun.db sba_database:/usr/src/app/storage/
docker cp /tmp/migrate_custom_commands.py sba_database:/usr/src/app/storage/

# 3. Run dry-run migration
docker exec sba_database python /usr/src/app/storage/migrate_custom_commands.py \
  --source /usr/src/app/storage/sba_is_fun.db \
  --target /usr/src/app/storage/sba_master.db \
  --dry-run

# 4. Run actual migration
docker exec sba_database python /usr/src/app/storage/migrate_custom_commands.py \
  --source /usr/src/app/storage/sba_is_fun.db \
  --target /usr/src/app/storage/sba_master.db

Post-Migration Validation

• Check API stats: https://sba.manticorum.com/api/v3/custom_commands/stats
• Test command lookup: https://sba.manticorum.com/api/v3/custom_commands/by_name/yeli
• Verify autocomplete: https://sba.manticorum.com/api/v3/custom_commands/autocomplete
• Check creator statistics in database
• Test Discord bot integration (if applicable)

Known Issues & Considerations

Grace Period Implementation

• Commands not used in 60+ days get last_used updated to migration date
• Prevents mass deletion of historical commands
• Gives commands a grace period to be discovered/used again

Duplicate Handling

• Migration script checks for existing creators by discord_id
• Migration script checks for existing commands by name
• Existing entries are skipped with warnings logged

Performance

• 140 commands migrate in ~1 second
• No performance impact on API during migration
• Indexes created for optimal query performance

Rollback Plan

If migration issues occur:

1. Delete migrated entries: DELETE FROM custom_commands WHERE tags LIKE '%migrated%'
2. Delete migrated creators: DELETE FROM custom_command_creators WHERE id > [original_max_id]
3. Re-run migration after fixing issues

Contact Information

Development Team: Claude Code AI Assistant
Migration Date: August 17, 2025
Production URL: https://sba.manticorum.com/api/v3/custom_commands
Local Test URL: http://localhost:8001/api/v3/custom_commands

File Locations

• Migration Script: /mnt/NV2/Development/major-domo/database/migrate_custom_commands.py
• Source Database: /mnt/NV2/Development/major-domo/sba_is_fun.db
• Custom Commands Module: /mnt/NV2/Development/major-domo/database/app/routers_v3/custom_commands.py
• Main API File: /mnt/NV2/Development/major-domo/database/app/main.py
• Database Models: /mnt/NV2/Development/major-domo/database/app/db_engine.py

---

Status: Ready for production migration. All testing completed successfully.