grateful-journal/backend/MIGRATION_GUIDE.md

Grateful Journal — Migration Guide

Version: 2.0 → 2.1 (Database Refactoring)
Date: 2026-03-05

---

Overview

This guide walks you through migrating your MongoDB database from the old schema (with duplicate users and string userId references) to the new refactored schema.

⚠️ IMPORTANT: Backup your database before starting. This process modifies your data.

---

Pre-Migration Checklist

• No active users using the application
• Database backup created
• Python dependencies installed
• FastAPI backend stopped
• MongoDB running and accessible

---

Step 1: Backup Your Database

Critical: Always backup before running migrations.

# Create timestamped backup
mongodump --db grateful_journal --out ./backup-$(date +%Y%m%d-%H%M%S)

# Verify backup
ls -lh backup-*/

This creates a directory like backup-2026-03-05-120000 with all your data.

Alternative: Cloud Backup (MongoDB Atlas)

If using MongoDB Atlas, create a snapshot in the dashboard before proceeding.

---

Step 2: Verify Current Database State

Before migration, inspect your current data:

# Check duplicate users by email
mongosh --db grateful_journal << 'EOF'
db.users.aggregate([
  { $group: { _id: "$email", count: { $sum: 1 }, ids: { $push: "$_id" } } },
  { $match: { count: { $gt: 1 } } }
])
EOF

Expected Output:
If you see results, you have duplicates. The migration script will consolidate them.

---

Step 3: Ensure Dependencies

The migration script uses PyMongo, which should already be installed:

cd /Users/jeet/Desktop/Jio/grateful-journal

# Check if pymongo is installed
python -c "import pymongo; print(pymongo.__version__)"

# If not installed:
pip install pymongo

---

Step 4: Run the Migration Script

Navigate to the backend directory and run the migration:

cd /Users/jeet/Desktop/Jio/grateful-journal/backend

# Run the migration
python scripts/migrate_data.py

Script Output:

The script will:

1. Report duplicate users found
2. Map old duplicate user IDs to the canonical (oldest) user
3. Update all entries to reference the canonical user
4. Convert userId from string to ObjectId
5. Add entryDate field to entries
6. Add encryption metadata to entries
7. Verify data integrity

Example Output:

✓ Connected to MongoDB: grateful_journal

======================================================================
STEP 1: Deduplicating Users (keeping oldest)
======================================================================

📧 Email: jeet.debnath2004@gmail.com
   Found 12 duplicate users
   Keeping (earliest): ObjectId('69a7d6749a69142259e40394')
   Deleting (later):   ObjectId('69a7db0f8fbb489ac05ab945')
   Deleting (later):   ObjectId('69a7db178fbb489ac05ab946')
   ...

✓ Removed 11 duplicate users

======================================================================
STEP 2: Migrating Entries (userId string → ObjectId, add entryDate)
======================================================================

Total entries to process: 42

  ✓ Processed 100/150 entries
  ✓ Updated 150/150 entries

✓ Updated 150 entries

======================================================================
STEP 3: Verifying Data Integrity
======================================================================

Users collection: 1
Entries collection: 150

✓ All entries have valid user references

Sample entry structure:
  _id (entry): ObjectId('...') (ObjectId: True)
  userId: ObjectId('...') (ObjectId: True)
  entryDate present: True
  encryption present: True

======================================================================
✓ Migration Complete
======================================================================
Duplicate users removed: 11
Entries migrated: 150
Orphaned entries found: 0

✓ Data integrity verified successfully!

---

Step 5: Create Indexes

After migration, create indexes for optimized performance:

python backend/scripts/create_indexes.py

Expected Output:

✓ Connected to MongoDB: grateful_journal

Creating indexes for 'users' collection...
  ✓ Created unique index on email
  ✓ Created index on createdAt

Creating indexes for 'entries' collection...
  ✓ Created compound index on (userId, createdAt)
  ✓ Created compound index on (userId, entryDate)
  ✓ Created index on tags
  ✓ Created index on entryDate

============================================================
✓ Index Creation Complete
============================================================
Total indexes created: 7
  • users.email_unique
  • users.createdAt_desc
  • entries.userId_createdAt
  • entries.userId_entryDate
  • entries.tags
  • entries.entryDate_desc

✓ Disconnected from MongoDB

---

Step 6: Verify Schema

Verify the new schema is correct:

mongosh --db grateful_journal << 'EOF'
// Check user structure
db.users.findOne()

// Check entry structure
db.entries.findOne()

// Count documents
db.users.countDocuments({})
db.entries.countDocuments({})

// Verify indexes
db.users.getIndexes()
db.entries.getIndexes()
EOF

Expected Sample Output:

// User document
{
  _id: ObjectId("507f1f77bcf86cd799439011"),
  email: "jeet.debnath2004@gmail.com",
  displayName: "Jeet Debnath",
  photoURL: "https://...",
  theme: "light",
  createdAt: ISODate("2026-03-04T06:51:32.598Z"),
  updatedAt: ISODate("2026-03-05T10:30:00.000Z")
}

// Entry document
{
  _id: ObjectId("507f1f77bcf86cd799439012"),
  userId: ObjectId("507f1f77bcf86cd799439011"),  // ← Now ObjectId!
  title: "Today's Gratitude",
  content: "I'm grateful for...",
  mood: "grateful",
  tags: ["family", "work"],
  isPublic: false,
  entryDate: ISODate("2026-03-05T00:00:00.000Z"),  // ← New field!
  createdAt: ISODate("2026-03-05T12:30:15.123Z"),
  updatedAt: ISODate("2026-03-05T12:30:15.123Z"),
  encryption: {                                      // ← New field!
    encrypted: false,
    iv: null,
    algorithm: null
  }
}

---

Step 7: Test Backend

Start the backend and verify it works with the new schema:

cd /Users/jeet/Desktop/Jio/grateful-journal/backend

# Start the backend (in a new terminal)
python -m uvicorn main:app --reload --port 8001

Test endpoints:

# Health check
curl http://localhost:8001/health

# Get user by email (replace with your email)
curl -X GET "http://localhost:8001/api/users/by-email/jeet.debnath2004@gmail.com"

# Get user entries
curl -X GET "http://localhost:8001/api/entries/{user_id}?limit=10&skip=0"

Expected: All requests succeed with 200 status.

---

Step 8: Restart Frontend

Once confident the backend works, restart the frontend:

# In a new terminal
cd /Users/jeet/Desktop/Jio/grateful-journal
npm run dev  # or your dev command

Test the full application:

• Login via Google
• Create an entry
• View entries in history
• Check calendar view

---

Rollback Procedure

If something goes wrong:

# Restore from backup
mongorestore --drop --db grateful_journal ./backup-2026-03-05-120000

# Restart backend and frontend

This will revert the database to its pre-migration state.

---

Troubleshooting

Issue: "invalid ObjectId" errors

Cause: Some entries still have string userId references.
Fix: Re-run the migration script:

python backend/scripts/migrate_data.py

Issue: Entries not showing up

Cause: userId is still a string in old entries.
Fix: Check the entry structure:

mongosh --db grateful_journal
db.entries.findOne()  # Check userId type

If userId is a string, run migration again.

Issue: "duplicate key error" on email index

Cause: Index creation failed due to duplicate emails.
Fix: The migration script handles this, but if you hit this:

# Rerun migration
python scripts/migrate_data.py

Issue: Script won't run

# Ensure you're in the backend directory
cd /Users/jeet/Desktop/Jio/grateful-journal/backend

# Check Python path
python --version

# Run with explicit module path
python -m scripts.migrate_data

Issue: MongoDB connection refused

# Check if MongoDB is running
mongosh

# If not running, start it:
# On macOS with Homebrew:
brew services start mongodb-community

# Or manually:
mongod

---

Post-Migration

Update Documentation

• Update SCHEMA.md with new schema
• Update models.py
• Update router docstrings

Performance Tuning

Monitor slow queries:

mongosh --db grateful_journal << 'EOF'
// Monitor slow queries
db.setProfilingLevel(1, { slowms: 100 })

// Check profiling
db.system.profile.find().pretty()
EOF

Data Analysis

Check migration statistics:

mongosh --db grateful_journal << 'EOF'
// Total users and entries
db.users.countDocuments({})
db.entries.countDocuments({})

// Entries with encryption
db.entries.countDocuments({ "encryption.encrypted": true })

// Entries without entryDate (should be 0)
db.entries.countDocuments({ entryDate: { $exists: false } })
EOF

---

Next Steps

1. Monitor: Watch logs for any errors or warnings
2. Test: Thoroughly test all features (login, create, read, update, delete)
3. Celebrate: You've successfully migrated! 🎉

---

Support

If you encounter issues:

1. Check SCHEMA.md for schema details
2. Review backend logs: tail -f logs/backend.log
3. Inspect MongoDB: Use mongosh to query directly
4. Consult the code: Check routers/users.py and routers/entries.py

---

Happy journaling! 📔