grateful-journal/backend/MIGRATION_GUIDE.md

# Grateful Journal — Migration Guide

**Version:** 2.0 → 2.1 (Database Refactoring)  
**Date:** 2026-03-05

---

## Overview

This guide walks you through migrating your MongoDB database from the old schema (with duplicate users and string userId references) to the new refactored schema.

⚠️ **IMPORTANT:** Backup your database before starting. This process modifies your data.

---

## Pre-Migration Checklist

- [ ] No active users using the application
- [ ] Database backup created
- [ ] Python dependencies installed
- [ ] FastAPI backend stopped
- [ ] MongoDB running and accessible

---

## Step 1: Backup Your Database

**Critical:** Always backup before running migrations.

```bash
# Create timestamped backup
mongodump --db grateful_journal --out ./backup-$(date +%Y%m%d-%H%M%S)

# Verify backup
ls -lh backup-*/
```

This creates a directory like `backup-2026-03-05-120000` with all your data.

**Alternative: Cloud Backup (MongoDB Atlas)**

If using MongoDB Atlas, create a snapshot in the dashboard before proceeding.

---

## Step 2: Verify Current Database State

Before migration, inspect your current data:

```bash
# Check duplicate users by email
mongosh --db grateful_journal << 'EOF'
db.users.aggregate([
  { $group: { _id: "$email", count: { $sum: 1 }, ids: { $push: "$_id" } } },
  { $match: { count: { $gt: 1 } } }
])
EOF
```

**Expected Output:**  
If you see results, you have duplicates. The migration script will consolidate them.

---

## Step 3: Ensure Dependencies

The migration script uses PyMongo, which should already be installed:

```bash
cd /Users/jeet/Desktop/Jio/grateful-journal

# Check if pymongo is installed
python -c "import pymongo; print(pymongo.__version__)"

# If not installed:
pip install pymongo
```

---

## Step 4: Run the Migration Script

Navigate to the backend directory and run the migration:

```bash
cd /Users/jeet/Desktop/Jio/grateful-journal/backend

# Run the migration
python scripts/migrate_data.py
```

**Script Output:**

The script will:

1. Report duplicate users found
2. Map old duplicate user IDs to the canonical (oldest) user
3. Update all entries to reference the canonical user
4. Convert `userId` from string to ObjectId
5. Add `entryDate` field to entries
6. Add `encryption` metadata to entries
7. Verify data integrity

**Example Output:**

```
✓ Connected to MongoDB: grateful_journal

======================================================================
STEP 1: Deduplicating Users (keeping oldest)
======================================================================

📧 Email: jeet.debnath2004@gmail.com
   Found 12 duplicate users
   Keeping (earliest): ObjectId('69a7d6749a69142259e40394')
   Deleting (later):   ObjectId('69a7db0f8fbb489ac05ab945')
   Deleting (later):   ObjectId('69a7db178fbb489ac05ab946')
   ...

✓ Removed 11 duplicate users

======================================================================
STEP 2: Migrating Entries (userId string → ObjectId, add entryDate)
======================================================================

Total entries to process: 42

  ✓ Processed 100/150 entries
  ✓ Updated 150/150 entries

✓ Updated 150 entries

======================================================================
STEP 3: Verifying Data Integrity
======================================================================

Users collection: 1
Entries collection: 150

✓ All entries have valid user references

Sample entry structure:
  _id (entry): ObjectId('...') (ObjectId: True)
  userId: ObjectId('...') (ObjectId: True)
  entryDate present: True
  encryption present: True

======================================================================
✓ Migration Complete
======================================================================
Duplicate users removed: 11
Entries migrated: 150
Orphaned entries found: 0

✓ Data integrity verified successfully!
```

---

## Step 5: Create Indexes

After migration, create indexes for optimized performance:

```bash
python backend/scripts/create_indexes.py
```

**Expected Output:**

```
✓ Connected to MongoDB: grateful_journal

Creating indexes for 'users' collection...
  ✓ Created unique index on email
  ✓ Created index on createdAt

Creating indexes for 'entries' collection...
  ✓ Created compound index on (userId, createdAt)
  ✓ Created compound index on (userId, entryDate)
  ✓ Created index on tags
  ✓ Created index on entryDate

============================================================
✓ Index Creation Complete
============================================================
Total indexes created: 7
  • users.email_unique
  • users.createdAt_desc
  • entries.userId_createdAt
  • entries.userId_entryDate
  • entries.tags
  • entries.entryDate_desc

✓ Disconnected from MongoDB
```

---

## Step 6: Verify Schema

Verify the new schema is correct:

```bash
mongosh --db grateful_journal << 'EOF'
// Check user structure
db.users.findOne()

// Check entry structure
db.entries.findOne()

// Count documents
db.users.countDocuments({})
db.entries.countDocuments({})

// Verify indexes
db.users.getIndexes()
db.entries.getIndexes()
EOF
```

**Expected Sample Output:**

```javascript
// User document
{
  _id: ObjectId("507f1f77bcf86cd799439011"),
  email: "jeet.debnath2004@gmail.com",
  displayName: "Jeet Debnath",
  photoURL: "https://...",
  theme: "light",
  createdAt: ISODate("2026-03-04T06:51:32.598Z"),
  updatedAt: ISODate("2026-03-05T10:30:00.000Z")
}

// Entry document
{
  _id: ObjectId("507f1f77bcf86cd799439012"),
  userId: ObjectId("507f1f77bcf86cd799439011"),  // ← Now ObjectId!
  title: "Today's Gratitude",
  content: "I'm grateful for...",
  mood: "grateful",
  tags: ["family", "work"],
  isPublic: false,
  entryDate: ISODate("2026-03-05T00:00:00.000Z"),  // ← New field!
  createdAt: ISODate("2026-03-05T12:30:15.123Z"),
  updatedAt: ISODate("2026-03-05T12:30:15.123Z"),
  encryption: {                                      // ← New field!
    encrypted: false,
    iv: null,
    algorithm: null
  }
}
```

---

## Step 7: Test Backend

Start the backend and verify it works with the new schema:

```bash
cd /Users/jeet/Desktop/Jio/grateful-journal/backend

# Start the backend (in a new terminal)
python -m uvicorn main:app --reload --port 8001
```

**Test endpoints:**

```bash
# Health check
curl http://localhost:8001/health

# Get user by email (replace with your email)
curl -X GET "http://localhost:8001/api/users/by-email/jeet.debnath2004@gmail.com"

# Get user entries
curl -X GET "http://localhost:8001/api/entries/{user_id}?limit=10&skip=0"
```

Expected: All requests succeed with 200 status.

---

## Step 8: Restart Frontend

Once confident the backend works, restart the frontend:

```bash
# In a new terminal
cd /Users/jeet/Desktop/Jio/grateful-journal
npm run dev  # or your dev command
```

Test the full application:

- Login via Google
- Create an entry
- View entries in history
- Check calendar view

---

## Rollback Procedure

If something goes wrong:

```bash
# Restore from backup
mongorestore --drop --db grateful_journal ./backup-2026-03-05-120000

# Restart backend and frontend
```

This will revert the database to its pre-migration state.

---

## Troubleshooting

### Issue: "invalid ObjectId" errors

**Cause:** Some entries still have string userId references.  
**Fix:** Re-run the migration script:

```bash
python backend/scripts/migrate_data.py
```

### Issue: Entries not showing up

**Cause:** userId is still a string in old entries.  
**Fix:** Check the entry structure:

```bash
mongosh --db grateful_journal
db.entries.findOne()  # Check userId type
```

If userId is a string, run migration again.

### Issue: "duplicate key error" on email index

**Cause:** Index creation failed due to duplicate emails.  
**Fix:** The migration script handles this, but if you hit this:

```bash
# Rerun migration
python scripts/migrate_data.py
```

### Issue: Script won't run

```bash
# Ensure you're in the backend directory
cd /Users/jeet/Desktop/Jio/grateful-journal/backend

# Check Python path
python --version

# Run with explicit module path
python -m scripts.migrate_data
```

### Issue: MongoDB connection refused

```bash
# Check if MongoDB is running
mongosh

# If not running, start it:
# On macOS with Homebrew:
brew services start mongodb-community

# Or manually:
mongod
```

---

## Post-Migration

### Update Documentation

- [x] Update [SCHEMA.md](./SCHEMA.md) with new schema
- [x] Update [models.py](./models.py)
- [x] Update router docstrings

### Performance Tuning

Monitor slow queries:

```bash
mongosh --db grateful_journal << 'EOF'
// Monitor slow queries
db.setProfilingLevel(1, { slowms: 100 })

// Check profiling
db.system.profile.find().pretty()
EOF
```

### Data Analysis

Check migration statistics:

```bash
mongosh --db grateful_journal << 'EOF'
// Total users and entries
db.users.countDocuments({})
db.entries.countDocuments({})

// Entries with encryption
db.entries.countDocuments({ "encryption.encrypted": true })

// Entries without entryDate (should be 0)
db.entries.countDocuments({ entryDate: { $exists: false } })
EOF
```

---

## Next Steps

1. **Monitor**: Watch logs for any errors or warnings
2. **Test**: Thoroughly test all features (login, create, read, update, delete)
3. **Celebrate**: You've successfully migrated! 🎉

---

## Support

If you encounter issues:

1. Check [SCHEMA.md](./SCHEMA.md) for schema details
2. Review backend logs: `tail -f logs/backend.log`
3. Inspect MongoDB: Use mongosh to query directly
4. Consult the code: Check [routers/users.py](./routers/users.py) and [routers/entries.py](./routers/entries.py)

---

_Happy journaling! 📔_