update db str

2026-03-05 12:43:44 +05:30
parent eabf295f2e
commit 6e184dc590
27 changed files with 2780 additions and 146 deletions
--- a/backend/MIGRATION_GUIDE.md
+++ b/backend/MIGRATION_GUIDE.md
@@ -0,0 +1,437 @@
+# Grateful Journal — Migration Guide
+
+**Version:** 2.0 → 2.1 (Database Refactoring)  
+**Date:** 2026-03-05
+
+---
+
+## Overview
+
+This guide walks you through migrating your MongoDB database from the old schema (with duplicate users and string userId references) to the new refactored schema.
+
+⚠️ **IMPORTANT:** Backup your database before starting. This process modifies your data.
+
+---
+
+## Pre-Migration Checklist
+
+- [ ] No active users using the application
+- [ ] Database backup created
+- [ ] Python dependencies installed
+- [ ] FastAPI backend stopped
+- [ ] MongoDB running and accessible
+
+---
+
+## Step 1: Backup Your Database
+
+**Critical:** Always backup before running migrations.
+
+```bash
+# Create timestamped backup
+mongodump --db grateful_journal --out ./backup-$(date +%Y%m%d-%H%M%S)
+
+# Verify backup
+ls -lh backup-*/
+```
+
+This creates a directory like `backup-2026-03-05-120000` with all your data.
+
+**Alternative: Cloud Backup (MongoDB Atlas)**
+
+If using MongoDB Atlas, create a snapshot in the dashboard before proceeding.
+
+---
+
+## Step 2: Verify Current Database State
+
+Before migration, inspect your current data:
+
+```bash
+# Check duplicate users by email
+mongosh --db grateful_journal << 'EOF'
+db.users.aggregate([
+  { $group: { _id: "$email", count: { $sum: 1 }, ids: { $push: "$_id" } } },
+  { $match: { count: { $gt: 1 } } }
+])
+EOF
+```
+
+**Expected Output:**  
+If you see results, you have duplicates. The migration script will consolidate them.
+
+---
+
+## Step 3: Ensure Dependencies
+
+The migration script uses PyMongo, which should already be installed:
+
+```bash
+cd /Users/jeet/Desktop/Jio/grateful-journal
+
+# Check if pymongo is installed
+python -c "import pymongo; print(pymongo.__version__)"
+
+# If not installed:
+pip install pymongo
+```
+
+---
+
+## Step 4: Run the Migration Script
+
+Navigate to the backend directory and run the migration:
+
+```bash
+cd /Users/jeet/Desktop/Jio/grateful-journal/backend
+
+# Run the migration
+python scripts/migrate_data.py
+```
+
+**Script Output:**
+
+The script will:
+1. Report duplicate users found
+2. Map old duplicate user IDs to the canonical (oldest) user
+3. Update all entries to reference the canonical user
+4. Convert `userId` from string to ObjectId
+5. Add `entryDate` field to entries
+6. Add `encryption` metadata to entries
+7. Verify data integrity
+
+**Example Output:**
+
+```
+✓ Connected to MongoDB: grateful_journal
+
+======================================================================
+STEP 1: Deduplicating Users (keeping oldest)
+======================================================================
+
+📧 Email: jeet.debnath2004@gmail.com
+   Found 12 duplicate users
+   Keeping (earliest): ObjectId('69a7d6749a69142259e40394')
+   Deleting (later):   ObjectId('69a7db0f8fbb489ac05ab945')
+   Deleting (later):   ObjectId('69a7db178fbb489ac05ab946')
+   ...
+
+✓ Removed 11 duplicate users
+
+======================================================================
+STEP 2: Migrating Entries (userId string → ObjectId, add entryDate)
+======================================================================
+
+Total entries to process: 42
+
+  ✓ Processed 100/150 entries
+  ✓ Updated 150/150 entries
+
+✓ Updated 150 entries
+
+======================================================================
+STEP 3: Verifying Data Integrity
+======================================================================
+
+Users collection: 1
+Entries collection: 150
+
+✓ All entries have valid user references
+
+Sample entry structure:
+  _id (entry): ObjectId('...') (ObjectId: True)
+  userId: ObjectId('...') (ObjectId: True)
+  entryDate present: True
+  encryption present: True
+
+======================================================================
+✓ Migration Complete
+======================================================================
+Duplicate users removed: 11
+Entries migrated: 150
+Orphaned entries found: 0
+
+✓ Data integrity verified successfully!
+```
+
+---
+
+## Step 5: Create Indexes
+
+After migration, create indexes for optimized performance:
+
+```bash
+python backend/scripts/create_indexes.py
+```
+
+**Expected Output:**
+
+```
+✓ Connected to MongoDB: grateful_journal
+
+Creating indexes for 'users' collection...
+  ✓ Created unique index on email
+  ✓ Created index on createdAt
+
+Creating indexes for 'entries' collection...
+  ✓ Created compound index on (userId, createdAt)
+  ✓ Created compound index on (userId, entryDate)
+  ✓ Created index on tags
+  ✓ Created index on entryDate
+
+============================================================
+✓ Index Creation Complete
+============================================================
+Total indexes created: 7
+  • users.email_unique
+  • users.createdAt_desc
+  • entries.userId_createdAt
+  • entries.userId_entryDate
+  • entries.tags
+  • entries.entryDate_desc
+
+✓ Disconnected from MongoDB
+```
+
+---
+
+## Step 6: Verify Schema
+
+Verify the new schema is correct:
+
+```bash
+mongosh --db grateful_journal << 'EOF'
+// Check user structure
+db.users.findOne()
+
+// Check entry structure
+db.entries.findOne()
+
+// Count documents
+db.users.countDocuments({})
+db.entries.countDocuments({})
+
+// Verify indexes
+db.users.getIndexes()
+db.entries.getIndexes()
+EOF
+```
+
+**Expected Sample Output:**
+
+```javascript
+// User document
+{
+  _id: ObjectId("507f1f77bcf86cd799439011"),
+  email: "jeet.debnath2004@gmail.com",
+  displayName: "Jeet Debnath",
+  photoURL: "https://...",
+  theme: "light",
+  createdAt: ISODate("2026-03-04T06:51:32.598Z"),
+  updatedAt: ISODate("2026-03-05T10:30:00.000Z")
+}
+
+// Entry document
+{
+  _id: ObjectId("507f1f77bcf86cd799439012"),
+  userId: ObjectId("507f1f77bcf86cd799439011"),  // ← Now ObjectId!
+  title: "Today's Gratitude",
+  content: "I'm grateful for...",
+  mood: "grateful",
+  tags: ["family", "work"],
+  isPublic: false,
+  entryDate: ISODate("2026-03-05T00:00:00.000Z"),  // ← New field!
+  createdAt: ISODate("2026-03-05T12:30:15.123Z"),
+  updatedAt: ISODate("2026-03-05T12:30:15.123Z"),
+  encryption: {                                      // ← New field!
+    encrypted: false,
+    iv: null,
+    algorithm: null
+  }
+}
+```
+
+---
+
+## Step 7: Test Backend
+
+Start the backend and verify it works with the new schema:
+
+```bash
+cd /Users/jeet/Desktop/Jio/grateful-journal/backend
+
+# Start the backend (in a new terminal)
+python -m uvicorn main:app --reload --port 8001
+```
+
+**Test endpoints:**
+
+```bash
+# Health check
+curl http://localhost:8001/health
+
+# Get user by email (replace with your email)
+curl -X GET "http://localhost:8001/api/users/by-email/jeet.debnath2004@gmail.com"
+
+# Get user entries
+curl -X GET "http://localhost:8001/api/entries/{user_id}?limit=10&skip=0"
+```
+
+Expected: All requests succeed with 200 status.
+
+---
+
+## Step 8: Restart Frontend
+
+Once confident the backend works, restart the frontend:
+
+```bash
+# In a new terminal
+cd /Users/jeet/Desktop/Jio/grateful-journal
+npm run dev  # or your dev command
+```
+
+Test the full application:
+- Login via Google
+- Create an entry
+- View entries in history
+- Check calendar view
+
+---
+
+## Rollback Procedure
+
+If something goes wrong:
+
+```bash
+# Restore from backup
+mongorestore --drop --db grateful_journal ./backup-2026-03-05-120000
+
+# Restart backend and frontend
+```
+
+This will revert the database to its pre-migration state.
+
+---
+
+## Troubleshooting
+
+### Issue: "invalid ObjectId" errors
+
+**Cause:** Some entries still have string userId references.  
+**Fix:** Re-run the migration script:
+```bash
+python backend/scripts/migrate_data.py
+```
+
+### Issue: Entries not showing up
+
+**Cause:** userId is still a string in old entries.  
+**Fix:** Check the entry structure:
+```bash
+mongosh --db grateful_journal
+db.entries.findOne()  # Check userId type
+```
+
+If userId is a string, run migration again.
+
+### Issue: "duplicate key error" on email index
+
+**Cause:** Index creation failed due to duplicate emails.  
+**Fix:** The migration script handles this, but if you hit this:
+```bash
+# Rerun migration
+python scripts/migrate_data.py
+```
+
+### Issue: Script won't run
+
+```bash
+# Ensure you're in the backend directory
+cd /Users/jeet/Desktop/Jio/grateful-journal/backend
+
+# Check Python path
+python --version
+
+# Run with explicit module path
+python -m scripts.migrate_data
+```
+
+### Issue: MongoDB connection refused
+
+```bash
+# Check if MongoDB is running
+mongosh
+
+# If not running, start it:
+# On macOS with Homebrew:
+brew services start mongodb-community
+
+# Or manually:
+mongod
+```
+
+---
+
+## Post-Migration
+
+### Update Documentation
+
+- [x] Update [SCHEMA.md](./SCHEMA.md) with new schema
+- [x] Update [models.py](./models.py)
+- [x] Update router docstrings
+
+### Performance Tuning
+
+Monitor slow queries:
+
+```bash
+mongosh --db grateful_journal << 'EOF'
+// Monitor slow queries
+db.setProfilingLevel(1, { slowms: 100 })
+
+// Check profiling
+db.system.profile.find().pretty()
+EOF
+```
+
+### Data Analysis
+
+Check migration statistics:
+
+```bash
+mongosh --db grateful_journal << 'EOF'
+// Total users and entries
+db.users.countDocuments({})
+db.entries.countDocuments({})
+
+// Entries with encryption
+db.entries.countDocuments({ "encryption.encrypted": true })
+
+// Entries without entryDate (should be 0)
+db.entries.countDocuments({ entryDate: { $exists: false } })
+EOF
+```
+
+---
+
+## Next Steps
+
+1. **Monitor**: Watch logs for any errors or warnings
+2. **Test**: Thoroughly test all features (login, create, read, update, delete)
+3. **Celebrate**: You've successfully migrated! 🎉
+
+---
+
+## Support
+
+If you encounter issues:
+
+1. Check [SCHEMA.md](./SCHEMA.md) for schema details
+2. Review backend logs: `tail -f logs/backend.log`
+3. Inspect MongoDB: Use mongosh to query directly
+4. Consult the code: Check [routers/users.py](./routers/users.py) and [routers/entries.py](./routers/entries.py)
+
+---
+
+_Happy journaling! 📔_