insertAll using mongodb bulk API
Problem
batch inserts would be good for perf (with in the model.create method). obstacles in 2.x: 1) dependence on triggering each inserted documents 'save' hooks. the hook system isn't flexible enough for us to trigger them without actually calling doc.save. 2) doc.save() is tightly coupled to its persistence method to the db (e.g. collection.insert, collection.update). we don't actually want to .insert() it. we want to trigger hooks without calling doc.save(), then use insertAll.
Unverified for your environment
Select your OS to check compatibility.
1 Fix
Implement Bulk Insert with Pre-Save Hooks in Mongoose
Mongoose's default behavior triggers save hooks during individual document saves, which is not compatible with bulk operations like insertAll. The save method is tightly coupled with the persistence logic, preventing the use of bulk operations without losing hook functionality.
Awaiting Verification
Be the first to verify this fix
- 1
Create a Custom Bulk Insert Function
Develop a function that prepares documents for bulk insertion while triggering necessary pre-save hooks manually.
javascriptasync function bulkInsertWithHooks(Model, docs) { const hooks = []; for (const doc of docs) { const hookResult = await Model.hooks.pre('save').call(doc); if (hookResult !== false) { hooks.push(doc.toObject()); } } await Model.insertMany(hooks); } - 2
Modify the Model to Include Pre-Save Hook Logic
Ensure that the model includes the necessary pre-save hooks that need to be triggered during the bulk insert process.
javascriptconst mongoose = require('mongoose'); const MySchema = new mongoose.Schema({ /* schema definition */ }); MySchema.pre('save', function(next) { /* hook logic */ next(); }); const MyModel = mongoose.model('MyModel', MySchema); - 3
Test the Bulk Insert Function
Create unit tests to validate that the bulk insert function triggers hooks correctly and inserts documents as expected.
javascriptdescribe('Bulk Insert with Hooks', () => { it('should trigger pre-save hooks and insert documents', async () => { const docs = [{ /* doc1 */ }, { /* doc2 */ }]; await bulkInsertWithHooks(MyModel, docs); // assertions here }); }); - 4
Handle Errors Gracefully
Implement error handling within the bulk insert function to ensure that any issues during the hook execution or insertion are logged and managed appropriately.
javascripttry { await Model.insertMany(hooks); } catch (error) { console.error('Bulk insert failed:', error); throw error; } - 5
Document the Changes
Update the project documentation to reflect the new bulk insert functionality and how to use it, including any limitations or considerations.
Validation
Confirm that the bulk insert function correctly triggers pre-save hooks and successfully inserts the documents into the database without errors. Use unit tests to validate functionality.
Sign in to verify this fix
Environment
Submitted by
Alex Chen
2450 rep