[SERVER-50512] Mongodb migration causes duplicate key : Created: 25/Aug/20  Updated: 15/Sep/20  Resolved: 15/Sep/20

Status: Closed
Project: Core Server
Component/s: Write Ops
Affects Version/s: 4.2 Required
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Tita Youlika Assignee: Eric Sedor
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Operating System: ALL
Participants:

 Description   

Hello,

In an old project we are trying to migrate from basic spring to spring boot 2.3.1. To do so and because we have a mongo database, we have to migrate from spring-data-mongodb:1.10.18 where this code was written :

DBCollection contextCollection = this.mongoTemplate.getCollection("productStock");
BulkWriteOperation builder = contextCollection.initializeUnorderedBulkOperation();
StockType stockItem = stockMessage.getStockItem();
 
final BasicDBObject id = new BasicDBObject("storeID", stockItem.getStoreID()).append("productID", stockItem.getProductID());
BulkWriteRequestBuilder bulkWriteRequestBuilder = builder.find(new BasicDBObject("_id", id));
HashMap<String, Object> stock = new HashMap<>();
Date currentDate = Calendar.getInstance().getTime();
 
stock.put("value", stockItem.getValue());
if (stockItem.getAssociateDate() != null) {
    stock.put("associateDate", stockItem.getAssociateDate());
}
 
if (stockItem.getLastAccessDateSource() != null) {
    stock.put("lastAccessDateSource", stockItem.getLastAccessDateSource());
 
    // check
    BasicDBObject ltLast = new BasicDBObject("$lt", stockItem.getLastAccessDateSource());
    BasicDBList dbList = new BasicDBList();
    dbList.add(new BasicDBObject(stockItem.getStockCategory() + ".lastAccessDateSource", ltLast));
    dbList.add(new BasicDBObject(stockItem.getStockCategory() + ".lastAccessDateSource", null));
    bulkWriteRequestBuilder = builder.find(new BasicDBObject("_id", id).append("$or", dbList));
} else {
    stock.put("lastAccessDateSource", currentDate);
}
 
stock.put("lastUpdateDate", currentDate);
BasicDBObject set = new BasicDBObject(stockItem.getStockCategory(), new Document(stock));
bulkWriteRequestBuilder.upsert().updateOne(new BasicDBObject("$set", set));
builder.execute();

 

to spring-boot-starter-data-mongodb (with mongodb-driver-sync-4.0.4) with this updated code :

 

Map<String, List<StockType>> mapMultiUpdate = new HashMap<>();
StockType stockItem = stockMessage.getStockItem();
 
final Document id = new Document("storeID", stockItem.getStoreID()).append("productID", stockItem.getProductID());
HashMap<String, Object> stock = new HashMap<>();
Date currentDate = Calendar.getInstance().getTime();
Document searchQuery = new Document("_id", id).append("$or", dbList);
stock.put("value", stockItem.getValue());
if (stockItem.getAssociateDate() != null) {
    stock.put("associateDate", stockItem.getAssociateDate());
}
 
if (stockItem.getLastAccessDateSource() != null) {
    stock.put("lastAccessDateSource", stockItem.getLastAccessDateSource());
 
    // check
    Document ltLast = new Document("$lt", stockItem.getLastAccessDateSource());
    List<Document> dbList = Lists.newArrayList();
    dbList.add(new Document(stockItem.getStockCategory() + ".lastAccessDateSource", ltLast));
    dbList.add(new Document(stockItem.getStockCategory() + ".lastAccessDateSource", null));
} else {
    stock.put("lastAccessDateSource", currentDate);
}
 
 
//Bulk write options
BulkWriteOptions bulkWriteOptions = new BulkWriteOptions();
bulkWriteOptions.ordered(false);
bulkWriteOptions.bypassDocumentValidation(true);
 
MongoCollection<Document> mongoCollection = this.mongoTemplate.getCollection("productStoreStock");
mongoCollection.bulkWrite(updateDocuments, bulkWriteOptions);

 

But when the new code is executed on an already existing object we get a duplicated key error :

om.mongodb.MongoBulkWriteException: Bulk write operation error on server localhost:27017. Write errors: [BulkWriteError{index=0, code=11000, message='E11000 duplicate key error collection: test.productStoreStock index: _id_ dup key: { : { storeID: 400, productID: 100000 } }', details={}}]*

We use last version of mongodb 4.2.x



 Comments   
Comment by Eric Sedor [ 27/Aug/20 ]

Hi tiengpub2020@gmail.com, For assistance troubleshooting duplicate key (E11000) errors, could you please start with our community by posting on the MongoDB Developer Community Forums or on Stack Overflow with the mongodb tag? Then, return to the SERVER project if you have identified a bug or a specific improvement to request. Does this make sense?

Gratefully,
Eric

Comment by Tita Youlika [ 25/Aug/20 ]

EDIT :

This error is thrown during the 3rd step of a test phase, on an empty database / collection. the steps :

  • initiating the collection with one stock on a product at a specific date
  • checking the value of the base
  • modifying the value of the stock in java but not the date and trying to update it
  • checking the value is still the first one because of the LT filter on the mongo query

We never reach the checking value and before the migration, everything was good on this test..

Generated at Thu Feb 08 05:22:51 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.