[JAVA-1029] Mongo throws com.mongodb.MongoException$Network: exception after trying to update a large number of documents in collection. Created: 18/Oct/13  Updated: 04/Dec/13  Resolved: 04/Dec/13

Status: Closed
Project: Java Driver
Component/s: Connection Management
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Critical - P2
Reporter: edrin gjoleka Assignee: Unassigned
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

linux, windows



 Description   

We are using spring with mongo java driver in order to access the mongo data.
The code we are using in order to update a big set of data is :

loop( in all newApplicationid)//some kind of loop in a list {
   final DBObject query = new BasicDBObject();
   query.put("in", instanceId);
   query.put("data.aid", oldApplicationId);
   final BasicDBObject updatedDocument = new BasicDBObject();
   updatedDocument.append("$set", new BasicDBObject().append("data.aid", newApplicationId));
   appEventsCollection.updateMulti(query, updatedDocument);
}

After running for a certain time round 2 minutes and drying all the memory of the machine than the following exception is thrown:
com.mongodb.MongoException$Network: Write operation to server localhost/127.0.0.1:27017 failed on database aas5_logging

After digging in the git repository for mongo driver i see that the exceptions is thrown from the file DBTCPConnector.java in line 157.
The problem is that in a production environment this can cause a serious problem. furthermore if you run the update again is going to execute for few more seconds and still throws the same exception. In order to complete succesfully the whole update several times the update has to be triggered automatically after each failure.
Below the peace of the java driver where this exception is thrown:

catch ( IOException ioe ){
            _myPort.error(port, ioe);
            _error( ioe, false );     
 if ( concern.raiseNetworkErrors() )
                throw new MongoException.Network("Write operation to server " + port.host() + " failed on database " + db , ioe );
 
            CommandResult res = new CommandResult(port.serverAddress());
            res.put( "ok" , false );
            res.put( "$err" , "NETWORK ERROR" );
            return new WriteResult( res , concern );
        }



 Comments   
Comment by Henrik Ingo (Inactive) [ 21/Oct/13 ]

Hi all. I met with Edrin in Stockholm ~2 weeks ago where this issue came up. I recommended to file this ticket, but don't know much more than what is in the description + what I'll add here.

Edrin:
Can you augment some information here. You said that when you run this long update on a desktop, MongoDB will consume most of the memory and then "crash" (possibly killed by kernel due to out-of-memory). Can you attach some information here from the server side to help understand that:

top output before the crash

after crash:
grep -i "Killed process" /var/log/syslog
grep -i "Killed process" /var/log/messages

And of course the full mongodb log from /var/log/mongodb/mongodb.log

A sample document from the collection you are updating. If you cannot share this publicly, could you email it to me?

Statistics of the collection being updated: db.collectionname.stats(1024*1024)

Hardware specs of the machine MongoDB was running on, in particular how much RAM it has.

Comment by Jeffrey Yemin [ 18/Oct/13 ]

Can you provide the full stack trace of the exception, including the IOException that is the cause?

Thanks,
Jeff

Generated at Thu Feb 08 08:53:38 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.