[SERVER-22887] Mongo fails in non-busy hours Created: 29/Feb/16  Updated: 09/Feb/17  Resolved: 28/Mar/16

Status: Closed
Project: Core Server
Component/s: Admin
Affects Version/s: 3.2.3
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Edik Mkoyan Assignee: Kelsey Schubert
Resolution: Incomplete Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Operating System: ALL
Participants:

 Description   

I am running a cron job that queries some data from MongoDB during the night, when nobody else is using MongoDB to read or write data, and I get this error Error: server.lab:27017: Read timed out after reading 0 bytes, waited for 30.000000 seconds During the day the query lasts about 60 ms to return data. I have enough RAM, and even doubling it doesn't help. MongoDB is running on XFS file system. I use version 3.2.3 (newest). I know that MongoDB uses linux cache, and most probably the problem is the warming up mechanism of MongoDB.

Here is the query.

db.event.aggregate([
{
     $match : {"schedule.sessions.attendance.0" : {$exists : true}}
},
{
    $unwind : "$schedule"
},
{
    $unwind : "$schedule.sessions"
},
{
    $project : {
    '_id' : false,
    'schedule.sessions.date_start' : 1,
    'schedule.sessions.date_finish' : 1,
    'schedule.sessions.attendance' : 1
    }
},
{
    $match : {
        "schedule.sessions.attendance.0" : {$exists : true},
        'schedule.sessions.date_start' : {$gte : ISODate("2016-02-14T20:00:00.000+0000"), $lt : ISODate("2016-02-15T20:00:00.000+0000") }
    }
}
 
])

event collection contains about 5000 documents.

And here is the structure of the document

mongos> db.event.findOne()
{
    "_id" : ObjectId("563cc6993764872c268b45db"),
    "schedule" : [
        {
            "_id" : ObjectId("563cc6993764872c268b45d7"),
            "sessions" : [
                {
                    "_id" : ObjectId("5694d3c842600772f64bcc1a"),
                    "date_start" : ISODate("2016-02-08T11:30:00Z"),
                    "date_finish" : ISODate("2016-02-08T13:30:00Z"),
                    "place" : [ ],
                    "attendance" : [
                        ObjectId("55ed6d03b3b852c812003b92"),
                        ObjectId("55ed6ce1b3b852c8120023f2"),
                        ObjectId("55eee520b3b852ac930003cb"),
                        ObjectId("55ed6cfdb3b852c812003722"),
                        ObjectId("55ed6d01b3b852c812003a52"),
                        ObjectId("55ed6d01b3b852c812003a7d")
                    ],
                    "date_created" : null,
                    "date_updated" : null,
                    "shard_location_id" : null
                }
            ]
        },
        {
            "_id" : ObjectId("563cc6993764872c268b45d8"),
            "sessions" : [
                {
                    "_id" : ObjectId("5694d3c842600772f64bcc1b"),
                    "date_start" : ISODate("2016-02-15T11:30:00Z"),
                    "date_finish" : ISODate("2016-02-15T13:30:00Z"),
                    "place" : [ ],
                    "attendance" : [
                        ObjectId("55ed6d03b3b852c812003b92"),
                        ObjectId("55ed6ce1b3b852c8120023f2"),
                        ObjectId("55eee520b3b852ac930003cb"),
                        ObjectId("55ed6cfdb3b852c812003722"),
                        ObjectId("55ed6d01b3b852c812003a7d")
                    ],
                    "date_created" : null,
                    "date_updated" : null,
                    "shard_location_id" : null
                }
            ]
        },
        {
            "_id" : ObjectId("563cc6993764872c268b45d9"),
            "sessions" : [
                {
                    "_id" : ObjectId("5694d3c842600772f64bcc1c"),
                    "date_start" : ISODate("2016-02-22T11:30:00Z"),
                    "date_finish" : ISODate("2016-02-22T13:30:00Z"),
                    "place" : [ ],
                    "attendance" : [ ],
                    "date_created" : null,
                    "date_updated" : null,
                    "shard_location_id" : null
                }
            ]
        },
        {
            "_id" : ObjectId("563cc6993764872c268b45da"),
            "sessions" : [
                {
                    "_id" : ObjectId("5694d3c842600772f64bcc1d"),
                    "date_start" : ISODate("2016-02-29T11:30:00Z"),
                    "date_finish" : ISODate("2016-02-29T13:30:00Z"),
                    "place" : [ ],
                    "attendance" : [ ],
                    "date_created" : null,
                    "date_updated" : null,
                    "shard_location_id" : null
                }
            ]
        },
        {
            "_id" : ObjectId("56b9d21c376487714d2c1c0a"),
            "sessions" : [
                {
                    "_id" : ObjectId("56b9d21c376487714d2c1c09"),
                    "date_start" : ISODate("2016-02-14T11:30:00Z"),
                    "date_finish" : ISODate("2016-02-14T13:30:00Z"),
                    "place" : [ ],
                    "attendance" : [
                        ObjectId("55ed6d03b3b852c812003b92"),
                        ObjectId("55ed6ce1b3b852c8120023f2"),
                        ObjectId("55eee520b3b852ac930003cb"),
                        ObjectId("55ed6cfdb3b852c812003722"),
                        ObjectId("55ed6d01b3b852c812003a7d")
                    ],
                    "date_created" : null,
                    "date_updated" : null,
                    "shard_location_id" : null
                }
            ],
            "date_created" : ISODate("2016-02-09T11:48:44.400Z"),
            "date_updated" : ISODate("2016-02-09T11:48:44.400Z"),
            "shard_location_id" : null
        },
        {
            "_id" : ObjectId("56b9d24c376487794e2c1c0a"),
            "sessions" : [
                {
                    "_id" : ObjectId("56b9d24c376487794e2c1c09"),
                    "date_start" : ISODate("2016-02-21T11:30:00Z"),
                    "date_finish" : ISODate("2016-02-21T13:30:00Z"),
                    "place" : [ ],
                    "attendance" : [ ],
                    "date_created" : null,
                    "date_updated" : null,
                    "shard_location_id" : null
                }
            ],
            "date_created" : ISODate("2016-02-09T11:49:32.324Z"),
            "date_updated" : ISODate("2016-02-09T11:49:32.324Z"),
            "shard_location_id" : null
        },
        {
            "_id" : ObjectId("56b9d2583764870e4e2c1c0b"),
            "sessions" : [
                {
                    "_id" : ObjectId("56b9d2583764870e4e2c1c0a"),
                    "date_start" : ISODate("2016-02-28T11:30:00Z"),
                    "date_finish" : ISODate("2016-02-28T13:30:00Z"),
                    "place" : [ ],
                    "attendance" : [ ],
                    "date_created" : null,
                    "date_updated" : null,
                    "shard_location_id" : null
                }
            ],
            "date_created" : ISODate("2016-02-09T11:49:44.108Z"),
            "date_updated" : ISODate("2016-02-09T11:49:44.108Z"),
            "shard_location_id" : null
        },
        {
            "_id" : ObjectId("56c47ff416d37ce33f1e8144"),
            "sessions" : [
                {
                    "_id" : ObjectId("56c47ff416d37ce33f1e8143"),
                    "date_start" : ISODate("2016-03-06T11:30:00Z"),
                    "date_finish" : ISODate("2016-03-06T13:30:00Z"),
                    "place" : [ ],
                    "attendance" : [ ],
                    "date_created" : ISODate("2016-02-17T14:13:08.652Z"),
                    "date_updated" : ISODate("2016-02-17T14:13:08.652Z"),
                    "shard_location_id" : null
                }
            ],
            "date_created" : ISODate("2016-02-17T14:13:08.652Z"),
            "date_updated" : ISODate("2016-02-17T14:13:08.652Z"),
            "shard_location_id" : null
        }
    ],
    "subscription" : [
        {
            "_id" : ObjectId("56af42ff16d37c4901afc81d"),
            "user_id" : ObjectId("55ed6d00b3b852c8120039e7"),
            "status" : "approved",
            "type" : NumberLong(2),
            "date_created" : ISODate("2016-02-01T11:35:27.324Z"),
            "date_updated" : ISODate("2016-02-01T11:35:27.324Z")
        },
        {
            "_id" : ObjectId("56af441958f591da05954741"),
            "user_id" : ObjectId("55eee515b3b852ac9300024a"),
            "status" : "approved",
            "type" : NumberLong(2),
            "date_created" : ISODate("2016-02-01T11:40:09.034Z"),
            "date_updated" : ISODate("2016-02-01T11:40:09.034Z")
        },
        {
            "_id" : ObjectId("56af44b6f5417201179f6822"),
            "user_id" : ObjectId("55ed6cfcb3b852c8120036b5"),
            "status" : "approved",
            "type" : NumberLong(2),
            "date_created" : ISODate("2016-02-01T11:42:46.296Z"),
            "date_updated" : ISODate("2016-02-01T11:42:46.296Z")
        },
        {
            "_id" : ObjectId("56af450d16d37cb20aafc83c"),
            "user_id" : ObjectId("55ed6d03b3b852c812003b92"),
            "status" : "approved",
            "type" : NumberLong(2),
            "date_created" : ISODate("2016-02-01T11:44:13.048Z"),
            "date_updated" : ISODate("2016-02-01T11:44:13.048Z")
        },
        {
            "_id" : ObjectId("56af4543f541728c1a9f67d5"),
            "user_id" : ObjectId("55ed6ce9b3b852c812002924"),
            "status" : "approved",
            "type" : NumberLong(2),
            "date_created" : ISODate("2016-02-01T11:45:07.372Z"),
            "date_updated" : ISODate("2016-02-01T11:45:07.372Z")
        },
        {
            "_id" : ObjectId("56af458a16d37cb20aafc8c1"),
            "user_id" : ObjectId("55ed6cffb3b852c812003912"),
            "status" : "approved",
            "type" : NumberLong(2),
            "date_created" : ISODate("2016-02-01T11:46:18.765Z"),
            "date_updated" : ISODate("2016-02-01T11:46:18.765Z")
        },
        {
            "_id" : ObjectId("56af4612f54172991d9f67f5"),
            "user_id" : ObjectId("55ed6cd6b3b852c812001b93"),
            "status" : "approved",
            "type" : NumberLong(2),
            "date_created" : ISODate("2016-02-01T11:48:34.755Z"),
            "date_updated" : ISODate("2016-02-01T11:48:34.755Z")
        },
        {
            "_id" : ObjectId("56af45f116d37c680fafc8e3"),
            "user_id" : ObjectId("55ed6cfdb3b852c812003723"),
            "status" : "approved",
            "type" : NumberLong(2),
            "date_created" : ISODate("2016-02-01T11:48:01.395Z"),
            "date_updated" : ISODate("2016-02-01T11:48:01.395Z")
        },
        {
            "_id" : ObjectId("56af46bc16d37c2f11afc8ac"),
            "user_id" : ObjectId("55eee513b3b852ac9300020a"),
            "status" : "approved",
            "type" : NumberLong(2),
            "date_created" : ISODate("2016-02-01T11:51:24.653Z"),
            "date_updated" : ISODate("2016-02-01T11:51:24.653Z")
        },
        {
            "_id" : ObjectId("56af46e2376487b31fa41f34"),
            "user_id" : ObjectId("55eee519b3b852ac930002ca"),
            "status" : "approved",
            "type" : NumberLong(2),
            "date_created" : ISODate("2016-02-01T11:52:02.953Z"),
            "date_updated" : ISODate("2016-02-01T11:52:02.953Z")
        },
        {
            "_id" : ObjectId("56af4756f5417214259f67c0"),
            "user_id" : ObjectId("55ed6ce1b3b852c8120023f2"),
            "status" : "approved",
            "type" : NumberLong(2),
            "date_created" : ISODate("2016-02-01T11:53:58.595Z"),
            "date_updated" : ISODate("2016-02-01T11:53:58.595Z")
        },
        {
            "_id" : ObjectId("56af477af5417261259f67c3"),
            "user_id" : ObjectId("55eee520b3b852ac930003cb"),
            "status" : "approved",
            "type" : NumberLong(2),
            "date_created" : ISODate("2016-02-01T11:54:34.308Z"),
            "date_updated" : ISODate("2016-02-01T11:54:34.308Z")
        },
        {
            "_id" : ObjectId("56af478e376487a822a41f52"),
            "user_id" : ObjectId("55ed6d04b3b852c812003c02"),
            "status" : "approved",
            "type" : NumberLong(2),
            "date_created" : ISODate("2016-02-01T11:54:54.084Z"),
            "date_updated" : ISODate("2016-02-01T11:54:54.084Z")
        },
        {
            "_id" : ObjectId("56b87f27f54172375e7b23cb"),
            "user_id" : ObjectId("55ed6ce1b3b852c812002389"),
            "type" : NumberLong(1),
            "status" : "approved",
            "date_created" : ISODate("2016-02-08T11:42:31.240Z"),
            "date_updated" : ISODate("2016-02-08T11:42:31.240Z"),
            "shard_location_id" : NumberLong(60074372)
        },
        {
            "_id" : ObjectId("56b87f27f54172375e7b23cc"),
            "user_id" : ObjectId("55ed6cdcb3b852c81200202d"),
            "type" : NumberLong(1),
            "status" : "approved",
            "date_created" : ISODate("2016-02-08T11:42:31.269Z"),
            "date_updated" : ISODate("2016-02-08T11:42:31.269Z"),
            "shard_location_id" : NumberLong(85072479)
        },
        {
            "_id" : ObjectId("56b87f27f54172375e7b23ce"),
            "user_id" : ObjectId("55ed6cdab3b852c812001ea6"),
            "type" : NumberLong(1),
            "status" : "approved",
            "date_created" : ISODate("2016-02-08T11:42:31.299Z"),
            "date_updated" : ISODate("2016-02-08T11:42:31.299Z"),
            "shard_location_id" : NumberLong(54819081)
        },
        {
            "_id" : ObjectId("56b87f27f54172375e7b23cf"),
            "user_id" : ObjectId("55ed6cdbb3b852c812001fba"),
            "type" : NumberLong(1),
            "status" : "approved",
            "date_created" : ISODate("2016-02-08T11:42:31.331Z"),
            "date_updated" : ISODate("2016-02-08T11:42:31.331Z"),
            "shard_location_id" : NumberLong(65625871),
            "status_deleted" : NumberLong(1)
        },
        {
            "_id" : ObjectId("56b87f27f54172375e7b23d0"),
            "user_id" : ObjectId("55ed6ce9b3b852c812002929"),
            "type" : NumberLong(1),
            "status" : "approved",
            "date_created" : ISODate("2016-02-08T11:42:31.360Z"),
            "date_updated" : ISODate("2016-02-08T11:42:31.360Z"),
            "shard_location_id" : NumberLong(79641334)
        },
        {
            "_id" : ObjectId("56b87f27f54172375e7b23d1"),
            "user_id" : ObjectId("55ed6cfab3b852c812003504"),
            "type" : NumberLong(1),
            "status" : "approved",
            "date_created" : ISODate("2016-02-08T11:42:31.388Z"),
            "date_updated" : ISODate("2016-02-08T11:42:31.388Z"),
            "shard_location_id" : NumberLong(94737821)
        },
        {
            "_id" : ObjectId("56b8806df5417242647b23c7"),
            "user_id" : ObjectId("55ed6cfeb3b852c812003845"),
            "type" : NumberLong(1),
            "status" : "approved",
            "date_created" : ISODate("2016-02-08T11:47:57.320Z"),
            "date_updated" : ISODate("2016-02-08T11:47:57.320Z"),
            "shard_location_id" : NumberLong(66955040)
        },
        {
            "_id" : ObjectId("56b8806df5417242647b23c8"),
            "user_id" : ObjectId("55ed6cfab3b852c812003596"),
            "type" : NumberLong(1),
            "status" : "approved",
            "date_created" : ISODate("2016-02-08T11:47:57.349Z"),
            "date_updated" : ISODate("2016-02-08T11:47:57.349Z"),
            "shard_location_id" : NumberLong(83983525)
        },
        {
            "_id" : ObjectId("56b8806df5417242647b23c9"),
            "user_id" : ObjectId("55ed6d00b3b852c8120039ac"),
            "type" : NumberLong(1),
            "status" : "approved",
            "date_created" : ISODate("2016-02-08T11:47:57.544Z"),
            "date_updated" : ISODate("2016-02-08T11:47:57.544Z"),
            "shard_location_id" : NumberLong(70202545),
            "status_deleted" : NumberLong(1)
        },
        {
            "_id" : ObjectId("56b8806df5417242647b23ca"),
            "user_id" : ObjectId("55ed6d03b3b852c812003b98"),
            "type" : NumberLong(1),
            "status" : "approved",
            "date_created" : ISODate("2016-02-08T11:47:57.572Z"),
            "date_updated" : ISODate("2016-02-08T11:47:57.572Z"),
            "shard_location_id" : NumberLong(83981351)
        },
        {
            "_id" : ObjectId("56b8806df5417242647b23cb"),
            "user_id" : ObjectId("55ed6cfdb3b852c812003722"),
            "type" : NumberLong(1),
            "status" : "approved",
            "date_created" : ISODate("2016-02-08T11:47:57.630Z"),
            "date_updated" : ISODate("2016-02-08T11:47:57.630Z"),
            "shard_location_id" : NumberLong(68552635)
        },
        {
            "_id" : ObjectId("56b8806df5417242647b23cc"),
            "user_id" : ObjectId("55ed6cffb3b852c812003894"),
            "type" : NumberLong(1),
            "status" : "approved",
            "date_created" : ISODate("2016-02-08T11:47:57.748Z"),
            "date_updated" : ISODate("2016-02-08T11:47:57.748Z"),
            "shard_location_id" : NumberLong(80433985)
        },
        {
            "_id" : ObjectId("56b8806df5417242647b23cd"),
            "user_id" : ObjectId("55ed6cfbb3b852c8120035ed"),
            "type" : NumberLong(1),
            "status" : "approved",
            "date_created" : ISODate("2016-02-08T11:47:57.957Z"),
            "date_updated" : ISODate("2016-02-08T11:47:57.957Z"),
            "shard_location_id" : NumberLong(67251008)
        },
        {
            "_id" : ObjectId("56b8806ef5417242647b23ce"),
            "user_id" : ObjectId("55ed6d01b3b852c812003a52"),
            "type" : NumberLong(1),
            "status" : "approved",
            "date_created" : ISODate("2016-02-08T11:47:58.029Z"),
            "date_updated" : ISODate("2016-02-08T11:47:58.029Z"),
            "shard_location_id" : NumberLong(98099161)
        },
        {
            "_id" : ObjectId("56b8806ef5417242647b23cf"),
            "user_id" : ObjectId("55ed6d01b3b852c812003a7d"),
            "type" : NumberLong(1),
            "status" : "approved",
            "date_created" : ISODate("2016-02-08T11:47:58.106Z"),
            "date_updated" : ISODate("2016-02-08T11:47:58.106Z"),
            "shard_location_id" : NumberLong(87539171)
        },
        {
            "_id" : ObjectId("56b8806ef5417242647b23d0"),
            "user_id" : ObjectId("55ed6cffb3b852c81200391d"),
            "type" : NumberLong(1),
            "status" : "approved",
            "date_created" : ISODate("2016-02-08T11:47:58.229Z"),
            "date_updated" : ISODate("2016-02-08T11:47:58.229Z"),
            "shard_location_id" : NumberLong(78085842)
        }
    ],
    "media" : [ ],
    "template_id" : ObjectId("5626c6c3b3b852bc31007ad6"),
    "title_en" : "3D Modeling I",
    "group_id" : ObjectId("563cc6993764872c268b45d5"),
    "user_slots" : [
        "15:30"
    ],
    "date_start" : ISODate("2016-02-07T20:00:00Z"),
    "date_finish" : ISODate("2016-03-05T20:00:00Z"),
    "place_id" : null,
    "points" : null,
    "date_created" : ISODate("2015-11-06T15:26:17.468Z"),
    "date_updated" : ISODate("2016-02-17T14:01:07.769Z"),
    "shard_location_id" : NumberLong(84354891),
    "shifted" : 5,
    "title_hy" : "3D մոդելավորում I",
    "public_access" : "0",
    "location_access" : [
        NumberLong(0)
    ],
    "description_hy" : "",
    "description_en" : "",
    "min_age_access" : "",
    "max_age_access" : "",
    "max_users" : ""
}



 Comments   
Comment by Edik Mkoyan [ 09/Feb/17 ]

Apologies for such a long delay, We have change the logic of our application few months ago, so it tries few times until the data is gathered properly. We can't confirm that the problem with mongo exist or resolved.

Comment by Ramon Fernandez Marina [ 28/Mar/16 ]

edikmkoyan, we haven't heard back from you for some time so we're closing this ticket. If this is still an issue for you please provide the information requested above by Thomas and we'll reopen this ticket.

Regards,
Ramón.

Comment by Kelsey Schubert [ 04/Mar/16 ]

Hi edikmkoyan,

Thank you for reporting this behavior. So we can get a better understanding of what is going on, please answer the following questions:

  1. Which driver are you using to connect to the mongod? Please include the version as well.
  2. Which storage engine are you using? MMAPv1 or WiredTiger?
  3. Are you able to successfully execute this query from the shell when the mongod is idle?
  4. Can you please upload the contents of the diagnostic.data directory in your dbpath to this ticket?
  5. Can you please provide the server logs for the time that the cronjob was executed? It would be useful to see 30 minutes before and a few minutes after.
  6. Can you also check the the disk activity when you execute the cronjob?

So we can investigate this issue more quickly, please explicitly state the times when the cronjob was started.

Thank you for your help,
Thomas

Generated at Thu Feb 08 04:01:42 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.