[SERVER-13537] Large skip and and limit values can cause crash in blocking sort stage Created: 10/Apr/14  Updated: 11/Jul/16  Resolved: 14/Apr/14

Status: Closed
Project: Core Server
Component/s: Querying
Affects Version/s: 2.6.0
Fix Version/s: 2.6.1, 2.7.0

Type: Bug Priority: Major - P3
Reporter: Tianxiang Ma Assignee: David Storch
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Duplicate
Related
related to DOCS-3157 add warning about numerical overflow ... Closed
Operating System: ALL
Backport Completed:
Participants:

 Description   
Issue Status as of April 15, 2014

ISSUE SUMMARY
Very large values for .skip() and .limit() can cause an out-of-bounds error in the sort stage of a query and crash the server. Specifically, the error is caused when both skip and limit values fit into 32-bit signed ints individually but their sum overflows a signed 32-bit int and flips a positive limit value to a negative one.

USER IMPACT
Limited, but users can be affected as this is a regression and large limit values previously worked.

WORKAROUNDS
Use a lower limit value to avoid the issue.

RESOLUTION
Cast skip and limit to unsigned int values to ensure the sum doesn't result in a negative value.

AFFECTED VERSIONS
Version 2.6.0 is affected by this bug.

PATCHES
The patch is included in the 2.6.1 production release.

Original description

2014-04-10T02:45:04.086+0800 [conn952] SEVERE: Got signal: 6 (Aborted).
Backtrace:0x11bd301 0x11bc6de 0x3355832960 0x33558328e5 0x33558340c5 0x335582ba0e 0x335582bad0 0xac7944 0xaca025 0xaa6f8f 0xaa082c 0xaaaa81 0xac3661 0xd44534 0xd44f89 0xd47b8a 0xd4acf8 0xb96382 0xb98962 0x76b76f
./mongod(_ZN5mongo15printStackTraceERSo+0x21) [0x11bd301]
./mongod() [0x11bc6de]
/lib64/libc.so.6() [0x3355832960]
/lib64/libc.so.6(gsignal+0x35) [0x33558328e5]
/lib64/libc.so.6(abort+0x175) [0x33558340c5]
/lib64/libc.so.6() [0x335582ba0e]
/lib64/libc.so.6(__assert_perror_fail+0) [0x335582bad0]
./mongod(_ZN5mongo9SortStage11addToBufferERKNS0_16SortableDataItemE+0x454) [0xac7944]
./mongod(_ZN5mongo9SortStage4workEPm+0x865) [0xaca025]
./mongod(_ZN5mongo7OrStage4workEPm+0x12f) [0xaa6f8f]
./mongod(_ZN5mongo18KeepMutationsStage4workEPm+0x3c) [0xaa082c]
./mongod(_ZN5mongo15ProjectionStage4workEPm+0x51) [0xaaaa81]
./mongod(_ZN5mongo9SkipStage4workEPm+0x61) [0xac3661]
./mongod(_ZN5mongo15MultiPlanRunner12workAllPlansEPNS_7BSONObjE+0x1b4) [0xd44534]
./mongod(_ZN5mongo15MultiPlanRunner12pickBestPlanEPmPNS_7BSONObjE+0xa9) [0xd44f89]
./mongod(_ZN5mongo15MultiPlanRunner7getNextEPNS_7BSONObjEPNS_7DiskLocE+0x3fa) [0xd47b8a]
./mongod(ZN5mongo11newRunQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1+0x958) [0xd4acf8]
./mongod() [0xb96382]
./mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0x442) [0xb98962]
./mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x9f) [0x76b76f]



 Comments   
Comment by Githook User [ 14/Apr/14 ]

Author:

{u'username': u'dstorch', u'name': u'David Storch', u'email': u'david.storch@10gen.com'}

Message: SERVER-13537 better handle numerical overflow of large limit values

(cherry picked from commit 3e4b9474749fc3b4df182d813e09a26b0e02fcc6)
Branch: v2.6
https://github.com/mongodb/mongo/commit/9f1ef97401471aaeb430cb5c4ad9c06ffe7e60c6

Comment by Githook User [ 14/Apr/14 ]

Author:

{u'username': u'dstorch', u'name': u'David Storch', u'email': u'david.storch@10gen.com'}

Message: SERVER-13537 better handle numerical overflow of large limit values
Branch: master
https://github.com/mongodb/mongo/commit/3e4b9474749fc3b4df182d813e09a26b0e02fcc6

Comment by hari.khalsa@10gen.com [ 11/Apr/14 ]

tma995@gmail.com Thank you very much for finding the query! We'll fix this ASAP. Please do file further tickets for any other issues like this (though I hope there aren't any!).

Comment by Tianxiang Ma [ 11/Apr/14 ]

Hi Thomas,

It turns out not difficult to reproduce the crash, as it was caused by a scheduled daily task which leads to the same crash again today. After debugging, here comes the culprit query:

db.T_ScheduleResult.find({"scheduleType" : "IMMEDIATE"}, {_id:1}).sort({time:-1}).skip(1000).limit(2147483647)

The "limit number" seems too large and makes no sense, however it works for a long time in previous mongodb versions. I tried for a few other db & collections and MongoDB crash at the same stage if query in similar format(find->sort->skip->limit with a large number as above).

For info of our dev system, please refer to below:
CentOS 6.4
MongoDB 2.6, with primary replicaset (the only replicaset)

Anyway, by redefine the "limit" size to a small number we are able to get around the crash. Hope this helps.
Thanks,

TMA

Comment by Thomas Rueckstiess [ 10/Apr/14 ]

Hi Tianxiang,

Thanks for reporting this issue. In order to investigate and reproduce the crash we would need some more information:

  • Could you attach more of the log file to this ticket?
  • Can you tell us details about your system (OS, sharded / replicaset / standalone node?, etc).
  • Do you know what query/sort was causing the crash?
  • Can you also provide a list of your indexes, with

    use <database>
    db.<collection>.getIndexes()

    (insert your database and collection above).

Thanks,
Thomas

Comment by hari.khalsa@10gen.com [ 10/Apr/14 ]

Hello! Are you able to reproduce the crash? If so, can you tell me exactly how to do so?

Generated at Thu Feb 08 03:32:03 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.