[SERVER-13562] Queries that use tailable cursors do not stream results if skip() is applied Created: 11/Apr/14  Updated: 11/Jul/16  Resolved: 15/Apr/14

Status: Closed
Project: Core Server
Component/s: Querying
Affects Version/s: 2.6.0
Fix Version/s: 2.6.1, 2.7.0

Type: Bug Priority: Major - P3
Reporter: Steve Green Assignee: David Storch
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Related
related to SERVER-13566 Using the OplogReplay flag with extra... Closed
Operating System: ALL
Backport Completed:
Participants:

 Description   
Issue Status as of April 15, 2014

ISSUE SUMMARY
Queries that use tailable cursors do not stream results if .skip() or negative values for .limit() are applied but instead close the cursor after returning the first batch.

USER IMPACT
Internal operations (replication etc.) are not affected but user-issued queries using tailable cursors may return incomplete results.

WORKAROUNDS
Try avoiding .skip() with tailable cursors and use other query predicates (e.g. $gt) if possible.

RESOLUTION
Tailable cursors now correctly continue to return batches via the getmore command when combined with .skip() or negative values of .limit().

AFFECTED VERSIONS
Version 2.6.0 is affected by this bug.

PATCHES
The patch is included in the 2.6.1 production release.

Original description

Queries that use tailable cursors do not stream results if skip() is applied.

e.g.

db.foo.find().addOption(34).skip(1)

Original description:

A capped collection created by an older version of the server can no longer be 'tracked' by a tailable cursor in 2.6.0; there was no issue with these collections using tailable cursors in v2.4.9. The capped collection being used was probably created pre-v2.2, but I'm not sure and can't find a way to determine. This collection did not have an index on '_id', which was added prior to the v2.6.0. upgrade.

This is not an issue if the capped collection is created with v2.6.0.

Steps to Reproduce:

Using a capped collection created in an older version of the server, e.g. 'test.cap1', attempt to track the inserts to this collection using two mongodb shell sessions.

In the first shell, perform inserts, e.g.,

use test
db.cap1.insert( { 'msg' : 'test' } )

In the second shell, attempt to track new insertions, e.g.,

use test
db.cap1.find().addOption(34) // 2 = tailable, 32 = await_data

When running with 2.6.0, no updates are displayed.

If a new capped collection is created using v2.6.0, and the shell commands above are run, the updates are displayed.



 Comments   
Comment by Githook User [ 15/Apr/14 ]

Author:

{u'username': u'dstorch', u'name': u'David Storch', u'email': u'david.storch@10gen.com'}

Message: SERVER-13562 limit and skip stages no longer return eof without calling work() on the child

Doing so could cause missing results for a tailable cursor.

(cherry picked from commit 9d1f365bf8f0c3d6192284cffab4e3927722e17c)
Branch: v2.6
https://github.com/mongodb/mongo/commit/1b7b02db89e8a574cb2f5bff717090a8719d4fb9

Comment by David Storch [ 15/Apr/14 ]

A clarification on the behavior of tailable cursors combined with limit:

  2.4.x 2.6.0 2.6.1-pre-
limit > 0 no limit enforced no limit enforced no limit enforced
limit < 0 no limit enforced one batch only (streaming broken) limit enforced

I think the 2.6.1-pre- behavior is reasonable in that usually limits don't apply to tailable cursors, but if you really want a limit you can use a negative value.

Comment by Githook User [ 15/Apr/14 ]

Author:

{u'username': u'dstorch', u'name': u'David Storch', u'email': u'david.storch@10gen.com'}

Message: SERVER-13562 limit and skip stages no longer return eof without calling work() on the child

Doing so could cause missing results for a tailable cursor.
Branch: master
https://github.com/mongodb/mongo/commit/9d1f365bf8f0c3d6192284cffab4e3927722e17c

Comment by J Rassi [ 12/Apr/14 ]

Confirmed the issue; updated summary/description accordingly. Thanks for the report.

hari.khalsa@10gen.com, david.storch: please take a look.

Comment by Steve Green [ 12/Apr/14 ]

Jason,

I've been playing with this all day, and I'm now pretty convinced that the actual issue has nothing to do with capped collections created by older versions (although that may still be an issue, I don't really have any way to test it again, now that my server has been upgraded to 2.6). What appears to be different is the way the 'skip' parameter is handled on a v2.4 server compared to a v2.6 server in a query that results in a tailable cursor.

The capped collections we have are used to hold audit logs. I have scripts in several languages (e.g., perl, c) that use a tailable cursor to emulate the unix 'tail -f' behavior. So the script counts the number of entries in the collection, subtracts the number of entries we want to display, and uses the skip parameter in the query to set the cursor on the first record we want to display. If I don't skip any records, the tailable cursor works as expected. If I skip a single record, the tailable cursor never sees any new data (note that this was not a problem with v2.4).

So, to reproduce the issue, create a capped collection and add 2 records (as described in the original post), then use the following command to create a tailable cursor

db.cap1.find().skip(1).addOption(34)

Now, when you add new data to the capped collection, the new entries are not displayed.

The new entries are displayed when this is done on a v2.4 server

Comment by J Rassi [ 12/Apr/14 ]

slg1013: after following "Steps to Reproduce", I am unable to reproduce your findings. Can you give another set of instructions for reproducing either of the unexpected behaviors you're reporting on?

Comment by Steve Green [ 11/Apr/14 ]

Perhaps I should file another bug, but I'm seeing strange behavior with tailable cursors even for capped collections created with v2.6.0. In one case, two capped collections created with the same command, where the tailable cursor appears to work with one and not the other; the only difference between the two collections being the number of documents inserted into each. In another case, using a perl script, the tail seems to work initially, but after I restart the script, new data doesn't appear to be sent to the cursor. I'm not able to nail down any of these differences in behavior to a single cause...

Comment by Steve Green [ 11/Apr/14 ]

I should have mentioned that this is a 64 bit Windows Server 2008 installation.

Generated at Thu Feb 08 03:32:06 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.