[JAVA-2547] Provide generic api for cursor based paging Created: 01/Jul/17  Updated: 06/Apr/23  Resolved: 08/Nov/17

Status: Closed
Project: Java Driver
Component/s: API
Affects Version/s: 3.4.2
Fix Version/s: None

Type: New Feature Priority: Minor - P4
Reporter: Christian Finckler Assignee: Unassigned
Resolution: Won't Fix Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   

For large amounts of documents it is not performant to use page-based paging with skip and limit.
Then it is better to use cursor based paging.That means you query by any attribute sorted and remember the value of the sorted field of the last document you found and give it to the client.
If the user in the client clicks on "next", you send the last value to the backend and query with a $gt-operator.

Here is an article which explains the idea in detail:
https://www.sitepoint.com/paginating-real-time-data-cursor-based-pagination/

The problem is, that you have to implement a lot to support that.Inside the java api it could be implemented much easier.

api-suggestion:
MongoCursor<DocumentWithPagingInfo> cursor = collection.findPagable()

The Class DocumentWithPagingInfo has two methods:
getDocument() and getPagingKey()
The pagingKey is a string, which encodes the informations about the used filter and sorting and the actual values of the filtered values.

This string can be given to the client. And if it passed it to the server application, the server application would call a new method collection.findWithPagingKey(String pagingKey)

Perhaps the api could be done better.

You also might ask, why that could not be implemented outside of the framework:
The pagingcursor must encapsule the filter sort and the actual values. These informations are all known inside the api. Outside it is not possible to implement a generic solution, because outside you cannot get these informations from a query-Object (it is a builder pattern).



 Comments   
Comment by Ross Lawley [ 05/Jul/17 ]

Hi chris123,

Thanks for your suggestion, while I agree a generic pagination class can make sense for certain users, I think it would be best suited at a higher level than the Java Driver. There are a couple of reasons for this, firstly, as a drivers team we are unifying the API across driver implementations. So if this were to be implemented it would have to be specified and implemented across all the drivers. Secondly, pagination of results, are often only done when displaying results as part of some web interface. For many users of the driver it wouldn't really make sense.

As you've mentioned there are patterns that can be used to implement pagination successfully. When used correctly, they rely on looking up an indexed value and knowing the last value seen so to issue a $gt query. This already can be achieved using the Filters helpers in the Java driver, when wanting to paginate the results and use the Filters.and(originalFilter, Filters.gt('fieldName', fieldValue)). You will just need to be able to recreate the originalFilter on each request and extend that value, to create the actual query. To get the json value of the originalFilter if you convert it to a BasicDBObject, Document or BsonDocument you can use the #toJson method.

While a common pattern for web applications, a generic API as described should be easily achievable if built on-top of the library and work according to your needs. However, there are many different UX pagination patterns in use! From the previous and next style described above, to numbered lists with ranges of values and so on. I don't think you can effectively model a single efficient generic pagination API covering all pagination scenarios. For those reasons I think any pagination API is best being written at a higher level than the driver.

I hope that makes sense,

Ross

Generated at Thu Feb 08 08:57:30 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.