[SERVER-10123] Support Timestamp(0,0) server replace for update (mods) Created: 08/Jul/13 Updated: 10/Dec/14 Resolved: 11/Oct/13 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Write Ops |
| Affects Version/s: | None |
| Fix Version/s: | None |
| Type: | Improvement | Priority: | Minor - P4 |
| Reporter: | Scott Hernandez (Inactive) | Assignee: | Scott Hernandez (Inactive) |
| Resolution: | Duplicate | Votes: | 6 |
| Labels: | timestamp, transformation, update | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||||||||||||||||||
| Backwards Compatibility: | Fully Compatible | ||||||||||||||||||||||||
| Participants: | |||||||||||||||||||||||||
| Description |
|
If a Timestamp(0,0) is sent via an update then do the server replacement of the value. Currently this behavior is only supported for the first 2 fields during an insert operation, but not updates. |
| Comments |
| Comment by Scott Hernandez (Inactive) [ 30/Nov/13 ] |
|
If you follow |
| Comment by Jeremy [ 29/Nov/13 ] |
|
What version / release will this feature be in? |
| Comment by Jeremy [ 11/Oct/13 ] |
|
Great. Can't wait to try it out. |
| Comment by Scott Hernandez (Inactive) [ 11/Oct/13 ] |
|
Yes, it is the same Timestamp type – http://bsonspec.org/#/specification |
| Comment by Jeremy [ 11/Oct/13 ] |
|
Hey Scott, Does $currentDate allow us to create the timestamp + unique increment like the Timestamp(0,0)? Thanks, Jeremy |
| Comment by Scott Hernandez (Inactive) [ 11/Oct/13 ] |
|
We now have $currentDate which allows you to set either a date or timestamp on the server – see |
| Comment by Jeremy [ 08/Aug/13 ] |
|
Regarding this: "This doesn't do what you suggest in a sharded environment since you don't get monotonically increasing values over a sharded collection, since there are multiple servers doing writes. Each shard (a replica set) basically will have its own oplog, and ordering of writes local to that shard." Actually, even in a shared environment, could this feature not still be used just by use the same shard key when incrementing is desired? Yes, it would only write to 1 place per key, but by using different keys for unrelated channels, you could still scale out writes. |
| Comment by Jeremy [ 30/Jul/13 ] |
|
Thanks Scott, Understood about sharding--I wouldn't expect sequencing to happen across multi-servers. In our case, there are many more readers than writers, so a single replica set with many secondaries works great...just that we need to improve on this model. I recently discovered that using the MongoTimestamp isn't exactly working 100% anyway since it can still be committed to the db out of order (rare but does happen). See below where my 'ts' i value is not in order (87 got committed before 86). { , , "_id": "0f882b762b6f9f00f76562caac6550ba" } } { "ts": { "t": 1374700873, "i": 89 }, "h": NumberLong("-6769405481799817919"), "v": 2, "op": "i", "ns": "rt01.notice", "o": { "ts": { "t": 1374700873, "i": 86 }, "vars": { ..object here. } , |
| Comment by Scott Hernandez (Inactive) [ 30/Jul/13 ] |
|
Jeremy, Replication no longer uses this. We will see if it will fit in the update refactoring or if another modifier is a better solution. This doesn't do what you suggest in a sharded environment since you don't get monotonically increasing values over a sharded collection, since there are multiple servers doing writes. Each shard (a replica set) basically will have its own oplog, and ordering of writes local to that shard. Feel free to collect votes and add ideas/feedback here. |
| Comment by Jeremy [ 29/Jul/13 ] |
|
HOW ABOUT THIS IDEA THAT WOULD ELIMINATE THE MongoTimestamp HACK? In the oplog, there is a 'ts'. Would it possible to just have a feature that will let us add this 'ts' to our own document instead of generating one with the null hack? If so, that would be a much better and more reliable solution. Maybe a new operator called '$oplogTimestamp'. db.test.save({ _id:'test', 'info':'insert and stamp', $oplogTimestamp: {'ts':1} ); } AND IN OPLOG db.oplog.rs.find().sort({$natural:-1}).limit(1).pretty() , } AND IN UPDATES db.test.update( { _id:'test'}, { $oplogTimestamp: {'tsu':1} } ); , } Am I just dreaming here? |
| Comment by Jeremy [ 20/Jul/13 ] |
|
Hey Scott. While the notion that this is hack may be true, please consider for a moment why many of us want to use this hack. Just having a '$now' option isn't good enough for a polling mechanism since it would be possible for 2 or more records inserted at the same time to have the same timestamp. What is quite lovely about your MongoTimestamp is that it is guaranteed to be unique (at least in non-sharded setup). Also, it not only provides a centrally generated timestamp, but also gives an order (increment) to these inserted records. This lets me do some neat stuff. For example: if 'ts' is a MongoTimestamp, it lets us make a query such as 'get the next 10 records where 'ts' is greater than last 'ts' received->sort by 'ts'. On response, we can take the last 'ts' and use it for the next 'gte' polling. If we did this just on a time, we would consistently be skipping records (limit 10 gte a $now might give us 10 back but, there could be 3 more after with same timestamp that wouldn't be captured on next polling). Here is our how we use this in our API: http://rtrt.me/docs/api/rest#polling If your Replication implementation needs this feature, why not let others use it for similar reasons. If you support this better, MongoDB could find many more uses...I can see it becoming a great backend store for a message bus. A a regular server side timestamp might be OK if you could also provide us with a server-side auto-incrementing option. I've seen people talk about a findAndModify option for this, but doesn't that require some more intense locking? And, getting a timestamp from client isn't very good idea since clients are never 100% in time sync. Finally, I'm not sure why the ticket says that currently this behavior is only supported for 'inserts' and not 'updates'. You can actually use the 'update' command, but you cannot do a '$set' of a field with this null timestamp and have it re-stamp the field. Being able to '$set' for this, and being able to use this on more than one field at a time is really all I am asking for. Thanks! Jeremy |
| Comment by Scott Hernandez (Inactive) [ 09/Jul/13 ] |
|
Once we have |