[SERVER-2701] dur commitifneeded needed in NamespaceDetails::capped? Created: 08/Mar/11  Updated: 29/May/12  Resolved: 14/Mar/11

Status: Closed
Project: Core Server
Component/s: Storage
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Minor - P4
Reporter: Dwight Merriman Assignee: Aaron Staple
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Operating System: ALL
Participants:

 Description   

Mon Mar 7 20:57:48 [conn14] debug nsincecommitifneeded:880 bytes:17530880
Mon Mar 7 20:57:48 [conn14] debug nsincecommitifneeded:920 bytes:17694720
Mon Mar 7 20:57:48 [conn14] debug nsincecommitifneeded:960 bytes:17858560
Mon Mar 7 20:57:48 [conn14] debug nsincecommitifneeded:1000 bytes:18022400
Mon Mar 7 20:57:48 [conn14] debug nsincecommitifneeded:1040 bytes:18186240
Mon Mar 7 20:57:48 [conn14] debug nsincecommitifneeded:1080 bytes:18350080
Mon Mar 7 20:57:48 [conn14] debug nsincecommitifneeded:1120 bytes:18513920
Mon Mar 7 20:57:48 [conn14] debug nsincecommitifneeded:1160 bytes:18677760
Mon Mar 7 20:57:48 [conn14] debug nsincecommitifneeded:1200 bytes:18841600
0x80e6c3 0x8d30a9 0x8cab9c 0x8cae0f 0x986867 0x98be2d 0x98ca49 0x98794e 0x986ac8 0x9b4f49 0x974a42 0x974f34 0x99b674 0x999334 0xa810f0 0x86eb91 0x86eb41 0x86eb
0 0x7f1106bdad60 0x3959e077e1
./mongod(_ZN5mongo15printStackTraceERSo+0x27) [0x80e6c3]
./mongod(_ZN5mongo3dur9CommitJob4noteEPvi+0x2a7) [0x8d30a9]
./mongod(_ZN5mongo3dur11DurableImpl18declareWriteIntentEPvj+0x2c) [0x8cab9c]
./mongod(_ZN5mongo3dur11DurableImpl10writingPtrEPvj+0x3d) [0x8cae0f]
./mongod(_ZN5mongo16NamespaceDetails13addDeletedRecEPNS_13DeletedRecordENS_7DiskLocE+0x35) [0x986867]
./mongod(_ZN5mongo16NamespaceDetails7compactEv+0x209) [0x98be2d]
./mongod(_ZN5mongo16NamespaceDetails11cappedAllocEPKci+0x337) [0x98ca49]
./mongod(_ZN5mongo16NamespaceDetails6_allocEPKci+0x52) [0x98794e]
./mongod(_ZN5mongo16NamespaceDetails5allocEPKciRNS_7DiskLocE+0x4e) [0x986ac8]
./mongod(_ZN5mongo11DataFileMgr17fast_oplog_insertEPNS_16NamespaceDetailsEPKci+0xf9) [0x9b4f49]
./mongod() [0x974a42]
./mongod(_ZN5mongo5logOpEPKcS1_RKNS_7BSONObjEPS2_Pb+0x5e) [0x974f34]
./mongod(_ZN5mongo14receivedInsertERNS_7MessageERNS_5CurOpE+0x38c) [0x99b674]
./mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_8SockAddrE+0x6bb) [0x999334]
./mongod(_ZN5mongo10connThreadEPNS_13MessagingPortE+0x2d3) [0xa810f0]
./mongod(_ZN5boost3_bi5list1INS0_5valueIPN5mongo13MessagingPortEEEEclIPFvS5_ENS0_5list0EEEvNS0_4typeIvEERT_RT0_i+0x47) [0x86eb91]
./mongod(_ZN5boost3_bi6bind_tIvPFvPN5mongo13MessagingPortEENS0_5list1INS0_5valueIS4_EEEEEclEv+0x3f) [0x86eb41]
./mongod(_ZN5boost6detail11thread_dataINS_3_bi6bind_tIvPFvPN5mongo13MessagingPortEENS2_5list1INS2_5valueIS6_EEEEEEE3runEv+0x1e) [0x86eb00]
/usr/lib64/libboost_thread-mt.so.5(thread_proxy+0x60) [0x7f1106bdad60]
/lib64/libpthread.so.0() [0x3959e077e1]
Mon Mar 7 20:57:48 [conn14] debug nsincecommitifneeded:1240 bytes:19005440



 Comments   
Comment by Dwight Merriman [ 14/Mar/11 ]

i think this may actually be ok as-is.

Comment by Dwight Merriman [ 09/Mar/11 ]

i'm pretty sure in general it's not safe to commitIfNeeded herein – imagine a capped collection with a bunch of indexes. if adding the index keys comes second, that becomes a separate "transaction".

the oplog is generally the last thing we do, in that scenario it is probably ok, but as said above doesn't seem to generalize. if it only goes up to 3x bsonsizemax it is ok anyway.

Comment by Dwight Merriman [ 09/Mar/11 ]

can get it to this or higher:

Tue Mar 8 19:05:40 [conn42] debug nsincecommitifneeded:9720 bytes:54607872

to repro run with --oplogSize 128, run several times:

./mongo jstests/b*js

big_object1.js is the thing that makes it happen. normal this will take some, just seems to be going a bit higher than expected. maybe that is the right number. for a big object we have
(1) the object itself
(2) its copy in the oplog
(3) the cleanup of the capped coll first.

so 3X max bsonobjsize is expected.

Comment by Dwight Merriman [ 08/Mar/11 ]

only if it's super safe

Comment by Eliot Horowitz (Inactive) [ 08/Mar/11 ]

I think we need to do this for 1.8.0, or no?

Comment by Dwight Merriman [ 08/Mar/11 ]

btw : would it possible to do this incrementally, so that we don't have such a large txn?

Comment by Aaron Staple [ 08/Mar/11 ]

There can potentially be lots of writes happening in the loop within cappedAlloc(). It would be fairly easy to add a commitIfNeeded() as part of that loop. We'd need to check the call sites of cappedAlloc() and its ancestor callers for write pointers that might be invalidated by a commit.

Dwight, could you assign a priority / version so I'll know when I should work on this?

Generated at Thu Feb 08 03:00:57 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.