[SERVER-26545] Remove fixed-size limitation on WiredTiger hazard pointers Created: 10/Oct/16  Updated: 16/Jul/17  Resolved: 19/Dec/16

Status: Closed
Project: Core Server
Component/s: WiredTiger
Affects Version/s: None
Fix Version/s: 3.2.12, 3.4.2, 3.5.2

Type: Improvement Priority: Major - P3
Reporter: Alexander Gorrod Assignee: Keith Bostic (Inactive)
Resolution: Done Votes: 2
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Related
Backwards Compatibility: Fully Compatible
Participants:
Case:

 Description   

A configuration setting for WiredTiger limits how many operations can happen in a single thread context at a point in time, that is controlled via the wiredtiger_open::hazard_max configuration option.

There are some MongoDB workloads that try to use more than the configured maximum number of hazard pointers, which leads to an invariant failure in MongoDB of the form:

2016-09-04T21:28:55.564+0800 E STORAGE [conn858615] WiredTiger (0) [1472995735:547997][1153:0x7f48d9a80700], file:index-109--487061380080497386.wt, cursor.search_near: session 0x2dc23c0: hazard pointer table full
2016-09-04T21:28:55.571+0800 I - [conn858615] Invariant failure: ret resulted in status UnknownError 12: Cannot allocate memory at src/mongo/db/storage/wiredtiger/wiredtiger_index.cpp 732
2016-09-04T21:28:55.786+0800 I CONTROL [conn858615]

We should review handling of operations that could fail due to hazard pointer exhaustion, and see if we can improve error handling. Probably making the individual operation return an error, but allowing the server to stay alive.



 Comments   
Comment by Githook User [ 06/Jan/17 ]

Author:

{u'username': u'daveh86', u'name': u'David Hows', u'email': u'howsdav@gmail.com'}

Message: Import wiredtiger: 8d2324943364286056ae399043f70b8a937de312 from branch mongodb-3.4

ref: ca6eee06ff..8d23249433
for: 3.4.2

SERVER-26545 Remove fixed-size limitation on WiredTiger hazard pointers
WT-2336 Add a test validating schema operations via file system call monitoring
WT-2402 Misaligned structure accesses lead to undefined behavior
WT-2670 Inefficient I/O when read full DB (poor readahead)
WT-283 Add a way to change persistent object settings
WT-2960 Inserting multi-megabyte values can cause pathological lookaside usage
WT-2969 Possible snapshot corruption during compaction
WT-3014 Add GCC/clang support for ELF symbol visibility.
WT-3021 Fixes needed for Java log cursor example, Java raw mode cursors, log cursors in raw mode
WT-3025 fix error path in log_force_sync
WT-3028 Workloads with all dirty pages could trigger diagnostic stuck check
WT-3030 Test failure indicating invalid key order during traversal
WT-3034 Add support for single-writer named snapshots.
WT-3037 Fix some outdated comments in logging
WT-3048 WiredTiger maximum size warning uses the wrong format.
WT-3051 Remove external __wt_hex symbol.
WT-3052 Improve search if an index hint is wrong
WT-3053 Review Python and Java calls to internal WiredTiger functions
WT-3054 Java PackTest, PackTest03 do not compile
WT-3055 Java AsyncTest faults
WT-3056 For cursors with projections, keys should be allowed
WT-3057 WiredTiger hazard pointers should use the WT_REF, not the WT_PAGE.
WT-3061 syscall test runs with checkpoint_sync=false and doesn't acknowledge pwrite64
WT-3064 minor tree cleanups: .gitignore, NEWS misspelling
WT-3066 lint
WT-3068 Copy wtperf artifacts when running Jenkins tests
WT-3069 Fix build failures in LevelDB APIs
WT-3070 Fix search_near() for index cursor
WT-3071 Java: fix build with -Werror=sign-conversion
WT-3075 Document and enforce that WiredTiger now depends on Python 2.7
WT-3078 Fix a hang in the reconfiguration test.
WT-3084 Fix Coverity resource leak complaint.
Branch: v3.4
https://github.com/mongodb/mongo/commit/d2c64ac8c526b70eadeb859ec41370a5f03a64aa

Comment by Githook User [ 06/Jan/17 ]

Author:

{u'username': u'daveh86', u'name': u'David Hows', u'email': u'howsdav@gmail.com'}

Message: Import wiredtiger: d48181f6f4db08761ed7b80b0332908b272ad0d0 from branch mongodb-3.2

ref: 040e3d6f76..d48181f6f4
for: 3.2.12

SERVER-26545 Remove fixed-size limitation on WiredTiger hazard pointers
WT-2336 Add a test validating schema operations via file system call monitoring
WT-2402 Misaligned structure accesses lead to undefined behavior
WT-2670 Inefficient I/O when read full DB (poor readahead)
WT-283 Add a way to change persistent object settings
WT-2960 Inserting multi-megabyte values can cause pathological lookaside usage
WT-2969 Possible snapshot corruption during compaction
WT-3014 Add GCC/clang support for ELF symbol visibility.
WT-3021 Fixes needed for Java log cursor example, Java raw mode cursors, log cursors in raw mode
WT-3025 fix error path in log_force_sync
WT-3028 Workloads with all dirty pages could trigger diagnostic stuck check
WT-3030 Test failure indicating invalid key order during traversal
WT-3034 Add support for single-writer named snapshots.
WT-3037 Fix some outdated comments in logging
WT-3048 WiredTiger maximum size warning uses the wrong format.
WT-3051 Remove external __wt_hex symbol.
WT-3052 Improve search if an index hint is wrong
WT-3053 Review Python and Java calls to internal WiredTiger functions
WT-3054 Java PackTest, PackTest03 do not compile
WT-3055 Java AsyncTest faults
WT-3056 For cursors with projections, keys should be allowed
WT-3057 WiredTiger hazard pointers should use the WT_REF, not the WT_PAGE.
WT-3061 syscall test runs with checkpoint_sync=false and doesn't acknowledge pwrite64
WT-3064 minor tree cleanups: .gitignore, NEWS misspelling
WT-3066 lint
WT-3068 Copy wtperf artifacts when running Jenkins tests
WT-3069 Fix build failures in LevelDB APIs
WT-3070 Fix search_near() for index cursor
WT-3071 Java: fix build with -Werror=sign-conversion
WT-3075 Document and enforce that WiredTiger now depends on Python 2.7
WT-3078 Fix a hang in the reconfiguration test.
WT-3084 Fix Coverity resource leak complaint.
Branch: v3.2
https://github.com/mongodb/mongo/commit/52b68fa86ea43e909ad42c901d0579bced6b205f

Comment by Githook User [ 06/Jan/17 ]

Author:

{u'username': u'keithbostic', u'name': u'Keith Bostic', u'email': u'keith.bostic@mongodb.com'}

Message: SERVER-26545 Remove fixed-size limitation on WiredTiger hazard pointers (#3187)

  • Default to an initial 250 hazard slots and grow from there.
  • Make hazard_max undocumented, add an internal limit of 1000 eviction walks.
  • If we grow the hazard pointer array, schedule the original to be freed when the database is closed.
Comment by Githook User [ 06/Jan/17 ]

Author:

{u'username': u'keithbostic', u'name': u'Keith Bostic', u'email': u'keith.bostic@mongodb.com'}

Message: SERVER-26545 Remove fixed-size limitation on WiredTiger hazard pointers (#3187)

  • Default to an initial 250 hazard slots and grow from there.
  • Make hazard_max undocumented, add an internal limit of 1000 eviction walks.
  • If we grow the hazard pointer array, schedule the original to be freed when the database is closed.
Comment by Alexander Gorrod [ 19/Dec/16 ]

Some supported users have set the configuration option in the past in response to errors. We may want to document somewhere that it's no longer interesting to set this option.

Comment by Alexander Gorrod [ 19/Dec/16 ]

We have made a WiredTiger change to deprecate the hazard_max configuration option. MongoDB doesn't explicitly set hazard_max when configuring WiredTiger, so we don't need to make changes to the server code in order to take advantage of the new option.

Existing users who set an explicit limit are likely to see a warning message on startup saying that the configuration string is deprecated.

Comment by Githook User [ 19/Dec/16 ]

Author:

{u'username': u'agorrod', u'name': u'Alex Gorrod', u'email': u'alexander.gorrod@mongodb.com'}

Message: Import wiredtiger: 8d2324943364286056ae399043f70b8a937de312 from branch mongodb-3.6

ref: 1b6c815a3f..8d23249433
for: 3.5.2

SERVER-26545 Remove fixed-size limitation on WiredTiger hazard pointers
WT-2402 Misaligned structure accesses lead to undefined behavior
WT-283 Add a way to change persistent object settings
WT-3056 For cursors with projections, keys should be allowed
WT-3061 syscall test runs with checkpoint_sync=false and doesn't acknowledge pwrite64
WT-3066 lint
WT-3068 Copy wtperf artifacts when running Jenkins tests
WT-3069 Fix build failures in LevelDB APIs
WT-3070 Fix search_near() for index cursor
WT-3071 Java: fix build with -Werror=sign-conversion
WT-3075 Document and enforce that WiredTiger now depends on Python 2.7
WT-3078 Fix a hang in the reconfiguration test.
WT-3084 Fix Coverity resource leak complaint.
Branch: master
https://github.com/mongodb/mongo/commit/4a185996c83862c2e4b911ce637591c4ff535108

Comment by Githook User [ 15/Dec/16 ]

Author:

{u'username': u'keithbostic', u'name': u'Keith Bostic', u'email': u'keith.bostic@mongodb.com'}

Message: SERVER-26545 Remove fixed-size limitation on WiredTiger hazard pointers (#3187)

  • Default to an initial 250 hazard slots and grow from there.
  • Make hazard_max undocumented, add an internal limit of 1000 eviction walks.
  • If we grow the hazard pointer array, schedule the original to be freed when the database is closed.
Generated at Thu Feb 08 04:12:28 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.