[SERVER-15369] explicitly zero .ns files on creation Created: 24/Sep/14  Updated: 15/Oct/15  Resolved: 26/Sep/14

Status: Closed
Project: Core Server
Component/s: Storage
Affects Version/s: None
Fix Version/s: 2.4.12, 2.6.5, 2.7.7

Type: Improvement Priority: Critical - P2
Reporter: Bruce Lucas (Inactive) Assignee: Mathias Stearn
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
Duplicate
is duplicated by SERVER-14137 db stats Closed
Related
Backwards Compatibility: Fully Compatible
Backport Completed:
Participants:
Linked BF Score: 0

 Description   
Issue Status as of Oct 03, 2014

ISSUE SUMMARY
MongoDB installations on some Linux systems running on VMWare may see corruption in namespace files.

MongoDB uses namespace (.ns) files to store database and collection metadata, and requires initial zeroing of these .ns files upon creation. A request to the operating system to zero a .ns file may fail to completely zero the file, leaving the previous contents of some disk locations exposed as part of the .ns file. This incomplete zeroing yields a corrupt .ns file, and may affect MongoDB in various ways.

Only .ns files created on systems running the following configuration are affected by this issue:

  • certain 3.x Linux kernels, including Ubuntu 12.x, 14.x, and Fedora 20; and
  • VMWare platform, including VMWare ESXi, Fusion, and Workstation; and
  • virtual SCSI disks managed by LVM

See the bug report on Launchpad for more information.

USER IMPACT
Symptoms that have been observed include:

  • failures of various operations that require enumerating namespaces, such as db.stats(), db.repairDatabase(), db.dropDatabase(), and others
  • operations such as the above that don’t fail may report nonsense database names
  • failure of mongod to start

These failures may be accompanied by logged error messages such as:

  • exception: assertion src/mongo/db/catalog/database.cpp:472
  • Assertion failure _name == nsToDatabaseSubstring( ns ) src/mongo/db/catalog/database.cpp 472
  • exception: nsToDatabase: db too long
  • Fatal Assertion 16253
  • ERROR: Uncaught std::exception: basic_string::substr, terminating
  • Error messages like: {"ok" : 0, "errmsg" : "missing ns: ..."}

Additional symptoms may be possible depending on the exact nature of the left-over data in the .ns file.

WORKAROUNDS
There are no workarounds for affected systems. For new installations using the above setup users may either apply a vendor-supplied patch to their Linux kernel, use IDE/SATA disks instead of SCSI, or avoid the use of LVM.

AFFECTED VERSIONS
MongoDB production releases up to 2.6.4 are affected by this issue.

FIX VERSION
MongoDB versions 2.6.5 and 2.4.12 include a workaround for this issue. This workaround does not fix .ns files that are already corrupted.

RESOLUTION DETAILS
Make mongod manually allocate and zero .ns files.

Original description

Certain versions of Linux may under some conditions use SCSI WRITE SAME to implement the requirement to zero a file on initial allocation. However, certain platforms do not properly implement WRITE SAME. The result is that the initial zeroing of a file may be unreliable on those platforms, causing potentially fatal corruption in the .ns file, which depends on the initial zeroing of the file to initialize the hash table. As a defensive measure mongod could explicitly zero the .ns files on creation.



 Comments   
Comment by Githook User [ 29/Sep/14 ]

Author:

{u'username': u'monkey101', u'name': u'Dan Pasette', u'email': u'dan@10gen.com'}

Message: SERVER-15369: Clean up partially created .ns files

(cherry picked from commit 243f11cd82d7925c77df0b04fad58b18f70cf031)
Branch: v2.4
https://github.com/mongodb/mongo/commit/e4a5649bb5bed607e0e907ce6f2d08e607dcde12

Comment by Githook User [ 29/Sep/14 ]

Author:

{u'username': u'monkey101', u'name': u'Dan Pasette', u'email': u'dan@10gen.com'}

Message: SERVER-15369: Clean up partially created .ns files

(cherry picked from commit 689c0faf75e0cc3f1663b962c3db55360e0ef3b5)

Conflicts:
jstests/dur/diskfull.js
Branch: v2.6
https://github.com/mongodb/mongo/commit/243f11cd82d7925c77df0b04fad58b18f70cf031

Comment by Githook User [ 29/Sep/14 ]

Author:

{u'username': u'monkey101', u'name': u'Dan Pasette', u'email': u'dan@10gen.com'}

Message: SERVER-15369: Clean up partially created .ns files
Branch: master
https://github.com/mongodb/mongo/commit/689c0faf75e0cc3f1663b962c3db55360e0ef3b5

Comment by Githook User [ 25/Sep/14 ]

Author:

{u'name': u'Dan Pasette', u'email': u'dan@10mongodb.com'}

Message: SERVER-15369 Explicitly write zeros when creating ns files

Backported from commit b39f5ccc3e5c74918c8f2cd46d33c5b6316f4b06
Branch: v2.4
https://github.com/mongodb/mongo/commit/66bfc5b6fbb64e03cfb80c1c830ac88b1d3ec5ba

Comment by Githook User [ 25/Sep/14 ]

Author:

{u'name': u'Dan Pasette', u'email': u'dan@10mongodb.com'}

Message: SERVER-15369 Explicitly write zeros when creating ns files
(cherry picked from commit 60d379f9e0c3eaf85850e768328d94c49170a7d7)

Conflicts:
src/mongo/db/structure/catalog/namespace_index.cpp
Branch: v2.6
https://github.com/mongodb/mongo/commit/b39f5ccc3e5c74918c8f2cd46d33c5b6316f4b06

Comment by Githook User [ 25/Sep/14 ]

Author:

{u'name': u'Dan Pasette', u'email': u'dan@10mongodb.com'}

Message: SERVER-15369 Explicitly write zeros when creating ns files
Branch: master
https://github.com/mongodb/mongo/commit/60d379f9e0c3eaf85850e768328d94c49170a7d7

Generated at Thu Feb 08 03:37:49 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.