[SERVER-34865] Test archival fails when temporary files are removed Created: 07/May/18  Updated: 29/Oct/23  Resolved: 16/May/18

Status: Closed
Project: Core Server
Component/s: Testing Infrastructure
Affects Version/s: None
Fix Version/s: 3.4.16, 3.6.6, 4.0.0-rc0

Type: Bug Priority: Major - P3
Reporter: Jonathan Abrahams Assignee: Jonathan Abrahams
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
Depends
Backwards Compatibility: Fully Compatible
Operating System: ALL
Backport Requested:
v3.6, v3.4
Sprint: TIG 2018-05-21
Participants:
Linked BF Score: 0
Story Points: 2

 Description   

The following error occurred when archiving a failed test:

[2018/05/07 11:27:26.351] [executor:fsm_workload_test:job0] 2018-05-07T15:27:26.349+0000 Archiving data files for test jstests/concurrency/fsm_workloads/yield_group.js from /data/db/job0/resmoke
[2018/05/07 11:27:26.356] [executor:fsm_workload_test:job0] 2018-05-07T15:27:26.355+0000 Encountered an error during test execution.
[2018/05/07 11:27:26.356] Traceback (most recent call last):
[2018/05/07 11:27:26.356]   File "/data/mci/6612d9aa5374fb14abe6c091b3ffcf03/src/buildscripts/resmokelib/testing/job.py", line 45, in __call__
[2018/05/07 11:27:26.356]     self._run(queue, interrupt_flag)
[2018/05/07 11:27:26.356]   File "/data/mci/6612d9aa5374fb14abe6c091b3ffcf03/src/buildscripts/resmokelib/testing/job.py", line 83, in _run
[2018/05/07 11:27:26.356]     self._execute_test(test)
[2018/05/07 11:27:26.356]   File "/data/mci/6612d9aa5374fb14abe6c091b3ffcf03/src/buildscripts/resmokelib/testing/job.py", line 113, in _execute_test
[2018/05/07 11:27:26.356]     self.archival.archive(self.logger, test, success)
[2018/05/07 11:27:26.356]   File "/data/mci/6612d9aa5374fb14abe6c091b3ffcf03/src/buildscripts/resmokelib/testing/hook_test_archival.py", line 78, in archive
[2018/05/07 11:27:26.356]     self._archive_test(logger, test, success)
[2018/05/07 11:27:26.356]   File "/data/mci/6612d9aa5374fb14abe6c091b3ffcf03/src/buildscripts/resmokelib/testing/hook_test_archival.py", line 69, in _archive_test
[2018/05/07 11:27:26.356]     self._archive_hook_or_test(logger, test_name, test)
[2018/05/07 11:27:26.356]   File "/data/mci/6612d9aa5374fb14abe6c091b3ffcf03/src/buildscripts/resmokelib/testing/hook_test_archival.py", line 105, in _archive_hook_or_test
[2018/05/07 11:27:26.356]     s3_bucket, s3_path)
[2018/05/07 11:27:26.356]   File "/data/mci/6612d9aa5374fb14abe6c091b3ffcf03/src/buildscripts/resmokelib/utils/archival.py", line 157, in archive_files_to_s3
[2018/05/07 11:27:26.356]     s3_bucket, s3_path)
[2018/05/07 11:27:26.356]   File "/data/mci/6612d9aa5374fb14abe6c091b3ffcf03/src/buildscripts/resmokelib/utils/archival.py", line 245, in _archive_files
[2018/05/07 11:27:26.356]     if file_list_size(input_files) > free_space(temp_file):
[2018/05/07 11:27:26.356]   File "/data/mci/6612d9aa5374fb14abe6c091b3ffcf03/src/buildscripts/resmokelib/utils/archival.py", line 39, in file_list_size
[2018/05/07 11:27:26.356]     file_bytes += directory_size(ifile)
[2018/05/07 11:27:26.356]   File "/data/mci/6612d9aa5374fb14abe6c091b3ffcf03/src/buildscripts/resmokelib/utils/archival.py", line 52, in directory_size
[2018/05/07 11:27:26.356]     dir_bytes += os.path.getsize(full_name)
[2018/05/07 11:27:26.356]   File "/data/mci/6612d9aa5374fb14abe6c091b3ffcf03/venv/lib/python2.7/genericpath.py", line 57, in getsize
[2018/05/07 11:27:26.356]     return os.stat(filename).st_size
[2018/05/07 11:27:26.356] OSError: [Errno 2] No such file or directory: '/data/db/job0/resmoke/shard1/node0/WiredTiger.turtle.set'
 

The code doe not handle the case where a temporary file is in a directory list and then subsequently deleted before it is examined:

def directory_size(directory):
    """Return size (in bytes) of files in 'directory' tree."""
    dir_bytes = 0
    for root_dir, _, files in os.walk(unicode(directory)):
        for name in files:
            full_name = os.path.join(root_dir, name)
            try:
                dir_bytes += os.path.getsize(full_name)
            except OSError:
                # Symlinks generate an error and are ignored.
                if os.path.islink(full_name):
                    pass
                else:
                    raise
    return dir_bytes

The OSError should handle this case.



 Comments   
Comment by Githook User [ 24/May/18 ]

Author:

{'username': 'hptabster', 'name': 'Jonathan Abrahams', 'email': 'jonathan@mongodb.com'}

Message: SERVER-34865 Test archival fails when temporary files are removed

(cherry picked from commit 0a03faba456a3acbbaabd6ad5694f98acfdf50bd)
Branch: v3.4
https://github.com/mongodb/mongo/commit/67bc1f44ec5be1564d549ff69a571a33f3cbdb99

Comment by Githook User [ 24/May/18 ]

Author:

{'username': 'hptabster', 'name': 'Jonathan Abrahams', 'email': 'jonathan@mongodb.com'}

Message: SERVER-34865 Test archival fails when temporary files are removed

(cherry picked from commit 0a03faba456a3acbbaabd6ad5694f98acfdf50bd)
Branch: v3.6
https://github.com/mongodb/mongo/commit/a6ced744cc89a148e6e37641e31c91390f42dfa7

Comment by Githook User [ 16/May/18 ]

Author:

{'email': 'jonathan@mongodb.com', 'username': 'hptabster', 'name': 'Jonathan Abrahams'}

Message: SERVER-34865 Test archival fails when temporary files are removed
Branch: master
https://github.com/mongodb/mongo/commit/0a03faba456a3acbbaabd6ad5694f98acfdf50bd

Generated at Thu Feb 08 04:38:06 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.