Uploaded image for project: 'WiredTiger'
  1. WiredTiger
  2. WT-11906

Testy failed with "Too many open files" (EMFILE err 24)

    • Type: Icon: Bug Bug
    • Resolution: Unresolved
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: None
    • Component/s: testy
    • Labels:
      None
    • Storage Engines
    • 5
    • StorEng - Defined Pipeline

      These are the logs:

      Oct 30 21:38:26 ip-172-31-12-172 systemd[1]: Started "testy-run: A 24/7 WiredTiger workload service".
      Oct 30 22:38:28 ip-172-31-12-172 bash[57985]: Warning: Unable to identify WiredTiger build. Please consider either:
      Oct 30 22:38:28 ip-172-31-12-172 bash[57985]: - Setting 'WT_BUILDDIR' environment variable to specify the build directory.
      Oct 30 22:38:28 ip-172-31-12-172 bash[57985]: - Calling workgen from the root of the build directory.
      Oct 30 22:38:28 ip-172-31-12-172 bash[57985]: [1698705508:66699][57985:0x7f26d3fff640], sweep-server: [WT_VERB_DEFAULT][WARNING]: Session 18 (@: 0x0x55ba76f8afd0 name: EMPTY) did not run a sweep for 60 minutes.
      Oct 30 22:38:28 ip-172-31-12-172 bash[57985]: [1698705508:66794][57985:0x7f26d3fff640], sweep-server: [WT_VERB_DEFAULT][WARNING]: Session 57 (@: 0x0x55ba76f96cb8 name: EMPTY) did not run a sweep for 60 minutes.
      Oct 31 01:59:05 ip-172-31-12-172 bash[57985]: [1698717545:935493][57985:0x7f2663fff640], file:table_MX2QE.wt, WT_SESSION.open_cursor: [WT_VERB_DEFAULT][ERROR]: __posix_open_file, 814: /srv/testy/data/table_MX2QE.wt: handle-open: open: Too many open files
      Oct 31 01:59:05 ip-172-31-12-172 bash[57985]: thread thread24 failed err=24
      Oct 31 01:59:06 ip-172-31-12-172 bash[57985]: [1698717546:437904][57985:0x7f26617fa640], file:table_zxIou.wt, WT_SESSION.open_cursor: [WT_VERB_DEFAULT][ERROR]: __posix_open_file, 814: /srv/testy/data/table_zxIou.wt: handle-open: open: Too many open files
      Oct 31 01:59:06 ip-172-31-12-172 bash[57985]: thread thread29 failed err=24
      Oct 31 01:59:06 ip-172-31-12-172 bash[57985]: [1698717546:634980][57985:0x7f26d37fe640], file:WiredTiger.wt, WT_SESSION.compact: [WT_VERB_DEFAULT][ERROR]: __posix_open_file, 814: /srv/testy/data/WiredTiger.turtle: handle-open: open: Too many open files
      Oct 31 01:59:06 ip-172-31-12-172 bash[57985]: [1698717546:635137][57985:0x7f26d37fe640], file:WiredTigerHS.wt, WT_SESSION.compact: [WT_VERB_DEFAULT][ERROR]: __wt_block_checkpoint_resolve, 1007: WiredTigerHS.wt: the checkpoint failed, the system must restart: Invalid argument
      Oct 31 01:59:06 ip-172-31-12-172 bash[57985]: [1698717546:635161][57985:0x7f26d37fe640], file:WiredTigerHS.wt, WT_SESSION.compact: [WT_VERB_DEFAULT][ERROR]: __wt_block_checkpoint_resolve, 1007: the process must exit and restart: WT_PANIC: WiredTiger library panic
      Oct 31 01:59:06 ip-172-31-12-172 bash[57985]: [1698717546:635174][57985:0x7f26d37fe640], file:WiredTigerHS.wt, WT_SESSION.compact: [WT_VERB_DEFAULT][ERROR]: __wt_abort, 28: aborting WiredTiger library
      Oct 31 01:59:06 ip-172-31-12-172 bash[57971]: /srv/testy/workloads/sample/sample.sh: line 12: 57985 Aborted                 (core dumped) python3 ${workload_dir}/sample/sample_run.py --home ${database_dir} --keep
      

      The Workgen workload used is attached: workgen_workload.py
      FTDC: WiredTigerStat.zip
      Commit used by Testy:

      commit a864c73b49d3dca43da81629256f78b6ec4afee8 (HEAD -> develop, origin/develop, origin/HEAD)
      Author: Will Korteland <korteland@users.noreply.github.com>
      Date:   Mon Oct 30 11:39:01 2023 +1100
      
          WT-11872 Correctly free resources in chunk cache metadata initialisation error scenarios (#9803)
      

        1. ftdc_files_open.png
          ftdc_files_open.png
          112 kB
        2. WiredTigerStat.zip
          417 kB
        3. workgen_workload.py
          4 kB

            Assignee:
            backlog-server-storage-engines [DO NOT USE] Backlog - Storage Engines Team
            Reporter:
            etienne.petrel@mongodb.com Etienne Petrel
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated: