Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-21666

Small stall during concurrent insert/update

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Minor - P4
    • Resolution: Incomplete
    • Affects Version/s: 3.0.1, 3.0.2, 3.0.3, 3.0.4, 3.0.5, 3.0.6
    • Fix Version/s: None
    • Component/s: WiredTiger
    • Labels:
      None
    • Operating System:
      ALL
    • Steps To Reproduce:
      Hide

      Run attatched java test program

      Show
      Run attatched java test program

      Description

      In some test case, Non periodic small stall is observed.

      First time, I have thought wiredtiger checkpoint cause this small stall.
      But, it's nothing to do with checkpoint (After some test and code eval, I found that wiredtiger table lock could cause this stall).
      And this small stall makes a lot of slow query(over 150ms).
      During this stall period, __wt_spin_lock(pthread_mutex_lock) took over 4~8 milliseconds.

      __wt_schema_open_table :: elapsed 21 us
      __wt_spin_lock :: elapsed:8394 us (pthread_mutex_lock :: session.open_cursor -> table creation(0))
      __wt_schema_add_table :: elapsed 8436 us (need table lock)
      __wt_schema_get_table :: elapsed 8437 us
      __wt_curtable_open :: elapsed 8439 us
      __wt_open_cursor :: elapsed 8456 us

      My test case is really simple, Just make 1000 client thread and doing update(upsert:true) and total documents is about 400000.
      And server have processed about 6~7000 update/sec. System load was under 1.0

      12 core cpu, 32GB memory, 2.6.32-573.7.1.el6.centos.plus.x86_64
      Looks like disk is nothing to do with this issue (I've tested NVME, SAS+RAID, SATA)

      Current implementation, table lock is implemented with WT_SPINLOCK.
      What do you think about changing it as WT_RWLOCK ?

        Attachments

        1. mongo_config_table_stats.txt
          6 kB
        2. slow-queries.png
          slow-queries.png
          69 kB
        3. SmallStall.tar.gz
          6 kB

          Activity

            People

            Assignee:
            david.hows David Hows
            Reporter:
            matt.lee Matt SeongUck Lee
            Participants:
            Votes:
            0 Vote for this issue
            Watchers:
            11 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: