Uploaded image for project: 'WiredTiger'
  1. WiredTiger
  2. WT-262

dumping complex tables fails with key/value not set

    • Type: Icon: Task Task
    • Resolution: Done
    • WT1.3
    • Affects Version/s: None
    • Component/s: None

      Michael, dumping relatively complex tables doesn't work because the key/value pair isn't set at the right level.

      In summary, we have:

      WT_CURSOR/WT_CURSOR_DUMP ->
          WT_CURSOR/WT_CURSOR_TABLE.cg_cursors[] ->
              WT_CURSOR/WT_CURSOR_DUMP ->
                  WT_CURSOR
      

      and it's only at the very bottom WT_CURSOR where WT_CURSTD_KEY/VALUE_SET are set.

      When we call WT_CURSOR.get_key, it looks for WT_CURSTD_KEY/VALUE_SET in whatever is referenced by the cg_cursors[] array cursors, which are the dump cursors, not the underlying object cursors.

      I'm guessing we either need to migrate WT_CURSTD_KEY/VALUE_SET up into the higher-level cursors somehow, or do our checks for WT_CURSTD_KEY/VALUE_SET being set at a lower level, but wanted to hear what you thought before I started whacking code.

      Here's a chunk of Python test code that builds a table you can't dump.

          # Build a complex table with multiple files.
          def complexPopulate(self):
              name = self.uri + self.name
              self.session.create(name,
                  'leaf_page_max=1024,key_format=i,value_format=SiSS,' +
                  'columns=(record,column2,column3,column4,column5),' +
                  'colgroups=(cgroup1,cgroup2,cgroup3,cgroup4,cgroup5,cgroup6)')
              cgname = 'colgroup:' + self.name
              self.session.create(cgname + ':cgroup1', 'columns=(column2)')
              self.session.create(cgname + ':cgroup2', 'columns=(column3)')
              self.session.create(cgname + ':cgroup3', 'columns=(column4)')
              self.session.create(cgname + ':cgroup4', 'columns=(column2,column3)')
              self.session.create(cgname + ':cgroup5', 'columns=(column3,column4)')
              self.session.create(cgname + ':cgroup6', 'columns=(column4,column5)')
              cursor = self.session.open_cursor(name, None, None)
              for i in range(0, self.nentries):
                      cursor.set_key(i)
                      cursor.set_value(
                          str(i) + ': abcdefghijklmnopqrstuvwxyz'[0:i%26],
                          i,
                          str(i) + ': abcdefghijklmnopqrstuvwxyz'[0:i%23],
                          str(i) + ': abcdefghijklmnopqrstuvwxyz'[0:i%18])
                      cursor.insert()
              cursor.close()
      

      Here's a long stack-trace of the cursors when WT_CURSOR.next is returning in the dump function, it shows the problem (feel free to ignore it, but I didn't want to lose it, it took me long enough to figure it out).

      `
      (gdb) where
      #0 dump_forward (cursor=0x800eb6b00, name=0x800e2b720 "table:test_truncate")
      at ../src/utilities/util_dump.c:25
      WT-1 0x000000000040220c in util_dump (session=0x800e213e0, argc=1,
      argv=0x7fffffffe8b0) at ../src/utilities/util_dump.c:121
      WT-2 0x0000000000404e93 in main (argc=2, argv=0x7fffffffe8a8)
      at ../src/utilities/util_main.c:121
      (gdb) whatis cursor
      type = WT_CURSOR *
      (gdb) p *cursor
      $25 = {session = 0x800e213e0, uri = 0x800e2b5e0 "table:test_truncate",
      key_format = 0x800e1e17c "i", value_format = 0x800eb7240 "SiSS",
      get_key = 0x800696660 <__curdump_get_key>,
      get_value = 0x800696f30 <__curdump_get_value>,
      set_key = 0x800696b80 <__curdump_set_key>,
      set_value = 0x800697270 <__curdump_set_value>,
      equals = 0x80069c470 <*cursor_equals>, next = 0x800697590 <*curdump_next>,
      prev = 0x8006975d0 <*curdump_prev>, reset = 0x800697610 <*curdump_reset>,
      search = 0x800697650 <__curdump_search>,
      search_near = 0x800697690 <__curdump_search_near>,
      insert = 0x8006976d0 <__curdump_insert>,
      update = 0x800697710 <__curdump_update>,
      remove = 0x800697750 <__curdump_remove>,
      close = 0x800697790 <__curdump_close>,
      reconfigure = 0x80069b260 <__wt_cursor_notsup>, q =

      {tqe_next = 0x800eb6c40, tqe_prev = 0x800e214a8}

      , recno = 0,
      raw_recno_buf = "\000\000\000\000\000\000\000\000", key =

      {data = 0x0, size = 0, flags = 0, mem = 0x0, memsize = 0}, value = {data = 0x0, size = 0, flags = 0, mem = 0x0, memsize = 0}

      , saved_err = 0, flags = 20}
      (gdb) p _(WT_CURSOR_DUMP _)cursor
      $27 = {iface = {session = 0x800e213e0,
      uri = 0x800e2b5e0 "table:test_truncate", key_format = 0x800e1e17c "i",
      value_format = 0x800eb7240 "SiSS",
      get_key = 0x800696660 <__curdump_get_key>,
      get_value = 0x800696f30 <__curdump_get_value>,
      set_key = 0x800696b80 <__curdump_set_key>,
      set_value = 0x800697270 <__curdump_set_value>,
      equals = 0x80069c470 <__cursor_equals>,
      next = 0x800697590 <*curdump_next>, prev = 0x8006975d0 <*curdump_prev>,
      reset = 0x800697610 <__curdump_reset>,
      search = 0x800697650 <__curdump_search>,
      search_near = 0x800697690 <__curdump_search_near>,
      insert = 0x8006976d0 <__curdump_insert>,
      update = 0x800697710 <__curdump_update>,
      remove = 0x800697750 <__curdump_remove>,
      close = 0x800697790 <__curdump_close>,
      reconfigure = 0x80069b260 <__wt_cursor_notsup>, q =

      { tqe_next = 0x800eb6c40, tqe_prev = 0x800e214a8}

      , recno = 0,
      raw_recno_buf = "\000\000\000\000\000\000\000\000", key =

      {data = 0x0, size = 0, flags = 0, mem = 0x0, memsize = 0}, value = {data = 0x0, size = 0, flags = 0, mem = 0x0, memsize = 0}

      , saved_err = 0,
      flags = 20}, child = 0x800eb6c40}
      (gdb) p _(WT_CURSOR_TABLE _)$27.child
      $29 = {iface = {session = 0x800e213e0,
      uri = 0x800e2b5e0 "table:test_truncate", key_format = 0x800e1e17c "i",
      value_format = 0x800eb7240 "SiSS",
      get_key = 0x80069d0d0 <__wt_curtable_get_key>,
      get_value = 0x80069d200 <__wt_curtable_get_value>,
      set_key = 0x80069d520 <__wt_curtable_set_key>,
      set_value = 0x80069d730 <__wt_curtable_set_value>,
      equals = 0x80069c470 <__cursor_equals>,
      next = 0x80069dac0 <__curtable_next>,
      prev = 0x80069dd70 <__curtable_prev>,
      reset = 0x80069de80 <__curtable_reset>,
      search = 0x80069df90 <__curtable_search>,
      search_near = 0x80069e0a0 <__curtable_search_near>,
      insert = 0x80069e220 <__curtable_insert>,
      update = 0x80069e490 <__curtable_update>,
      remove = 0x80069e8d0 <__curtable_remove>,
      close = 0x80069eb70 <__curtable_close>,
      reconfigure = 0x80069c7f0 <__cursor_reconfigure>, q =

      { tqe_next = 0x800eb6380, tqe_prev = 0x800eb6b98}

      , recno = 0,
      raw_recno_buf = "\000\000\000\000\000\000\000\000", key =

      {data = 0x0, size = 0, flags = 0, mem = 0x0, memsize = 0}, value = {data = 0x0, size = 0, flags = 0, mem = 0x0, memsize = 0}

      , saved_err = 0,
      flags = 20}, table = 0x800e2adc0, plan = 0x800e2b0c0 "0vn1vn2vn5vsn",
      cg_cursors = 0x800e20610, idx_cursors = 0x0}
      (gdb) p *$29.cg_cursors[0]
      $31 = {session = 0x800e213e0, 
        uri = 0x800e2b580 "file:test_truncate_cgroup1.wt", 
        key_format = 0x800e1e17a "i", value_format = 0x800e1e178 "S", 
        get_key = 0x800696660 <__curdump_get_key>, 
        get_value = 0x800696f30 <__curdump_get_value>, 
        set_key = 0x800696b80 <__curdump_set_key>, 
        set_value = 0x800697270 <__curdump_set_value>, 
        equals = 0x80069c470 <*cursor_equals>, next = 0x800697590 <*curdump_next>, 
        prev = 0x8006975d0 <*curdump_prev>, reset = 0x800697610 <*curdump_reset>, 
        search = 0x800697650 <__curdump_search>, 
        search_near = 0x800697690 <__curdump_search_near>, 
        insert = 0x8006976d0 <__curdump_insert>, 
        update = 0x800697710 <__curdump_update>, 
        remove = 0x800697750 <__curdump_remove>, 
        close = 0x800697790 <__curdump_close>, 
        reconfigure = 0x80069b260 <__wt_cursor_notsup>, q =

      {tqe_next = 0x800e51c00,      tqe_prev = 0x800e51298}

      , recno = 0, 
        raw_recno_buf = "\000\000\000\000\000\000\000\000", key =

      {data = 0x0,      size = 0, flags = 0, mem = 0x0, memsize = 0}, value = {data = 0x0,     size = 0, flags = 0, mem = 0x0, memsize = 0}

      , saved_err = 0, flags = 20}
      (gdb) p _(WT_CURSOR_DUMP _)$29.cg_cursors[0]
      $32 = {iface = {session = 0x800e213e0,
      uri = 0x800e2b580 "file:test_truncate_cgroup1.wt",
      key_format = 0x800e1e17a "i", value_format = 0x800e1e178 "S",
      get_key = 0x800696660 <__curdump_get_key>,
      get_value = 0x800696f30 <__curdump_get_value>,
      set_key = 0x800696b80 <__curdump_set_key>,
      set_value = 0x800697270 <__curdump_set_value>,
      equals = 0x80069c470 <__cursor_equals>,
      next = 0x800697590 <*curdump_next>, prev = 0x8006975d0 <*curdump_prev>,
      reset = 0x800697610 <__curdump_reset>,
      search = 0x800697650 <__curdump_search>,
      search_near = 0x800697690 <__curdump_search_near>,
      insert = 0x8006976d0 <__curdump_insert>,
      update = 0x800697710 <__curdump_update>,
      remove = 0x800697750 <__curdump_remove>,
      close = 0x800697790 <__curdump_close>,
      reconfigure = 0x80069b260 <__wt_cursor_notsup>, q =

      { tqe_next = 0x800e51c00, tqe_prev = 0x800e51298}

      , recno = 0,
      raw_recno_buf = "\000\000\000\000\000\000\000\000", key =

      {data = 0x0, size = 0, flags = 0, mem = 0x0, memsize = 0}, value = {data = 0x0, size = 0, flags = 0, mem = 0x0, memsize = 0}

      , saved_err = 0,
      flags = 20}, child = 0x800e51c00}
      (gdb) p *$32.child
      $34 = {session = 0x800e213e0,
      uri = 0x800e2b580 "file:test_truncate_cgroup1.wt",
      key_format = 0x800e1e17a "i", value_format = 0x800e1e178 "S",
      get_key = 0x80069b320 <__wt_cursor_get_key>,
      get_value = 0x80069b6c0 <__wt_cursor_get_value>,
      set_key = 0x80069b910 <__wt_cursor_set_key>,
      set_value = 0x80069bee0 <__wt_cursor_set_value>,
      equals = 0x80069c470 <*cursor_equals>, next = 0x8006979e0 <*curfile_next>,
      prev = 0x800697b60 <*curfile_prev>, reset = 0x800697c20 <*curfile_reset>,
      search = 0x800697ce0 <__curfile_search>,
      search_near = 0x800697dc0 <__curfile_search_near>,
      insert = 0x800697eb0 <__curfile_insert>,
      update = 0x800698000 <__curfile_update>,
      remove = 0x800698140 <__curfile_remove>,
      close = 0x800698250 <__curfile_close>,
      reconfigure = 0x80069c7f0 <__cursor_reconfigure>, q =

      {tqe_next = 0x0, tqe_prev = 0x800eb6a58}

      , recno = 0,
      raw_recno_buf = "\000\000\000\000\000\000\000\000", key =

      { data = 0x800ebb026, size = 1, flags = 0, mem = 0x0, memsize = 0}

      , value =

      { data = 0x800ebb028, size = 2, flags = 0, mem = 0x0, memsize = 0}

      ,
      saved_err = 0, flags = 156}

            Assignee:
            michael.cahill@mongodb.com Michael Cahill (Inactive)
            Reporter:
            keith.bostic@mongodb.com Keith Bostic (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated:
              Resolved: