Uploaded image for project: 'MongoDB Database Tools'
  1. MongoDB Database Tools
  2. TOOLS-1569

mongo-replay crashes during playback

    • Type: Icon: Bug Bug
    • Resolution: Done
    • Priority: Icon: Major - P3 Major - P3
    • 3.5.2, 3.4.2
    • Affects Version/s: None
    • Component/s: mongoreplay
    • None
    • Platforms 2017-01-23

      I am using mongoreplay to playback production traffic, and it crashed with this trace.

      [ec2-user@ip-10-2-0-98 ~]$ LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/media/ephemeral0/ ./mongoreplay play -p /media/ephemeral0/playback.bson --host mongodb://10.2.0.210:27017 --no-preprocess
      2016/12/08 01:57:35 Doing playback at 1.00x speed
      panic: runtime error: invalid memory address or nil pointer dereference
      [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x5c95a2]
      
      goroutine 1771 [running]:
      panic(0x73e620, 0xc4200140b0)
      	/opt/go/src/runtime/panic.go:500 +0x1a1
      github.com/10gen/llmgo.(*MongoSocket).SimpleQuery.func1(0xb913a0, 0xc420014150, 0x0, 0x0)
      	/data/mci/aec3fed01f1c23fbbb82149b78096a79/src/vendor/src/github.com/10gen/llmgo/socket.go:449 +0xc2
      github.com/10gen/llmgo.(*MongoSocket).kill(0xc421912b60, 0xb913a0, 0xc420014150, 0x1)
      	/data/mci/aec3fed01f1c23fbbb82149b78096a79/src/vendor/src/github.com/10gen/llmgo/socket.go:431 +0x492
      github.com/10gen/llmgo.(*MongoSocket).readLoop(0xc421912b60)
      	/data/mci/aec3fed01f1c23fbbb82149b78096a79/src/vendor/src/github.com/10gen/llmgo/socket.go:677 +0x13d2
      created by github.com/10gen/llmgo.NewSocket
      	/data/mci/aec3fed01f1c23fbbb82149b78096a79/src/vendor/src/github.com/10gen/llmgo/socket.go:277 +0x278
      
      

      I was concern this may related to ulimit for fd, but it is set to 20k

      [ec2-user@ip-10-2-0-98 ~]$ ulimit -n
      20000
      

      and on mongod, the number of connections peaked at 230

      insert query update delete getmore command dirty  used flushes vsize   res qrw arw net_in net_out conn           set repl                time
          *0    *0     *0     *0       0    96|0  0.0% 80.0%       0 23.1G 22.2G 0|0 0|0  78.1k   80.4k  230 logkeeper-rs0  PRI Dec  8 01:58:32.216
      

      From stack, seems related to driver? The old mongotape build I tried works iirc.

            Assignee:
            gabriel.russell@mongodb.com Gabriel Russell (Inactive)
            Reporter:
            rui.zhang Rui Zhang (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: