[CDRIVER-591] Double free or corruption error while running topology tests Created: 24/Mar/15  Updated: 17/Aug/15  Resolved: 02/Jul/15

Status: Closed
Project: C Driver
Component/s: None
Affects Version/s: None
Fix Version/s: 1.2-beta0

Type: Bug Priority: Major - P3
Reporter: Jeremy Mikola Assignee: Unassigned
Resolution: Cannot Reproduce Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   

I came across this while running the test suite for CDRIVER-590. The commit head (I had yet to stage any commits for URI fixes) was 9293ec1. I was running tests against my localhost mongod (v3.0.0), and I've not been able to reproduce this. It simply happened during a test run and I caught the backtrace:

    { "status": "PASS", "name": "/Topology/client_creation", "seed": "3726736168", "elapsed": 0.000395546 },
    { "status": "PASS", "name": "/Topology/client_pool_creation", "seed": "1267060720", "elapsed": 0.000108615 },
    { "status": "PASS", "name": "/Topology/invalidate_server", "seed": "2029659797", "elapsed": 0.000272005 },
    { "status": "PASS", "name": "/Topology/invalid_cluster_node", "seed": "2590074792", "elapsed": 0.000816252 },
*** Error in `/home/jmikola/workspace/mongodb/libmongoc/.libs/lt-test-libmongoc': double free or corruption (!prev): 0x00007f1904000b60 ***
/bin/bash: line 1:  8373 Aborted                 (core dumped) ./$TEST_PROG -f -p -F test.log
Makefile:4587: recipe for target 'test' failed
make: *** [test] Error 134

Full backtrace:

(gdb) bt full
#0  0x00007f196b19be37 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
        resultvar = 0
        pid = 8373
        selftid = 8373
#1  0x00007f196b19d528 in __GI_abort () at abort.c:89
        save_stage = 2
        act = {__sigaction_handler = {sa_handler = 0x0, sa_sigaction = 0x0}, sa_mask = {__val = {0 <repeats 16 times>}}, sa_flags = 0, sa_restorer = 0x0}
        sigs = {__val = {32, 0 <repeats 15 times>}}
#2  0x00007f196b1dcf74 in __libc_message (do_abort=do_abort@entry=1, fmt=fmt@entry=0x7f196b2e5f00 "*** Error in `%s': %s: 0x%s ***\n") at ../sysdeps/posix/libc_fatal.c:175
        ap = {{gp_offset = 40, fp_offset = 32767, overflow_arg_area = 0x7fff19b91610, reg_save_area = 0x7fff19b915a0}}
        fd = 32
        on_2 = <optimized out>
        list = <optimized out>
        nlist = <optimized out>
        cp = <optimized out>
        written = <optimized out>
#3  0x00007f196b1e4586 in malloc_printerr (ptr=<optimized out>, str=0x7f196b2e5fe8 "double free or corruption (!prev)", action=1) at malloc.c:4996
        buf = "00007f1904000b60"
        cp = <optimized out>
#4  _int_free (av=<optimized out>, p=<optimized out>, have_lock=0) at malloc.c:3840
        size = <optimized out>
        fb = <optimized out>
        nextchunk = <optimized out>
        nextsize = <optimized out>
        nextinuse = <optimized out>
        prevsize = <optimized out>
        bck = <optimized out>
        fwd = <optimized out>
        errstr = <optimized out>
        locked = <optimized out>
#5  0x00007f196b7657d2 in bson_free (mem=0x7f1904000b60) at src/bson/bson-memory.c:216
No locals.
#6  0x00007f196b759d51 in bson_destroy (bson=0x2210370) at src/bson/bson.c:2153
        __PRETTY_FUNCTION__ = "bson_destroy"
#7  0x00007f196bfa227f in mongoc_server_description_reset (sd=0x2210270) at src/mongoc/mongoc-server-description.c:76
        __PRETTY_FUNCTION__ = "mongoc_server_description_reset"
#8  0x0000000000422518 in test_max_wire_version_race_condition () at tests/test-mongoc-topology.c:254
        scanner_node = 0x2219570
        sd = 0x2210270
        database = 0x22101b0
        pool = 0x220de10
        client = 0x2219cc0
        stream = 0x0
        error = {domain = 2, code = 6, 
          message = "No stream available for server_id 1\000\000\000\000\000@\353\"\002", '\000' <repeats 20 times>, "|\202\036k\031\177\000\000\000\000\000\000\000\000\000\000\265 \000\000\000\000\000\000\220\024\271\031\377\177\000\000`\200@\000\000\000\000\000 \035\271\031\377\177\000\000\000\025\271\031\377\177\000\000p\026\271\031\377\177\000\000FN\370k\031\177\000\000@\353\"\002\
000\000\000\000\034\227uk\031\177\000\000\360\002!\002\000\000\000\000P\335 \002\000\000\000\000\340\024\271\031\377\177\000\000\355\375$k\031\177\000\000\001\000\000\000\377\177\000\000\021\070\370k\031\177\000\000"...}
        uri = 0x2218ec0
        id = 1
        r = 1
        __FUNCTION__ = "test_max_wire_version_race_condition"
        __PRETTY_FUNCTION__ = "test_max_wire_version_race_condition"
#9  0x0000000000426dea in TestSuite_AddHelper (cb_=0x4222f0 <test_max_wire_version_race_condition>) at tests/TestSuite.c:270
        cb = 0x4222f0 <test_max_wire_version_race_condition>
#10 0x0000000000427162 in TestSuite_RunTest (suite=0x7fff19b91bf0, test=0x220cf00, mutex=0x7fff19b91b70, count=0x7fff19b91b64) at tests/TestSuite.c:393
        ts1 = {tv_sec = 76134, tv_nsec = 200084928}
        ts2 = {tv_sec = 76134, tv_nsec = 200059915}
        ts3 = {tv_sec = 0, tv_nsec = 816252}
        name = "/Topology/max_wire_version_race_condition\000read/SecondaryPreferr"
        buf = "    { \"status\": \"PASS\", \"name\": \"/Topology/invalid_cluster_node\", \"seed\": \"2590074792\", \"elapsed\": 0.000816252 },\n\000\071\067\067\", \"elapsed\": 0.000012577 },\n", '\000' <repeats 16 times>, "#43-UЫRk\031\177\000\000\265 \000\000\000\000\000\000\240\033\271\031\377\177\000\000`\200@\000\000\000\000\000\065"...
        status = 0
#11 0x000000000042782b in TestSuite_RunSerial (suite=0x7fff19b91bf0) at tests/TestSuite.c:652
        test = 0x220cf00
        mutex = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __elision = 0, __list = {__prev = 0x0, __next = 0x0}}, 
          __size = '\000' <repeats 39 times>, __align = 0}
        count = 20
#12 0x0000000000427aed in TestSuite_Run (suite=0x7fff19b91bf0) at tests/TestSuite.c:715
No locals.
#13 0x0000000000409f78 in main (argc=5, argv=0x7fff19b91d28) at tests/test-libmongoc.c:170
        suite = {prgname = 0x21ff990 "/home/jmikola/workspace/mongodb/libmongoc/.libs/lt-test-libmongoc", name = 0x21fff50 "", testname = 0x0, tests = 0x21ff880, outfile = 0x21ff9e0, 
          flags = 10}
        ret = 32537
(gdb) 



 Comments   
Comment by A. Jesse Jiryu Davis [ 02/Jul/15 ]

Haven't seen this reproduced in some time.

Comment by Mira Carey [ 24/Mar/15 ]

It looks like a bunch of the topology scanner tests have races in them. Not that there are races in the actual code, but in the tests which play with internals. I'll do a more thorough audit and see if I can squash these better.

Generated at Wed Feb 07 21:09:59 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.