Loading...

Type: Bug
Resolution: Unresolved
Priority: Major - P3
Fix Version/s: None
Affects Version/s: None
Component/s: None
Labels:
- auto-reverted
- prodsec-team-server

Assigned Teams:

Server Security
Operating System:
ALL
CAR Domain/s:
None

Aha! Reference:
None
Tracking Level:
None
Risk Status:
None
Exec Notes:
None
Goal Name(s):
None
Goal Link:
None

FuzzTests are intended to run for a long period of time, and potentially surface multiple issues. As a result, the harness will observe crashes, record them to a special crash directory, and keep fuzzing. It seems to have a trick to keep itself from repeatedly hitting the same condition. After the target fuzz time elapses, the harness does not return a bad exit code to the shell. This causes resmoke to believe that the test "passed", and evergreen will mark the test as successful.

An an example, when we ran this test in evergreen, we produced logs that looked like:

[cpp_libfuzzer_test:bsoncolumn_decompress_fuzzer] I0000 00:00:1778121605.056228   23996 centipede.cc:342] [S0.17000] rerun-old: ft: 162124 cov: 22931 cmp: 139193 corp: 13839/13839 max/avg: 4125/125 d1/f55 exec/s: 0 mb: 2440
[cpp_libfuzzer_test:bsoncolumn_decompress_fuzzer] I0000 00:00:1778121635.070352   23996 centipede.cc:342] [S0.17409] rerun-old: ft: 162502 cov: 22969 cmp: 139533 corp: 14033/14033 max/avg: 4125/125 d1/f55 exec/s: 0 mb: 2414
[cpp_libfuzzer_test:bsoncolumn_decompress_fuzzer] I0000 00:00:1778121635.934190   23996 centipede.cc:750] Number of input seeds available: 32, number included in corpus: 14034
[cpp_libfuzzer_test:bsoncolumn_decompress_fuzzer] I0000 00:00:1778121635.935462   23996 centipede.cc:342] [S0.17441] init-done: ft: 162503 cov: 22969 cmp: 139534 corp: 14034/14034 max/avg: 4125/125 d1/f55 exec/s: 0 mb: 2616
[cpp_libfuzzer_test:bsoncolumn_decompress_fuzzer] I0000 00:00:1778121635.935821   23996 centipede.cc:578] Generate corpus stats [Before fuzzing]; stats_path: corpora/bsoncolumn_decompress_fuzzer/workdir.000/LLVMFuzzer.TestOneInput/corpus-stats-bsoncolumn_decompress_fuzzer.000000.initial.json
[cpp_libfuzzer_test:bsoncolumn_decompress_fuzzer] I0000 00:00:1778121639.150132   23996 centipede.cc:657] Generate rusage report [Before fuzzing]; env_.my_shard_index: 0 path: corpora/bsoncolumn_decompress_fuzzer/workdir.000/LLVMFuzzer.TestOneInput/rusage-report-bsoncolumn_decompress_fuzzer.000000.initial.txt
[cpp_libfuzzer_test:bsoncolumn_decompress_fuzzer] I0000 00:00:1778121639.300367   23996 centipede_default_callbacks.cc:88] Custom mutator detected; will use it.
[cpp_libfuzzer_test:bsoncolumn_decompress_fuzzer] I0000 00:00:1778121652.665395   23996 centipede.cc:342] [S0.1000] new-feature: ft: 162531 cov: 22969 cmp: 139562 corp: 14054/14054 max/avg: 4125/125 d1/f55 exec/s: 74 mb: 2551
[cpp_libfuzzer_test:bsoncolumn_decompress_fuzzer] I0000 00:00:1778121653.435102   23996 centipede.cc:878] ReportCrash[1]: Batch execution failed:
[cpp_libfuzzer_test:bsoncolumn_decompress_fuzzer] Binary               : 'bazel-bin/install/bin/bsoncolumn_decompress_fuzzer' '--fuzz_for=1h' '--corpus_database=corpora' '--llvm_fuzzer_wrapper_corpus_dir=' --internal_override_fuzz_test=LLVMFuzzer.TestOneInput --internal_override_total_time_limit=1h
[cpp_libfuzzer_test:bsoncolumn_decompress_fuzzer] Exit code            : 6
[cpp_libfuzzer_test:bsoncolumn_decompress_fuzzer] Failure              :
[cpp_libfuzzer_test:bsoncolumn_decompress_fuzzer] Signature            :
[cpp_libfuzzer_test:bsoncolumn_decompress_fuzzer] Number of inputs     : 1000
[cpp_libfuzzer_test:bsoncolumn_decompress_fuzzer] Number of inputs read: 33
[cpp_libfuzzer_test:bsoncolumn_decompress_fuzzer] Suspect input index  : 33
[cpp_libfuzzer_test:bsoncolumn_decompress_fuzzer] Crash log            :
[cpp_libfuzzer_test:bsoncolumn_decompress_fuzzer]
[cpp_libfuzzer_test:bsoncolumn_decompress_fuzzer]
[cpp_libfuzzer_test:bsoncolumn_decompress_fuzzer] CRASH LOG: Starting watchdog thread: timeout_per_input: 0 sec; timeout_per_batch: 3600 sec; rss_limit_mb: 0 MB; stack_limit_kb: 128 KB
[cpp_libfuzzer_test:bsoncolumn_decompress_fuzzer] CRASH LOG: Not using RLIMIT_AS; VmSize is 36865Gb, suspecting ASAN/MSAN/TSAN
[cpp_libfuzzer_test:bsoncolumn_decompress_fuzzer] CRASH LOG: Note: Google Test filter = LLVMFuzzer.TestOneInput
[cpp_libfuzzer_test:bsoncolumn_decompress_fuzzer] CRASH LOG: [==========] Running 1 test from 1 test suite.
[cpp_libfuzzer_test:bsoncolumn_decompress_fuzzer] CRASH LOG: [----------] Global test environment set-up.

However, the test passes without error:

[cpp_libfuzzer_test:bson_column_validate_fuzzer] I0000 00:00:1778124222.630899   23984 centipede.cc:342] [S0.2662000] new-feature: ft: 38782 cov: 3440 cmp: 35342 corp: 6324/6324 max/avg: 4125/595 d3/f3 exec/s: 744 mb: 2981
[cpp_libfuzzer_test:bson_column_validate_fuzzer] I0000 00:00:1778124230.771608   23984 centipede.cc:342] [S0.2666000] end-fuzz: ft: 38782 cov: 3440 cmp: 35342 corp: 6324/6324 max/avg: 4125/595 d3/f3 exec/s: 743 mb: 2996
[cpp_libfuzzer_test:bson_column_validate_fuzzer] I0000 00:00:1778124230.771709   23984 centipede.cc:578] Generate corpus stats [After fuzzing]; stats_path: corpora/bson_column_validate_fuzzer/workdir.000/LLVMFuzzer.TestOneInput/corpus-stats-bson_column_validate_fuzzer.000000.final.json
[cpp_libfuzzer_test:bson_column_validate_fuzzer] I0000 00:00:1778124231.023130   23984 centipede.cc:657] Generate rusage report [After fuzzing]; env_.my_shard_index: 0 path: corpora/bson_column_validate_fuzzer/workdir.000/LLVMFuzzer.TestOneInput/rusage-report-bson_column_validate_fuzzer.000000.final.txt
[cpp_libfuzzer_test:bson_column_validate_fuzzer] I0000 00:00:1778124231.086841   23984 resource_pool.cc:91] Creating pool with quota=[RSS: 25.00G | VSize: 0B | VPeak: 0B | Data: 0B | ShMem: 0B]
[cpp_libfuzzer_test:bson_column_validate_fuzzer] I0000 00:00:1778124231.087108 3832710 distill.cc:340] DISTILL[S.0]: Distilling to output shard 0; input shard indices:
[cpp_libfuzzer_test:bson_column_validate_fuzzer] 0
[cpp_libfuzzer_test:bson_column_validate_fuzzer] I0000 00:00:1778124232.146414 3832767 distill.cc:374] DISTILL[S.0]: batches: 1/1 inputs: 7860 written: 6648
[cpp_libfuzzer_test:bson_column_validate_fuzzer] I0000 00:00:1778124232.147468 3832710 distill.cc:382] DISTILL[S.0]: Done distilling to output shard 0
[cpp_libfuzzer_test:bson_column_validate_fuzzer] I0000 00:00:1778124232.166042 3832709 distill.cc:426] DISTILL[ALL]: ft: 38778 cov: 3440 cmp: 35338 inputs: 7860 unique: 7856 distilled: 6648
[cpp_libfuzzer_test:bson_column_validate_fuzzer] === Summary of detected crashes ===
[cpp_libfuzzer_test:bson_column_validate_fuzzer]
[cpp_libfuzzer_test:bson_column_validate_fuzzer] Binary ID    : bson_column_validate_fuzzer
[cpp_libfuzzer_test:bson_column_validate_fuzzer] Fuzz test    : LLVMFuzzer.TestOneInput
[cpp_libfuzzer_test:bson_column_validate_fuzzer] Total crashes: 0
[cpp_libfuzzer_test:bson_column_validate_fuzzer]
[cpp_libfuzzer_test:bson_column_validate_fuzzer] === End of summary of detected crashes ===
[cpp_libfuzzer_test:bson_column_validate_fuzzer]
[cpp_libfuzzer_test:bson_column_validate_fuzzer] [       OK ] LLVMFuzzer.TestOneInput (3601710 ms)
[cpp_libfuzzer_test:bson_column_validate_fuzzer] [----------] 1 test from LLVMFuzzer (3601710 ms total)
[cpp_libfuzzer_test:bson_column_validate_fuzzer]
[cpp_libfuzzer_test:bson_column_validate_fuzzer] [----------] Global test environment tear-down
[cpp_libfuzzer_test:bson_column_validate_fuzzer] [==========] 1 test from 1 test suite ran. (3601710 ms total)
[cpp_libfuzzer_test:bson_column_validate_fuzzer] [  PASSED  ] 1 test.
[cpp_libfuzzer_test:bson_column_validate_fuzzer] C++ libfuzzer test bazel-bin/install/bin/bson_column_validate_fuzzer finished. Duration of process 3601.861236s
[cpp_libfuzzer_test:bson_column_validate_fuzzer] Test succeeded, skipping symbolization

Any fuzztest that produces a crash should be flagged as failing in Evergreen.

Details

Description

Attachments

Activity

People

Dates