-
Type:
Improvement
-
Resolution: Unresolved
-
Priority:
Unknown
-
None
-
Affects Version/s: None
-
Component/s: None
-
None
Context
AppendBatchSequence and AppendBatchArray (x/mongo/driver) build the write payload for write operations (BulkWrite, InsertMany, etc.) by appending each encoded document to dst in a loop with no pre-sizing. append grows dst geometrically, reallocating and copying the whole buffer at each step — significant, avoidable allocation churn on large batches. It was a top allocation site in a profile of a write-heavy service. Additionally, AppendBatchArray builds each element key with strconv.Itoa, allocating a string per document (indices >= 100).
Document sizes are known up front, so the buffer capacity can be reserved in one shot, and the numeric array key can be written in place without a string. (Minimum Go is 1.19, so slices.Grow is unavailable.)
Benchmark — 1000 x 256-byte documents into a fresh buffer:
| Function | allocs/op | B/op | ns/op |
|---|---|---|---|
| AppendBatchSequence | 24 -> 3 | 1.19 MB -> 262 KB | ~130us -> ~29us |
| AppendBatchArray | 925 -> 3 | 1.19 MB -> 262 KB | ~166us -> ~38us |
Definition of done
- Both methods reserve capacity once before the append loop; no incremental reallocation.
- AppendBatchArray writes the element index key without allocating a string.
- Behavior unchanged: same n, byte-identical output, same maxCount/totalSize handling, and the same "nothing fits" early return (dst[:l]).
- No-op when dst already has capacity (reused/pooled buffer) — no regression.
- Compiles under Go 1.19; gofmt/lint/vet clean; existing tests pass.
- Tests cover multi-doc and non-zero-offset paths, plus an equivalence test proving the array output matches AppendDocumentElement/strconv.Itoa (incl. multi-digit keys); benchmarks added.
Pitfalls
- Compute the reserved size under the same maxCount/totalSize limits as the loop; summing all remaining docs over-allocates when batch splitting occurs.
- For the array path, include per-element framing (type byte + key + null) and the closing byte in the size, or the final append still reallocates.
- The in-place array key must be byte-identical to bsoncore.AppendDocumentElement.
- Preserve the n == 0 return of dst[:l] and keep the wire output byte-identical.