Description

On 32-bit builds of the C extension, rb_bson_expand_buffer computes the new
buffer capacity as required_size * 2 in size_t arithmetic. When
required_size reaches 2^31 the multiplication wraps modulo 2^32, so
ALLOC_N allocates a block far smaller than requested and the caller then
writes gigabytes past its end. The result is an immediate segfault.

ALLOC_N(char, new_size) does not catch this: sizeof(char) is 1, so
ruby_xmalloc2's multiplication-overflow check never fires. The wrapped value
is recorded as the buffer's capacity at ext/bson/bytebuf.c:65, so every
subsequent ENSURE_BSON_WRITE trusts it.

This does not affect 64-bit builds. There size_t is 8 bytes and
required_size cannot exceed ~2^32, so the doubling has no way to wrap. It
also does not affect JRuby or the pure-Ruby fallback, where integers are
arbitrary precision.

Raised as SECBUG-4387 (Aegis finding 41451a3919ea) and assessed there as not a
security vulnerability. Filing here as ordinary hardening.

Reproduction

Build the extension under a 32-bit Ruby. Verified with i386/ruby:3.1
(RUBY_PLATFORM = i686-linux, sizeof(size_t) == 4) against an unmodified
checkout:

require 'bson'
buf = BSON::ByteBuffer.new
buf.put_bytes('A' * 7)            # 7 bytes of existing content
buf.put_bytes('B' * (2**31 - 6))  # single large write

content bytes   = 7
string bytesize = 2147483642
required_size   = 2147483649
new_size (32b)  = 2                <-- ALLOC_N size after the wrap

/poc/poc.rb:27: [BUG] Segmentation fault at 0x0070b000
ruby 3.1.7p261 (2025-03-26 revision 0a3704f218) [i686-linux]
c:0003 p:---- s:0015 e:000014 CFUNC  :put_bytes

put_string is affected identically via ext/bson/write.c:204
(ENSURE_BSON_WRITE(b, length + 5)). The pvt_check_string_length cap of
INT32_MAX - 5 does not prevent this, since the wrap begins at 2^31.

Root cause

ext/bson/bytebuf.c:51-59:

const size_t required_size = buffer_ptr->write_position - buffer_ptr->read_position + length;  /* :51 */
...
const size_t new_size = required_size * 2;   /* :58 -- wraps when required_size >= 2^31 */
new_b_ptr = ALLOC_N(char, new_size);         /* :59 -- undersized block */
memcpy(new_b_ptr, READ_PTR(buffer_ptr), READ_SIZE(buffer_ptr));  /* :60 -- overflow */
buffer_ptr->size = new_size;                 /* :65 -- wrapped value recorded as capacity */

Write c for the buffer's content length
(write_position - read_position) and S for the incoming length. Then
required_size = c + S. Both c and S are bounded by 2^31-1, so the
addition at :51 is always exact; only the doubling at :58 can leave the
representable range. The trigger condition is therefore exactly:

c + S >= 2^31        (equivalently, 2 * (c + S) >= 2^32)

Whenever it holds the allocation is undersized, since
2(c + S) - 2^32 < c + S for all c + S < 2^32. There is no sub-case that
lands safely.

Boundary behaviour

Just below the threshold the code is already safe by accident. With c = 7,
S = 2^31 - 8, new_size is 4294967294 (~4 GiB), ALLOC_N fails, and
Ruby raises a catchable NoMemoryError with no corruption. Confirmed on
32-bit. The fix simply extends that clean failure across the boundary.

Proposed fix

Reject the case where the doubling would leave the representable range, before
performing it:

} else {
  char *new_b_ptr;
  size_t new_size;
  if (required_size > SIZE_MAX / 2) {
    rb_raise(rb_eArgError, "Buffer size exceeds maximum");
  }
  new_size = required_size * 2;
  ...

On 64-bit the new branch is unreachable, so there is no behaviour change and no
measurable cost on the platforms we ship and test. SIZE_MAX currently
appears nowhere in the extension; bytebuf.c and bson-native.h contain no
overflow guards at all.

ENSURE_BSON_WRITE (ext/bson/bson-native.h:65) computes
write_position + length, which cannot wrap for the same reason :51
cannot: both terms are bounded by 2^31-1. Rewriting it as
length > buffer_ptr->size - buffer_ptr->write_position is still worth doing
for clarity, but it is not required to fix this bug.

Scope and impact

Low.

• Reaching the threshold requires a single String of roughly 2 GiB handed to
  one put_bytes or put_string call, about 128x the 16 MB BSON document
  limit. Any such operation would fail regardless.
• Accumulating there gradually is not possible. Each expansion allocates
  2 * required_size and copies the old buffer, so a 32-bit process exhausts
  its address space first. Measured: appending 16 MiB chunks to a single
  ByteBuffer in a 3.5 GB 32-bit container stops with NoMemoryError at
  1040187392 bytes (0.97 GiB) of content, 48% of what the wrap needs.
• The outcome is a crash only, not a usable write primitive. The wrapped
  allocation size k = 2(c + S) - 2^32 is finely tunable, but the length of
  the overflowing memcpy is not: S >= 2^31 - c and c is capped near
  1 GiB, so the copy is always at least ~1 GiB. Verified by tuning k to
  exactly 200 bytes (c = 1000100, S = 2^31 - 1000000): the :60
  memcpy overflows by 999900 bytes and the write.c:134 memcpy then
  overflows by 2147483548 bytes, faulting immediately. write.c:134 executes
  on the line after rb_bson_expand_buffer returns, with no intervening Ruby
  code, GC safepoint, or allocator callback.
• No call site does a large ENSURE_BSON_WRITE followed by only a small
  write, which is the shape that would leave the tiny wrapped capacity in place
  with control back in Ruby. write.c:133, :324 and :657 write
  exactly what they reserve; :204 reserves length + 5 and writes 4 bytes
  followed two lines later by length bytes.
• The availability difference is the real defect: an uncatchable SIGSEGV where
  the caller should see a catchable NoMemoryError.
• Not reachable through mongo-ruby-driver by a remote attacker. Data of
  network origin that is re-serialized into a write buffer (SASL auth payloads,
  topologyVersion in push-monitor hellos, change stream resume tokens,
  cursor ids, compressed_message) all arrives under the inbound
  max_message_size guard at protocol/message.rb:251, ~48 MB, three
  orders of magnitude short of 2^31. Only the application itself, by passing an
  absurd argument, can reach the threshold.
• No 32-bit platform is currently built or tested in Evergreen, and the gem
  ships as source, so this is reachable only for users compiling on a 32-bit
  target.

Testing notes

The trigger cannot be exercised in CI: it needs a 32-bit build and roughly
2 GiB of address space. The required_size > SIZE_MAX / 2 guard is
unreachable on 64-bit, so a spec cannot cover it on any platform we test.

Recommend verifying manually under i386/ruby and recording the platform
constraint in a code comment, rather than adding a spec that would silently
pass without testing anything. If a regression spec is wanted anyway, gate it
behind STRESS=1 and skip unless [0].pack('J').bytesize == 4, alongside
the existing spec/bson/string_length_overflow_spec.rb from RUBY-3894.

Relationship to RUBY-3894

Adjacent but distinct. RUBY-3894 fixed signed narrowing in the string write
paths in ext/bson/write.c (RSTRING_LEN stored in int32_t). This is
unsigned multiplication wrap in the buffer-growth arithmetic in
ext/bson/bytebuf.c, which that patch did not touch.