[CDRIVER-2119] Enable libbson users to do manual buffer management Created: 05/Apr/17  Updated: 03/May/17  Resolved: 06/Apr/17

Status: Closed
Project: C Driver
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Improvement Priority: Major - P3
Reporter: Karolin Varner Assignee: A. Jesse Jiryu Davis
Resolution: Won't Fix Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   

I request that the following functions be exposed as a public api:

bool bson_reserve(bson_t *bson, uint32_t size);
uint32_t bson_capacity(bson_t *bson);

bson_reserve currently exists as the hidden function _bson_grow.
bson_capacity could be implemented like his:

uint32_t bson_capacity(bson_t *bson) {
   if (bson->flags & BSON_FLAG_INLINE) {
      return ((bson_impl_inline_t *) bson)->len;
   } else {
      return ((bson_impl_alloc_t *) bson)->len;
   }
}

These functions would be useful in cases where the developer is dynamically building a bson document – using the bson_append_*() functions – but already has a high degree of knowledge about the size of the resulting document (or just makes a guess and wants to remove the amount of calls to realloc() drastically).



 Comments   
Comment by Karolin Varner [ 06/Apr/17 ]

Jesse: I was aware of that and I would see not use in switching back to heap allocation.

bson_sized_new (size_t size)
{
   bson_impl_alloc_t *impl_a;
   bson_t *b;
 
   BSON_ASSERT (size <= INT32_MAX);
 
   b = bson_malloc (sizeof *b);
   impl_a = (bson_impl_alloc_t *) b;
 
   if (size <= BSON_INLINE_DATA_SIZE) {
      bson_init (b);
      b->flags &= ~BSON_FLAG_STATIC;
   } else {
      impl_a->flags = BSON_FLAG_NONE;
      impl_a->len = 5;
      impl_a->parent = NULL;
      impl_a->depth = 0;
      impl_a->buf = &impl_a->alloc;
      impl_a->buflen = &impl_a->alloclen;
      impl_a->offset = 0;
      impl_a->alloclen = BSON_MAX (5, size);
      impl_a->alloc = bson_malloc (impl_a->alloclen);
      impl_a->alloc[0] = 5;
      impl_a->alloc[1] = 0;
      impl_a->alloc[2] = 0;
      impl_a->alloc[3] = 0;
      impl_a->alloc[4] = 0;
      impl_a->realloc = bson_realloc_ctx;
      impl_a->realloc_func_ctx = NULL;
   }

If size > BSON_INLINE_DATA_SIZE, this code does two calls to malloc. Is this incorrect?
IMO only a single call would be required if _reserve was exposed, it could explicitly trigger a switch from the inline buffer to the heap allocated buffer.

b = bson_malloc (sizeof *b);
impl_a->alloc = bson_malloc (impl_a->alloclen);

Comment by A. Jesse Jiryu Davis [ 06/Apr/17 ]

A bson_t, if it outgrows its internal 500-byte buffer and does a heap allocation, never switches back to stack allocation.

Comment by Karolin Varner [ 06/Apr/17 ]

OK, but the bson_t does not need to live on heap to use heap allocation, does it? Or do you copy the contents of the bson_t itself into the allocated buffer after switching to stack allocation?

Comment by A. Jesse Jiryu Davis [ 06/Apr/17 ]

Ah, I'd forgotten we have bson_sized_new, that's what you should use, please, Karolin. A stack-allocated bson_t starts with about 500 bytes of room to grow, after that it must spill to heap. We can't easily implement a variable-sized stack-allocated bson_t.

Comment by Hannes Magnusson [ 06/Apr/17 ]

You want to increase the stack allocated bson_t by growing the stack space it reserved after having created it?

Once you overflow our stack allocated buffer we switch to malloced buffers. bson_sized_new () allows you to do just one allocation, rather then switching and reallocating.

If your data won't overflow the stack allocation then you don't need to grow the buffer, so no need for extra function for that.

Comment by Karolin Varner [ 06/Apr/17 ]

But bson_sized_new does an extra heap allocation, I'd prefer to keep the bson_t on the stack.
And I would like to reserve buffer space after initial creation…

Comment by Hannes Magnusson [ 06/Apr/17 ]

Sounds like you are looking for bson_sized_new?

If you overflow the initial estimates size, the buffer will still grow.

Comment by Karolin Varner [ 06/Apr/17 ]

Yes, I know about bson_reserve_buffer, but it increases len, doesn't it?

   if (bson->flags & BSON_FLAG_INLINE) {
      /* bson_grow didn't spill over */
      ((bson_impl_inline_t *) bson)->len = size;
   } else {
      ((bson_impl_alloc_t *) bson)->len = size;
   }

_bson_append_va appends at _bson_data(bson) + bson->len as far as I can see, so the following code would not work to build a bson, avoiding dynamic allocations:

bson_t b;
bson_init(&b);
bson_reserve_buffer(&b, 4096);
bson_append_int32(&b, "foo", -1, 42);
bson_append_int32(&b, "bar", -1, 23);
...

I want to grow bson->buf and increase bson->buf_len but not bson->len.

Comment by A. Jesse Jiryu Davis [ 05/Apr/17 ]

This code works correctly:

/* stack allocated, has the "static" flag set */
bson_t b = BSON_INITIALIZER;
 
/* if some_huge_string overflows the stack-allocated 512-byte internal
   buffer, then "b" spills to the heap and updates an internal flag
   to remember to free its heap-allocated buffer */
bson_append_utf8 (&b, some_huge_string);
 
/* checks if b has had to spill to the heap. if so, free the heap
   buffer, otherwise do nothing. */
bson_destroy (&b);

You could initialize b like this instead, same effect:

bson_t b;
bson_init (&b);

To answer your original request: we have bson_reserve_buffer, does that fit your need?:

http://mongoc.org/libbson/current/bson_reserve_buffer.html

Comment by Karolin Varner [ 05/Apr/17 ]

EDIT: Ignore. bson_free wouldn't be called, because BSON_FLAG_STATIC would be set

Note: I am also a bit puzzled over buffer management here in general; if I manually allocate a bson_t on the stack, and then perform many calls to bson_append_(), how do I free the underlying buffer? I can't really use bson_destroy(), because that also calls bson_free on the bson_t* itself, but that is allocated on the stack?

Generated at Wed Feb 07 21:14:14 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.