[GODRIVER-1946] Explore github.com/goccy/go-reflect performance Created: 03/Apr/21 Updated: 27/May/21 Resolved: 26/May/21 |
|
| Status: | Closed |
| Project: | Go Driver |
| Component/s: | None |
| Affects Version/s: | None |
| Fix Version/s: | None |
| Type: | Improvement | Priority: | Major - P3 |
| Reporter: | David Golden | Assignee: | Matt Dale |
| Resolution: | Done | Votes: | 0 |
| Labels: | matt | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Attachments: |
|
||||||||||||
| Issue Links: |
|
||||||||||||
| Documentation Changes: | Not Needed | ||||||||||||
| Description |
|
A friend tipped me off to github.com/goccy/go-reflect, which is API compatible with reflect, but does zero allocations. This might help address performance/memory issues in marshaling and unmarshaling. There is some risk departing from the core reflect library, but it might be worth an experiment to see what the benefits are. |
| Comments |
| Comment by Githook User [ 27/May/21 ] | |||||||||||||||||||||||||||||||||||||||||||||
|
Author: {'name': 'Matt Dale', 'email': '9760375+matthewdale@users.noreply.github.com', 'username': 'matthewdale'}Message: | |||||||||||||||||||||||||||||||||||||||||||||
| Comment by Matt Dale [ 26/May/21 ] | |||||||||||||||||||||||||||||||||||||||||||||
|
Closing this because the investigation is done. Conclusion is that using go-reflect won't reduce allocations in the BSON library, but there are other ways we can reduce allocations. | |||||||||||||||||||||||||||||||||||||||||||||
| Comment by Matt Dale [ 26/May/21 ] | |||||||||||||||||||||||||||||||||||||||||||||
|
Doing some memory profiling using the new BSON benchmarks revealed why switching to github.com/goccy/go-reflect didn't have the intended effect. The majority of allocations made while encoding and decoding BSON are in the bsonrw.valueWriter and bsonrw.valueReader types. We should focus our efforts on minimizing allocations and maximizing slice reuse in those types. I've attached the memory profiles for more details. I've created tickets for reducing unnecessary memory allocations in the BSON library based on the findings: | |||||||||||||||||||||||||||||||||||||||||||||
| Comment by Githook User [ 25/May/21 ] | |||||||||||||||||||||||||||||||||||||||||||||
|
Author: {'name': 'Matt Dale', 'email': '9760375+matthewdale@users.noreply.github.com', 'username': 'matthewdale'}Message: | |||||||||||||||||||||||||||||||||||||||||||||
| Comment by Matt Dale [ 18/May/21 ] | |||||||||||||||||||||||||||||||||||||||||||||
|
benchstat comparison after drop-in replacing all import "reflect" with import "github.com/goccy/go-reflect":
Actually slower and more allocations with github.com/goccy/go-reflect perplexingly. | |||||||||||||||||||||||||||||||||||||||||||||
| Comment by Matt Dale [ 18/May/21 ] | |||||||||||||||||||||||||||||||||||||||||||||
|
Here's a baseline for current marshal and unmarshal performance of the BSON, mgo BSON, extended JSON, and Go JSON libraries:
| |||||||||||||||||||||||||||||||||||||||||||||
| Comment by Matt Dale [ 14/May/21 ] | |||||||||||||||||||||||||||||||||||||||||||||
|
Looking into this. Previous investigations into BSON performance used the mgo BSON implementation as the benchmark, so I'll start by running benchmarks for both the mgo BSON library and mongo-go-driver BSON library to get a comparable baseline. Some history of BSON performance investigation and improvement:
|