Details
-
Bug
-
Resolution: Fixed
-
Unknown
-
1.6.1
-
Needed
-
Description
Add copy.existing.allow.disk.use configuration.
Allow the copy existing aggregation to use temporary disk storage if required. Defaults to true but can be disabled if the user doesn't have the permissions
for disk access.
============
Was: Exceeded memory limit for $group, but didn't allow external sort
Hi Team,
The Mongo Src connector is failing with the error :
connect | org.apache.kafka.connect.errors.ConnectException: com.mongodb.MongoCommandException: Command failed with error 16945 (Location16945): 'Exceeded memory limit for $group, but didn't allow external sort. Pass allowDiskUse:true to opt in.' on server inclusio-1-shard-00-02.kwe2e.mongodb.net:27017. The full response is {"operationTime": {"$timestamp": {"t": 1636449949, "i": 857}}, "ok": 0.0, "errmsg": "Exceeded memory limit for $group, but didn't allow external sort. Pass allowDiskUse:true to opt in.", "code": 16945, "codeName": "Location16945", "$clusterTime": {"clusterTime": {"$timestamp": {"t": 1636449949, "i": 857}}, "signature": {"hash": {"$binary": {"base64": "TZhmmTYxktZ6ivXdEWZkImFgqVA=", "subType": "00"}}, "keyId": 6991155560347336706}}} |
connect | Caused by: com.mongodb.MongoCommandException: Command failed with error 16945 (Location16945): 'Exceeded memory limit for $group, but didn't allow external sort. Pass allowDiskUse:true to opt in.' on server inclusio-1-shard-00-02.kwe2e.mongodb.net:27017. The full response is {"operationTime": {"$timestamp": {"t": 1636449949, "i": 857}}, "ok": 0.0, "errmsg": "Exceeded memory limit for $group, but didn't allow external sort. Pass allowDiskUse:true to opt in.", "code": 16945, "codeName": "Location16945", "$clusterTime": {"clusterTime": {"$timestamp": {"t": 1636449949, "i": 857}}, "signature": {"hash": {"$binary": {"base64": "TZhmmTYxktZ6ivXdEWZkImFgqVA=", "subType": "00"}}, "keyId": 6991155560347336706}}} |
The Src connector properties are fairly simple
name = prodMongoSrc
|
connector.class = com.mongodb.kafka.connect.MongoSourceConnector
|
tasks.max = 1
|
errors.retry.timeout = -1
|
errors.retry.delay.max.ms = 10000
|
errors.log.enable = true
|
errors.log.include.messages = true
|
connection.uri = mongodb+srv:/<######>
|
topic.prefix = prod
|
copy.existing = true
|
offset.partition.name = partition-03
|
I have hashed the credentials.
When i relaunch it works but again fails after sometimes.
Looking for support and help on this