ExportXMLWordPrintableJSON

    • Type: Bug
    • Resolution: Unresolved
    • Priority: Unknown
    • None
    • Affects Version/s: None
    • Component/s: Source
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      The source task can skip or replay change-stream events in three ways. Reproduced against mongo-kafka 3.1.2-SNAPSHOT (master at b4f4309) without a live Kafka cluster. Host order is not one of these cases: the driver sorts seed hosts, so reordering the same hosts keeps the same offset partition.

      1. A changed seed-host set hides the stored resume token

      The default offset partition is mongodb://host1,host2/db.coll, built by MongoSourceTask.createDefaultPartitionName from ConnectionString.getHosts() plus the database and collection, unless offset.partition.name is set. MongoSourceTask.getOffset looks that key up under ns. Replacing or adding a host changes the key, the stored resume token is not read, and the stream starts at startup.mode (default latest).

      Reproduction:

      • Store an offset for database=db, collection=coll, connection.uri=mongodb://a.example:27017,b.example:27017. The partition is mongodb://a.example:27017,b.example:27017/db.coll and the offset value is:
        {"_id": "{\"_data\": \"resume-token\"}"}
        
      • Start the same connector with connection.uri=mongodb://a.example:27017,c.example:27017 (one host replaced).
      • createDefaultPartitionName is now mongodb://a.example:27017,c.example:27017/db.coll.
      • OffsetStorageReader.offset is called with that new key. The resume token is not returned.

      KAFKA-505 measured the same loss on a live connector: about 6k events when a hostname became an IP, and 6224 events when a host was added. Reordering hosts lost 0. Setting offset.partition.name lost 0. An appName option lost 0. KAFKA-505 as shipped (b4f4309 / PR 236) only logs a warning. offset.partition.name is the workaround.

      2. errors.tolerance=all opens the next change stream without the resume token

      StartedMongoSourceTask.tryRecreateCursor runs when errors.tolerance=all and the server rejects the change stream (error codes 260, 280, 286, 10334, or an error message that mentions a resume token or resume point together with "not found", "does not exist", "invalid resume token", or "no longer be in the oplog"). It calls tryCreateCursor with no resume token, so startAfter is not set. Events after the last committed token and before now are skipped.

      Reproduction:

      • Config errors.tolerance=all and a stored offset:
        {"_id": "{\"_data\": \"8264\"}"}
        
      • The first cursor is created with startAfter on that token.
      • The cursor then fails with error code 286 and message "resume token was not found".
      • Recovery opens another cursor and does not call startAfter.

      Error 260 (invalidated resume token) takes a separate branch, invalidatedResumeToken, which also restarts with no token, and it does that even when errors.tolerance is not all.

      3. startup.mode=copy_existing scans before the change-stream token is captured

      MongoSourceTask.start constructs MongoCopyDataManager before StartedMongoSourceTask. The copy-manager constructor submits the snapshot aggregations immediately. setCachedResultAndResumeToken (the watch() that records the resume token) runs only after that, inside the StartedMongoSourceTask constructor. A write that lands after the snapshot aggregation has passed a document, and before the resume token is captured, is in neither the snapshot nor the stream.

      The class comment on MongoSourceTask already describes the safe order: capture the resume token, then copy. The implementation does the opposite. Duplicates from copying after the token are expected and are documented as idempotent. This gap is not.

      Reproduction:

      • startup.mode=copy_existing, database=db, collection=coll.
      • The collection aggregate used by the copy blocks once entered.
      • Constructing MongoCopyDataManager enters that aggregate.
      • At that moment watch() has not been called.
      • Constructing StartedMongoSourceTask with that manager is what first calls watch().

      Expected

      • The offset partition stays stable across seed-host changes. When connection.uri sets replicaSet, the partition should be the replica set name plus the namespace. An offset still stored under the host key should be read when that key can still be computed, and later offsets should be stored under the stable key. A host-list change that happens before any offset is written under the stable key cannot be recovered, because Connect looks up an exact partition map.
      • A resume token the server rejects must fail the task. The task must not continue the stream from now.
      • startup.mode=copy_existing must capture the change-stream resume token before copy threads run.

      Related

      • KAFKA-505 — default offset partition includes the host list. Warning only.
      • KAFKA-385 — exactly-once source support is not declared.

            Assignee:
            Unassigned
            Reporter:
            Luke Prochazka (EXT)
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated: