Summary

Currently, Astrolabe uses the unified spec test runner implemented in each driver as the integration point to run operations, monitor errors, and export results. Specifically, Astrolabe uses the loop test operation to force drivers to run operations in a loop until Astrolabe sends an interrupt signal. That creates a number of problems:

• The loop test operation is complex and time-consuming to implement and maintain. It is only used by Astrolabe; no other unified spec tests uses it.
  • Bugs in the implementation of the loop test operation must be detected and fixed separately in each of the 9 drivers that currently integrate with Astrolabe.
  • The different drivers implement the loop test operation in subtly different ways, requiring extra work to refine the spec and for drivers teams to implement fixes.
• The unified spec test runner integrations are brittle. Astrolabe is the only external dependency of the unified spec test runner for many drivers and is easy to forget about and unintentionally break. For example, Go Driver v2 changes the visibility of the unified spec test runner packages which breaks the Astrolabe integration. Other drivers have unintentionally broken the Astrolabe integrations as well, requiring work to fix.
  • Tickets for fixing drivers integrations: RUST-1474, PHPLIB-951, JAVA-4728, JAVA-4114, GODRIVER-3069, CSHARP-4323, CSHARP-5402, JAVA-5716, PYTHON-4863
  • The Java, Python, Ruby, and .NET integration validations currently fail due to small inconsistencies in the behavior of the unified spec test runners.

A better approach is to build an HTTP service in each driver language that uses the CRUD API, allowing Astrolabe to effectively call the CRUD API directly rather than delegate it to the unified spec test runners. That has several benefits:

• The CRUD API in drivers is much more stable than the unified spec test runner API, so there's less chance of the integration breaking.
• We only have to implement the Astrolabe test runner once, so detailed specification of the behavior of test operations is much less necessary.
  • We can remove the loop test operation from the Unified Test Format spec and move it to Astrolabe-specific documentation.

Example HTTP API based on per-operation HTTP routes:

PATH: /find
BODY:
{
  "database": "dat",
  "collection": "coll",
  "filter": {
    "x": {"$numberInt": 1}
  },
  "options": {
    "batchSize": {"$numberInt": 2}
  }
}

Results:

[
  {
    "_id": 1,
    "x": 1
  },
  {
    "_id": 2,
    "x": 1
  },
  {
    "_id": 3,
    "x": 1
  }
]

Additional benefits of building HTTP services:

• Can be used for benchmarking.
• Can be used to test drivers against server changes.

Motivation

Who is the affected end user?

The Astrolabe maintainers and the drivers devs who have to fix the integrations.

How does this affect the end user?

They have to spend time fixing an external integration with the unified spec test runner, which are otherwise only used inside drivers tests.

How likely is it that this problem or use case will occur?

Integrations seem to have problems about 2 times per year on average.

If the problem does occur, what are the consequences and how severe are they?

Wasted time and lack of visibility. Tickets for fixing integrations stay open from a few weeks to a few years, during which time either PR builds or Astrolabe builds always fail.

Is this issue urgent?

No.

Is this ticket required by a downstream team?

No.

Is this ticket only for tests?

No.

Acceptance Criteria

• Define a drivers integration point for the Astrolabe "workload executor" that is more stable than the unified spec test runner.
  • The current proposal is to implement an HTTP service with each driver with a uniform HTTP API.
• Define a spec in Astrolabe for the new integration point that describes how to build it for each driver.
• Implement the new integration for each driver.
  • It's possible that the Astrolabe maintainers can implement some or all of these integrations using public documentation for how to create an HTTP service using MongoDB drivers.