[SERVER-63729] SPIKE: generic task distributor Created: 16/Feb/22  Updated: 21/Mar/22  Resolved: 21/Mar/22

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Task Priority: Major - P3
Reporter: Iryna Zhuravlova Assignee: Ryan Egesdahl (Inactive)
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Sprint: Dev Platform 2022-03-21
Participants:

 Description   

We would like to investigate a simple and generic task distributor that can accept and run tasks generated programmatically in real-time and return outputs to the caller. The general idea is that it should function somewhat similarly to Icecream, which is able to schedule and run compiler tasks in real-time on remote hosts and return object files; however, we would like a new system which is not necessarily specific to compiler and/or linker output (such as unit test runs).

We do not want to replace Evergreen here. Evergreen works very well as a first-pass task distributor where the dependencies are knowable at task run time. What we want is a task distributor that can handle dynamic tasks, such as those generated by a build system. (In other words, Evergreen runs the build system as a first-pass task, and the build system can redistribute compile/link/test tasks after determining build task dependencies.) Another difference with Evergreen is that we want real-time and guaranteed-issue for the new system, similar to how Icecream works.


Generated at Thu Feb 08 05:58:31 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.