We currently rate limit event processing to 1000 events per job, and jobs run once per minute. This means that under heavy task completion volume, Evergreen can lag in sending events. Geoffrey Mishkin reported receiving a notification 20 minutes after the event was processed.
It takes roughly a minute to process the 1000 events so raising the limit may not be sufficient, and there may also be performance implications for the database. We should find out which pieces are slow and see if we can address the bottlenecks or consider alternative architectures.