We never noticed because we don't have the .zip linked from the download centre (only the .dmg), none of us seemed to have used it in ages (maybe just unlucky?) and the e2e tests in evergreen run on macOS 11 Big Sur which dates from November 12, 2020 and it doesn't seem to care.

The file in there does run if you un-quarantine it, though:

xattr -d com.apple.quarantine <path/to/the.app>

We sign it the same way as the .dmg, but apparently that's not good enough?

This is for the .zip you get on all PR builds, betas, GAs going back in time. (I only checked a year or so, but since it is caused by a change in macOS, not our files I'm assuming it goes back until the dawn of time.)