I (idly) wonder what would break if the SEDE refreshes were moved to Mondays. Just wondering if that would be any benefit to the staffers (taryn, Nick, et al) who would be more likely to be awake & online & working on a (US) Monday than a (US) Sunday.
@JeffSchaller well, not much. There is no active monitoring on SEDE. So when something fails on Sunday, they will only see it due to the e-mail that gets send for a failed job. On Monday the same thing will happen and it will not have high priority. I guess it will then get fixed on Tuesday ;)
Only thing is that SEDE refresh runs as part of the weekly full back of the production databases. That has to run on Sunday due to load.
@JeffSchaller I did ask / hinted here to have the on-call SRE be alerted but that was kindly refused by @Taryn in her answer. An e-mail is what we got out of it ;)
Yes, it's that low priority (and seeing Taryn's tweet about whether to exit vacation to fix one of the failures) that prompted the idea to move the work to a normal business day. I realized (after sleeping on it) that the refreshes are probably timed on Sundays to coincide with low loads, so I think that's the end of my great idea :)
Yeah, it always get me (despite that I know how huge their database is) that the 03:00 UTC backup it not ready at 03:01 UTC ... if you try to improve say, 10% of its process time, you win 42 minutes, given the whole refresh process takes 7 hours.
It has been a while that I looked into infrastructure stuff. I think they have the disks in the server so a backup probably goes over the GB network card to another box. I do remember that a NAS can do a snapshot and then backup the snapshot but I also recall that required that the clearcase server was first stopped. Stopping the server is an outage SE probably doesn't want to take.
