It is possible they got hacked but when you get a system of sufficient complexity it is unfortunately true that your own can do more damage.. A lot of time its old code that worked fine when they had only a few users but can not keep up with the demands of a billion more users. Then you mix it with new code and everyone thinks its the new code and not the old because 'it always worked before'. One that took the top programmers 3 weeks to find was something that failed to release its memory when it was done (but said it did) and had worked fine for 10 years before... until it was running with a new program that actually worked correctly but used up that last bit of memory the system had and then CRASH. So they test the new code for weeks and can't find it, because it was sneakily hiding in the old code that worked with less users for ten years and was by now considered sancrosanct. And of course when the top programmer tests the fixes to the code on a test system everything works fine because you dont have a billion users sucking up all the memory. That one had to get handed up three times and was only found by the guy with the phd and twenty years experience. And it was just a 'simple mistake' with 'a tiny amount of memory' but multiplied a billion times...
The actual replacement of the data shouldn't take THAT long as they should be using faster methods then in my day. The term 'rollback' is confusing. What the admins do is 'roll forward'. You start with a blank system and then start restoring your backups. Most data loss is confined to the period of time from the last backup, at like 2am or so to when the system dies at 6pm. With a database which im sure they are using, you restore the physical data and then it also records every single command to within 3 seconds or so of the crash and it takes some time to 'replay' the commands to the database, but not that long.



Reply With Quote