UPDATE: Everything should be up and running. Please report any issues you might have. – tatu
Woah, that did not go as planned…
So here’s what happened: during the migration, a mostly automated process, we noticed that the nodes had become unstable. They may have been out-of-synch before, but when the migration started, one of the nodes started to override itself and in turn remove large parts of the files it contained.
The process was stopped immediately, but in doing so we lost access to the data on the node. The node contained files from some of the hosted websites, including trescom, and more importantly our entire webserver configuration. Without it, all websites went down immediately. The databases, which contains the content to all websites were migrated without any issues, so in a worst-case scenario we would have had all the texts, but only parts of the files (images etc). There was a second backup strategy in place, on top of the replication of the data across the nodes, but that one failed us too, as it stopped synching data from the newer nodes. Regardless, we would have been able to recover data from there, but it would have taken a lot of time.
As of writing, we managed to recover the broken node and regain access to all data. We are now fixing remaining issues on the websites – Trescomforum for example is lacking the proper PHP version. So everything should be back to normal soon.
5 comments
As Hammond says – you need three Cray X-MP computers to move the data.
Accordingly TresCom server security was paramount, the tightest on the island.
I never doubted that you would straighten it out, Paul. Good Job. 😉
Fire Nedry, he must be behind this sabotage.
I pray that everything will turn out well!
By submitting a comment you grant TresCom a perpetual license to reproduce your words and name/web site in attribution. Inappropriate and irrelevant comments will be removed at an admin’s discretion. Your email is used for verification purposes only, it will never be shared.