UPDATE: Everything should be up and running. Please report any issues you might have. – tatu

 

Woah, that did not go as planned…

So here’s what happened: during the migration, a mostly automated process, we noticed that the nodes had become unstable. They may have been out-of-synch before, but when the migration started, one of the nodes started to override itself and in turn remove large parts of the files it contained. 

The process was stopped immediately, but in doing so we lost access to the data on the node. The node contained files from some of the hosted websites, including trescom, and more importantly our entire webserver configuration. Without it, all websites went down immediately. The databases, which contains the content to all websites were migrated without any issues, so in a worst-case scenario we would have had all the texts, but only parts of the files (images etc). There was a second backup strategy in place, on top of the replication of the data across the nodes, but that one failed us too, as it stopped synching data from the newer nodes. Regardless, we would have been able to recover data from there, but it would have taken a lot of time.

As of writing, we managed to recover the broken node and regain access to all data. We are now fixing remaining issues on the websites – Trescomforum for example is lacking the proper PHP version. So everything should be back to normal soon.