The website is working again and most users won't recognize any discrepancies.
Some XML files are not yet generated but we are already on it, so everybody will have his XML file soon.
Why did it all happen?
Scaleway, our server host, scheduled reboots on servers which didn't need a reboot, as "obligatory maintenance".
They said, it will take a few minutes but maximally up to one hour. It took over 7 hours.
After our MySQL server was rebooted, our Minio server was still stuck in a "rebooting" state for another few hours.
When we complained, they told us that our configuration is faulty, at the same time the server was available again miraculously, but all the data was wiped from it.
It was completely blank.
Now we needed another few hours to fix their shit and restore the data from backups.
Having only backups to work with and scheduled crown jobs not executing due to the server reboot, the whole system was messed up, therefore we have to recreate many XML files from scratch now.
We are sorry for the inconvenience.
We will finally keep our promise and move to another host.
Last time we believed their marketing team that it will be enough if we upgrade to higher tier dedicated servers for not having to deal with outages.
Apparently it wasn't true and now the glass is full.
We will add 5 days extra time to all paid accounts for free as a compensation for the troubles.
Some XML files are not yet generated but we are already on it, so everybody will have his XML file soon.
Why did it all happen?
Scaleway, our server host, scheduled reboots on servers which didn't need a reboot, as "obligatory maintenance".
They said, it will take a few minutes but maximally up to one hour. It took over 7 hours.
After our MySQL server was rebooted, our Minio server was still stuck in a "rebooting" state for another few hours.
When we complained, they told us that our configuration is faulty, at the same time the server was available again miraculously, but all the data was wiped from it.
It was completely blank.
Now we needed another few hours to fix their shit and restore the data from backups.
Having only backups to work with and scheduled crown jobs not executing due to the server reboot, the whole system was messed up, therefore we have to recreate many XML files from scratch now.
We are sorry for the inconvenience.
We will finally keep our promise and move to another host.
Last time we believed their marketing team that it will be enough if we upgrade to higher tier dedicated servers for not having to deal with outages.
Apparently it wasn't true and now the glass is full.
We will add 5 days extra time to all paid accounts for free as a compensation for the troubles.