(Tues. Nov. 28, 2023 1:00pm EST): We were made aware of the situation where you may be getting an error on the device while syncing records ("Cannot create media file on cloudfiles") or an image on the server is broken. We found that our host, Rackspace, is having issues with their storage system.
If possible, we would consider using one of our other media storage options, even if just for a short period of time until their issue is resolved.
------------------------
Updates from Rackspace:
On 29 November 2023, engineers continued to monitor the frequency of errors and introduced new drives into the environment. Further additional storage capacity was to be added in a controlled manner balancing object replication and deletion process to further restore the environment's health. Engineers continued to recommend that customers retry upload and delete actions as needed as this was the only workaround available.
On 30 November 2023, to increase storage capacity within the region and mitigate the impact being experienced, engineers continued to replace the utilised drives, pulling in inventory from Rackspace data centers globally. As part of this process, engineers began to utilize drives with higher capacity. Given the type of storage configuration in this environment, the drive replacement process necessitates that drives are replaced as a group, rather than individually. As of 21:41 UTC, engineers confirmed they successfully completed a group replacement and were monitoring the environment.
Additionally, engineers identified some drives, once installed, were not being recognized by the configuration, and must either be re-introduced or replaced once again, adding to the time of recovery. Once properly installed (a process that must be carefully managed by engineers so as not to introduce additional impact) and incorporated into the logical volume, this additional capacity would enable installation, deletion, replication and synchronization processes operate in a healthy environment.
On 01 December 2023, following the change on 30 November (introducing a new group of drives into the shared pool), engineers have observed a significant drop in the volume of returned errors.
Comments
5 comments
Rackspace Status Update:
As of 05 December 2023, engineers continue to monitor the environment, as well as continuing to steadily increase additional server capacity for the impacted region in controlled stages. While affected customers may continue to experience impact, engineers are observing incremental improvement in the environment (and a reduction in the rate of errors experienced by customers). They continue work to fully remediate the issue in the best possible time.
Rackspace Status Update:
As of 6 December at 14:00 UTC, Rackspace engineers reported error rates have greatly decreased since yesterday, although utilization rates remain high. The environment appears stable at this time. Engineers continue to make configuration changes as needed. Engineers will continue to add storage capacity and closely monitor the environment until the utilization rate drops below 80%. Additional storage expertise is being engaged to allow more focus on this issue.
Rackspace Status Update:
As of 07 December 2023, engineers reported that the error rates have consistently remained low since yesterday and utilization remains at a high level. Currently, the team is actively working with inventory across various scenarios to enhance capacity.
Rackspace Status Update:
As of 08 December 2023, engineers updated that they have applied the latest configuration changes and increased storage capacity. Our monitoring indicates that error rates are consistently low. Engineers intend to continue to add additional capacity for greater stability.
As of 11 December 2023, engineers reported the environment continues to experience low error rates and no significant spikes in usage.
As of today, December 12, 2023, Rackspace engineers conveyed that the environment is consistently encountering low error rates. Notably, Zerion’s internal utilization of Clouldfiles experienced a major increase in performance on Friday, December 8, and has remained error-free for the subsequent three days.
Feel free to revert to using Cloudfiles, however, should you encounter any issues with uploading media, consider switching back to the alternatives.
If you should have any follow-up questions, please reach out to support@zerionsoftware.com.
Please sign in to leave a comment.