Hi all. Over this past weekend we had a 90 minute outage across the Ceramic nodes operated by 3Box Labs due to an expired domain registration. We became aware of the issue because of reports from community members and internal test failure alerts. The outage caused the 3Box Labs nodes on Clay Testnet and Mainnet (in particular the anchor service) to be unreachable.
Applications that created stream updates using CACAO during the period of the incident (Sunday, Sept 25, 3:00PM - 4:30PM ET / 19:00 - 20:30 UTC) may have been affected. However, in our investigation of commits created on the network at this time we found no signs of data loss on Mainnet.
We were able to fix the issue and stabilize the affected services and we have some immediate next steps for domain and resource management on our team to ensure this will not happen again. Thanks all for your patience and please let us know if you experience any issues by posting here, on the Ceramic forum.