We want to bring your attention to a critical bug that we recently found. After our initial investigation, we have only identified a couple of teams impacted by this bug, who we have already informed directly. However, we wanted to share this update with you all to make sure that no other developers are affected by it, who we may not have been able to identify via our automated metrics.
The impacted version was any Ceramic version released after Oct 19, starting with version 2.42.0. The bug was fixed starting in version 3.2.0 which was released on Dec 11.
We recently discovered a bug in js-ceramic where anchor commits would be created, discovered, and applied successfully to the local Ceramic node, but then when the node restarts it forgets about the anchor commit entirely. Since anchor commits are required to give the node timestamp information about when a write happened, without anchor commits you can get CACAO timeout errors when loading Streams, since the node is unable to prove that the write happened during the time period when the session key (authorized by the CACAO) was valid.
To our knowledge, we have only a few teams that were directly impacted by this bug. However, if you have run any of the impacted versions of Ceramic (between v2.42.0 and v3.2.0), then you were most likely affected by this bug as well. The good news is that since the data was actually anchored, the streams can be repaired by discovering and re-applying the anchor commit to the affected streams. If you believe you may have been affected by this bug, please reach out to our team by replying to this thread ASAP, and we can use the data recovery tool we have built to restore your data.
We made a release on the 11th of December that contains a fix for the bug: ｜ceramic-releases. We highly recommend you upgrade your node to this new version ASAP to avoid new streams being affected by this issue.
Please let us know if you are experiencing any issues or need help with addressing the impact of this bug.