CACAO has expired

Devtools Ceramic SDKs
1

Hi there,

It seems like some of the data stored in ComposeDB has over the weekend somehow expired. Whenever I try to interact with certain models using their ID I am getting the error:

[2022-09-12T08:17:43.850Z] ERROR: Error: Can not verify signature for commit bagcqcerahvuiwpaepytlf2qvdmhgysekqrzkcemixsm46lgo5qbrw3ofv2aa: CACAO has expired

I’m logged in as the same user that created the model but now if I try to edit it, I get the above error. Now this item is in a weird state as I can’t edit it. I also see some other messages that might be related:

[2022-09-12T08:15:10.817Z] WARNING: Error loading stream kjzl6kcym7w8y6ew41azo9if2ljp3p8y51q2dthywzp7t9wha3diyfcttgxu91a at time undefined as part of a multiQuery request: Error: Can not verify signature for commit bagcqcerahvvtz6hffdmo7u27kpdw5jy6762kmz7ap2m7erfnwytg44fhti7a: CACAO has expired

Furthermore, I suspect the issue might lie in the fact that nothing I do with this node is synced to Ceramic? So I think my stream IDs are just on my node and nothing more is done with them?

Since starting to use ComposeDB and running the Ceramic daemon on a couple of different machines, I’ve never had a stream ID appear in Cerscan (using the clay testnet) – could this be an indication that something is wrong?

2

Cerscan does not currently support viewing ComposeDB streams yet, so that part is expected, it’s something they are working on. To understand the CACAO timeout errors, can you share the streamids for the streams you are seeing this on?

3

Ah okay that makes sense regarding Cerscan then.

Unfortunately I deleted the models and can’t load them back again. I still have the logs which says:

[2022-09-12T08:14:22.046Z] WARNING: Error loading stream kjzl6kcym7w8y6ksgbpq7rf3vg5ea1g0q76udqmv1ocgg3h3eljfvxbpbqx8gj0 at time undefined as part of a multiQuery request: Error: Can not verify signature for commit bagcqceraip7b4shoxyvvhfhnxwtajjwtl53vi4zqwirlribsaxnuttyrl2wa: CACAO has expired
[2022-09-12T08:14:22.950Z] WARNING: Error loading stream kjzl6kcym7w8y7ekf3v3yn2w856kqfxnxawapwv6033nviwsihedigheclivbbf at time undefined as part of a multiQuery request: Error: Can not verify signature for commit bagcqceramuw5h22wf24n5gxyxh6vwoge6xjjjadb6uqaqnurtzp6efvakefq: CACAO has expired
[2022-09-12T08:14:23.579Z] WARNING: Error loading stream kjzl6kcym7w8y6n8h1s2w7k6svke1l14x8z89u32em1kv9bzviqo6bhxbumnw2u at time undefined as part of a multiQuery request: Error: Can not verify signature for commit bagcqcerai233k3kfq7hsojmrmv77ttoedw6x7aamtgzhbharb77koxt5aata: CACAO has expired
[2022-09-12T08:14:32.874Z] WARNING: Error loading stream kjzl6kcym7w8y6zkd15l5t7ikktl7d3h5jb24v0loguta6y6zth8dfhsebmgf86 at time undefined as part of a multiQuery request: Error: Can not verify signature for commit bagcqcerakr224qveudchvci24tfuygrctu5of4yo7qtgdeq45zbnqr7domla: CACAO has expired

Is the first part the stream ID? So:

kjzl6kcym7w8y6ksgbpq7rf3vg5ea1g0q76udqmv1ocgg3h3eljfvxbpbqx8gj0
kjzl6kcym7w8y7ekf3v3yn2w856kqfxnxawapwv6033nviwsihedigheclivbbf
kjzl6kcym7w8y6n8h1s2w7k6svke1l14x8z89u32em1kv9bzviqo6bhxbumnw2u
kjzl6kcym7w8y6zkd15l5t7ikktl7d3h5jb24v0loguta6y6zth8dfhsebmgf86

Otherwise, I can monitor and see if it happens again and post here if it does.

4

Hi @nax, I would suggest running Ceramic and IPFS persistently somewhere not on local machine(s).

Running a node locally can work but you have to make sure it consistently stays up, otherwise fetching commits from your local IPFS can fail, and that causes them not to get anchored because unless the commit can be loaded, its signature can’t be verified.

Once this has happened a few times, your stream could get stuck in a bad state because updates never got anchored, and the CACAO expiration has passed for an earlier commit that failed to get anchored.

5

Hey @mohsin

Ah right, I think this could be the case. These logs are from a node that is running on a server, however sometimes I’d come back and check the server and Ceramic had stopped running. I’ve since made Ceramic run as a service, so if it stops it will restart automatically and I have noticed it has improved.

I’ll monitor again in case it happens again but hopefully that was the issue.

Thanks! :slight_smile:

1 Like
6

Ok, great, let me know how it goes!

7

It happened again :frowning: Again it was after a while, the stream ids seem to be expired and now I can’t edit them. The request that fails is:

POST /api/v0/multiqueries

Data is:

{"queries":[{"streamId":"kjzl6kcym7w8y8r6y2i2elekgvm5unmh0eof18n1t8fj1g81pl48ntmgtcw8yt6"}]}

Response is:

{"error":"Can not verify signature for commit bagcqceratnpv424r4dthhiv6jneymvgyo26geo7xadkmlihxltxatpq5ai5a: CACAO has expired"}

This has happened to multiple streams.

If I run:

❯ curl -X GET https://ceramic-clay.3boxlabs.com/api/v0/streams/kjzl6kcym7w8y8r6y2i2elekgvm5unmh0eof18n1t8fj1g81pl48ntmgtcw8yt6 | jq

I get:

{
  "error": "Indexing is not enabled"
}

:thinking:

8

Oh :confused: That’s strange, checking.

9

Can you try this against our Clay private node? Indexing is indeed not enabled on the public node.

10

Then I get:

❯ curl -X GET https://ceramic-private-clay.3boxlabs.com/api/v0/streams/kjzl6kcym7w8y8r6y2i2elekgvm5unmh0eof18n1t8fj1g81pl48ntmgtcw8yt6 | jq
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   128  100   128    0     0     43      0  0:00:02  0:00:02 --:--:--    43
{
  "error": "Can not verify signature for commit bagcqceratnpv424r4dthhiv6jneymvgyo26geo7xadkmlihxltxatpq5ai5a: CACAO has expired"
}
11

Ah, I see :confounded:

We’ve discovered/fixed a few issues related to this recently. Would it be possible for you to create fresh streams and see if the error persists?

Also, can you please make sure to update to the latest CLI/packages as suggested here and here?

Sorry for the trouble :frowning: