Error: Failed to load stream. CACAO has expired

vinsmoke · September 27, 2022, 12:44pm

Hello.
Im trying to build an app that is using ceramic to store VCs using hardcoded Aliases and DIDDataStore. Until today everything worked perfectly, but now I start getting errors: “Problem happened: Failed to load stream: kjzl6cwe1jw148ulgnabbg1vfx1slceuknt57y3gevnf35wgf4e6b15ypqj9ozv” when I try to store on ceramic.

Another error I get when I try to run the same command is “Problem happened: HTTP request to ‘https://ceramic-clay.3boxlabs.com/api/v0/multiqueries’ failed with status ‘’: {“error”:“Can not verify signature for commit bagcqcerar5z2325k6j55kzecpovmvpw24lhcnobw7snyvx6oyv57hrlishfq: CACAO has expired”}”.
Would really appreciate help!

mohsin · September 27, 2022, 2:41pm

Can you please read through this thread and update your package versions as recommended?

Please let me know if it still doesn’t work.

vinsmoke · September 28, 2022, 8:33am

I tried updating @ceramicnetwork/http-client to latest (2.4.0) and the error persists. I followed this tutorial so Im not using many other ceramic libs to update. I could not update @ceramicnetwork/blockchain-utils-linking because that breaks whole app since DIDSession is not using the latest version of the lib, making some classes incompatible.

With latest version I get this error: Problem happened: HTTP request to 'https://ceramic-clay.3boxlabs.com/api/v0/multiqueries' failed with status '': {"error":"Can not verify signature for commit bagcqcerar5z2325k6j55kzecpovmvpw24lhcnobw7snyvx6oyv57hrlishfq: CACAO has expired"}

vinsmoke · September 28, 2022, 9:08am

I switched to the latest ‘did-session’ as somebody from another thread suggested. Still get the “can not verify signature for commit” error.

These are the libs I’m using. I tried removing yarn lock and the error persists.
“@ceramicnetwork/blockchain-utils-linking”: “^2.0.12”,
“@ceramicnetwork/http-client”: “^2.3.4”,
“@glazed/did-datastore”: “^0.3.2”,
“did-session”: “^0.1.4”,
“dids”: “^3.3.0”,
“ceramic-cacao”: “^1.4.0”,

mohsin · September 28, 2022, 12:39pm

Oh, I see. @ukstv, any idea?

vinsmoke · September 29, 2022, 10:39am

I managed to fix the issue by generating a new aliases object with fresh schema and definition. The old one stopped working for some reason. Is this something that happened because of a network upgrade or can we expect schemas to stop working regularly? Currently, I use a hardcoded aliases object and it worked great until now.

mohsin · September 30, 2022, 11:45am

No, schemas should not stop working. As far as I can tell, this was a regression that we’ve fixed and should not be a regular occurrence.

Sorry for the trouble

nax · October 3, 2022, 6:33am

This is still happening for me unfortunately

ubuntu@discourse-graph-demo:~$ sudo journalctl -u ceramic | grep "bagcqceranxrqnzdh2dplolfc6ncim5zg7262onoo5lfgwrghxf3lna37iv6q"
Sep 30 10:27:20 discourse-graph-demo ceramic[13190]: [2022-09-30T10:27:20.024Z] WARNING: Anchor failed for commit bagcqceranxrqnzdh2dplolfc6ncim5zg7262onoo5lfgwrghxf3lna37iv6q of stream kjzl6kcym7w8y7mdozjvczdribcj4fvut18x5vn0aevbknu9cl3rbmgnn7zdgu5: Request has failed. Commit could not be loaded
Oct 02 17:55:24 discourse-graph-demo ceramic[13190]: [2022-10-02T17:55:24.585Z] ERROR: Error: Can not verify signature for commit bagcqceranxrqnzdh2dplolfc6ncim5zg7262onoo5lfgwrghxf3lna37iv6q: CACAO has expired

Also the same happens on the clay node:

curl -X GET https://ceramic-private-clay.3boxlabs.com/api/v0/streams/kjzl6kcym7w8y7mdozjvczdribcj4fvut18x5vn0aevbknu9cl3rbmgnn7zdgu5 | jq
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   128  100   128    0     0     40      0  0:00:03  0:00:03 --:--:--    40
{
  "error": "Can not verify signature for commit bagcqceranxrqnzdh2dplolfc6ncim5zg7262onoo5lfgwrghxf3lna37iv6q: CACAO has expired"
}

I’m using the new ceramic:

ceramic -v
2.7.0

ComposeDB:

composedb --version
@composedb/cli/0.2.4 linux-x64 node-v16.17.0

I also manually installed ceramic-cacao so it’s at version ^1.4.0.

Any ideas?

zkTRUTH · October 4, 2022, 11:53pm

This may be a known issue that I flagged over a month ago and which is supposedly going to be addressed within the next couple of weeks(?). There is an active PR here: fix(stream-tile-handler): load historical commits with CACAOs by oed · Pull Request #2428 · ceramicnetwork/js-ceramic · GitHub

Streams created by a SIWE session cannot be loaded by a specific commitID after any commit in the stream has aged over 24 hours. I would not be surprised if this is causing downstream issues beyond what I flagged for our use case.

Poking around, for example, I’m not sure this code is being used (or used as it appears) but it appears that schema streamIDs are converted to commitIDs to be loaded by the ceramic client:

(js-ceramic/schema-utils.ts at 1744d849568ae8e630345677446374e39fc04055 · ceramicnetwork/js-ceramic · GitHub)

In my experience over the last month+, this would fail with the CACAO has expired error after any commit in the schema stream ages past the signature expiration date.

Unfortunately I haven’t found a workaround after many sleepless nights, so there’s not much to do but wait for a fix.

The link that Mohsin shared is referring to a different problem, wherein all pkh signatures created before a certain date were silently deprecated and broken:

github.com

ceramicnetwork/js-ceramic/blob/e564dc37ec1e206c94e2d285f821abe88f1c0c02/packages/common/src/utils/signature_utils.ts#L8


      
          import type { Cacao } from '@didtools/cacao'
          import type { DID } from 'dids'
          import type { CommitData } from '../index.js'
          import type { StreamID } from '@ceramicnetwork/streamid'
          import { getEIP191Verifier } from '@didtools/pkh-ethereum'
          import { getSolanaVerifier } from '@didtools/pkh-solana'
          
          
const DEFAULT_CACAO_REVOCATION_PHASE_OUT = 24 * 60 * 60
          
          

          
// Register supported CACAO Verifiers
          const verifiersCACAO = {
            ...getEIP191Verifier(),
            ...getSolanaVerifier()
          }
          
          
/**
           * Utils related to cryptographic signatures

mohsin · October 9, 2022, 7:14pm

Hey @nax, @zkTRUTH, apologies for the super delayed response. I was OOO, then (and still) traveling. Just getting a chance to breathe.

In addition to the PR you linked to, @zkTRUTH, there was also another bug we fixed in release 2.8.0.

If the CAS was unreachable for any reason (e.g. network instability, CAS restart during upgrade, etc.), the Ceramic node would immediately mark the anchor request for an update as having failed, and then not retry it.

A consequence of this is one that you’re now familiar with - if the commit had an associated CACAO, since the commit didn’t get anchored, the commit would not get applied because the CACAO would be considered to have expired after 24 hours.

I know it’s been a real pain to have to constantly upgrade, but at least this issue has been fixed in release 2. 8.0

nax · October 17, 2022, 11:06am

Hey @mohsin,

No worries. I was also travelling. I am going to upgrade the packages now and hopefully that resolves the issue. Thank you!

If you don’t hear anything else from me then I guess that means it’s worked