Local node "Anchor failed for commit"

ComposeDB was not yet supported on the clay testnet 12 days ago, so anchor failures during that period would be expected. ComposeDB support was only released today. If you try again with new Models and new streams, do you still get errors anchoring?

I can’t see any output from IPFS. The process is running and I can see it using CPU/memory.

I re-created the models yesterday (I’ve installed newest versions of the libraries) and I’m still getting the errors, for example:

Sep 12 16:40:16 discourse-graph-demo ceramic[27835]: [2022-09-12T16:40:16.931Z] WARNING: Anchor failed for commit bagcqceratnzxosk5bdrwrqnk5qzhmwijn7hwcyokxe4g7hiv67twpacxb3tq of stream kjzl6kcym7w8ya52gslxoko5ve79dl0a7fa3z5rrhepo7ub4uftrhuwwjwhacs4: Request has failed. Commit could not be loaded
Sep 12 16:40:16 discourse-graph-demo ceramic[27835]: [2022-09-12T16:40:16.986Z] WARNING: Anchor failed for commit bagcqceraqubh3yi52ganlnas4o755k7x6plkqy73ufn3s4sb4httjnwszq6q of stream kjzl6kcym7w8y874lh32gffn8ogalwcystsyob2mpgwyoa4vl5o4qpeg3yokp4t: Request has failed. Commit could not be loaded

This is the full output since I added the new models and restarted the process yesterday: https://pastebin.com/raw/pra6mjae

1 Like

OK, thanks so much for the details. I’ll take a deeper look today.

Hey @nax, I just replied to your other thread here. Mind taking a look?

Hi @mohsin, thanks for that.

Are the issues related? I am still getting these messages on my node.

Hmm… I actually thought they were the same but on rereading your post, I think you’re right - this does seem to be different.

Can you run the following commands and let me know if you see something in the output of grep for each command?

The cURL command requests the list of swarm connected peers from your local IPFS daemon, and the peer IDs in the commands are from the peerlist that the Ceramic daemon will ask IPFS to swarm connect to on startup.

curl -X POST "http://127.0.0.1:5011/api/v0/swarm/peers" | grep QmWiY3CbNawZjWnHXx3p3DXsg21pZYTj4CRY1iwMkhP8r3
curl -X POST "http://127.0.0.1:5011/api/v0/swarm/peers" | grep QmSqeKpCYW89XrHHxtEQEWXmznp6o336jzwvdodbrGeLTk
curl -X POST "http://127.0.0.1:5011/api/v0/swarm/peers" | grep QmQotCKxiMWt935TyCBFTN23jaivxwrZ3uD58wNxeg5npi
curl -X POST "http://127.0.0.1:5011/api/v0/swarm/peers" | grep QmbeBTzSccH8xYottaYeyVX8QsKyox1ExfRx7T1iBqRyCd

I’m wondering if there is a connectivity issue between your local IPFS daemon and Clay nodes that is preventing the CAS from accessing commits created/stored locally.

Huh, I just started a local Ceramic/IPFS and was able to create a commit that’s accessible from the CAS without any additional steps besides updating to the latest RC (2.7.0-rc.0) and cleaning out the Ceramic state store and IPFS repo before starting the Ceramic daemon.

Can you check your network configuration? I wonder if your IPFS is unable to swarm connect properly with the Clay nodes.

Also, and this might be very verbose :grimacing: but can you add the --debug true flag to your Ceramic CLI command? If you see a bunch of pubsub messages, then things are probably networked correctly.

My locally created stream just got anchored on Clay too, not sure why your streams won’t… :confused:

Thanks for looking into this! :slight_smile:

I’m now running two nodes, one on a server that’s always running and another on my local machine, that I start on demand. When I run the cURL commands on the server I get some matches, when I run the commands on my local machine I don’t get any matches.

I agree maybe it’s to do with connectivity. For the server I’ve opened ports 80, 443, 7007, 4001, 5001 and 8080 (I looked up what ports are required for IPFS). On my local machine I’m not sure how the firewall is set up so it could be the case that some ports are being blocked from being accessed?

How are you able to check that the commits are accessible for CAS? It might be helpful for me if I create things on my local machine to be able to check that they are getting anchored.

I ran the Ceramic CLI on my local machine with the debug flag and I am seeing things like:

[Wed, 14 Sep 2022 08:08:54 GMT] service=pubsub peer=12D3KooWA4dq3YziSopypbhBiW5HXsqvt2cwWLnngMyqv9AD9kyc event=received topic=/ceramic/testnet-clay message.from=QmWiY3CbNawZjWnHXx3p3DXsg21pZYTj4CRY1iwMkhP8r3 message.seqno=17144b5540130aaa message.topicIDs.0=/ceramic/testnet-clay message.typ=0 message.doc=k2t6wyfsu4pfxxxgq2ueykz6bz62mz7d44u6631l7ha67u49gauqb4cn3jo5js message.stream=k2t6wyfsu4pfxxxgq2ueykz6bz62mz7d44u6631l7ha67u49gauqb4cn3jo5js message.tip=bafyreia6i7gmczt6q54gbo6zsrgbwogqvdt5mxpcwewl5sexhbkcw2ev5i

So that seems to be working?

Since running the daemon on a server that’s always running I’ve noticed it seems to be working better, so I think maybe the issue was running it on my local machine, and perhaps that my network is blocking some required traffic…

Yeah, I think both Windows and macOS block those ports by default and you have to explicitly open them up (I recall doing that for my Windows machine). This prevents updates from being synced across your local node and Clay nodes.

To check the anchoring status (or any status at all), you can query one of the Clay nodes for your stream like so (you might have to install curl and jq):

curl -X GET https://ceramic-clay.3boxlabs.com/api/v0/streams/k2t6wyfsu4pg0olld3hevk0xj67o5az70imph2jt2096hk5n5tmcpgs71sq7yh | jq
1 Like

Just being able to load your stream on one of the Clay nodes means that the stream was synced to them.

1 Like

We’re also experiencing this issue on our mainnet node. It’s been running consistently for a while with no changes. Our startup logs show that we’ve connected to the swarm and pubsub topic /ceramic/mainnet

I see, @zkTRUTH :frowning: Can you double-check your IPFS node? From the screenshot you posted in Discord, looks like commits could not be loaded from your IPFS node.

Following Spencer’s guidance in reference to "Signature does not belong to issuer" error - #8 by alexcarbon , I updated ceramic-cacao on my UI (and my ceramic node, just to be safe) and only receive a more explicit error:

1|ceramic daemon | [2022-09-21T04:43:19.286Z] WARNING: Error while applying commit bagcqcera4b5d5k4n4d364x3wzxte4a4ldqpcvrel6oijff4efpu2lbqjmhnq to stream k2t6wyfsu4pfznmyljas6jo9pa9vhogneopbbzcxuyj15r52uwscq6nqzhf9ih: Error: Signature does not belong to issuer
1|ceramic daemon | [2022-09-21T04:43:19.288Z] ERROR: Error: Signature does not belong to issuer

Oh :frowning:

Can you see if @ukstv’s suggestion from here (about removing the package lockfile) helps? I think that might do the trick.

Though I’m guessing you already tried that :confused: Just wanted to double-check.

Indeed I have. Removed yarn.lock, removed node_modules. Reinstalled.

As stated, this did something: my error levelled up from

WARNING: Anchor failed for commit bagcqcerawddeh7ux6jcsfzpu53dmaolfardon22bpl3ri3duafs6dgs75iwa of stream k2t6wyfsu4pfznmyljas6jo9pa9vhogneopbbzcxuyj15r52uwscq6nqzhf9ih: Request has failed. Commit could not be loaded

to also include

WARNING: Error while applying commit bagcqcerark7uv6f27mustoimzt5rwjmnanggv7xlvl7e2rvixfwfuougkhma to stream k2t6wyfsu4pfznmyljas6jo9pa9vhogneopbbzcxuyj15r52uwscq6nqzhf9ih: Error: Signature does not belong to issuer

ERROR: Error: Signature does not belong to issuer

I’ve also added yarn resolutions for the several ceramic packages which have been released since this known breaking change but still use the outdated ceramic-cacao as a dependency:

“resolutions”: {
@ceramicnetwork/blockchain-utils-linking//ceramic-cacao": “^1.4.0”,
"@ceramicnetwork/common/
/ceramic-cacao”: “^1.4.0”,
@ceramicnetwork/did-session//ceramic-cacao": “^1.4.0”,
"@ceramicnetwork/dids/
/ceramic-cacao”: “^1.4.0”
}

yarn why v1.22.5
[1/4] Why do we have the module “ceramic-cacao”…?
[2/4] Initialising dependency graph…
[3/4] Finding dependency…
[4/4] Calculating file sizes…
=> Found “ceramic-cacao@1.4.0”
info Reasons this module exists

  • project#@cambrian#app” depends on it
  • Hoisted from “project#@cambrian#app#ceramic-cacao”
  • Hoisted from “project#@cambrian#app#@ceramicnetwork#blockchain-utils-linking#ceramic-cacao”
  • Hoisted from “project#@cambrian#app#@ceramicnetwork#common#ceramic-cacao”
  • Hoisted from “project#@cambrian#app#did-session#ceramic-cacao”
  • Hoisted from “project#@cambrian#app#dids#ceramic-cacao”

I saw in Discord that it finally worked. Is that correct?

Sorry for the prolonged back-and-forth trying to figure this out :frowning:

All fixed. Described the solution here: "Signature does not belong to issuer" error - #12 by zkTRUTH

1 Like