Local node "Anchor failed for commit"

Nodes Running Nodes
1

Hey everyone!

I’ve been diving into the Ceramic world and I’m not sure I am fully understanding how the local node should work. Is it the case that data created using ComposeDB talking to a local Ceramic node should be synchronised to the Clay Testnet and appear in Cerscan?

I am able to read/write to ceramic locally using ComposeDB. The data persists locally and it seems to work fine. My understanding of Ceramic though would be that other users can also access this data from their machines as long as they have a running node to talk to. Is this correct?

I am getting error messages in the local node such as:

[2022-09-01T08:09:03.138Z] WARNING: Anchor failed for commit bagcqcerazpsqearn6zzg5fcwilu3koupdoz35gdz4etf35rva5ndklmusgmq of stream kjzl6hvfrbw6ca3fgrxe8nutu80imgx9a5tquas7r5pdetshpizrh7jmhgz11nd: Request has failed. Commit could not be loaded

This is always the case, I don’t think a commit has been successful. I suspect this might be causing the issues.

I looked a bit into the source of ceramic and logged the anchor status response from the above message:

{"status":4,"streamId":{"_type":2,"_cid":{"code":133,"version":1,"hash":{"0":18,"1":32,"2":203,"3":229,"4":2,"5":2,"6":45,"7":246,"8":114,"9":110,"10":148,"11":86,"12":66,"13":233,"14":181,"15":58,"16":143,"17":27,"18":179,"19":190,"20":152,"21":121,"22":225,"23":38,"24":93,"25":246,"26":53,"27":7,"28":90,"29":53,"30":45,"31":148,"32":145,"33":153}}},"cid":{"code":133,"version":1,"hash":{"0":18,"1":32,"2":203,"3":229,"4":2,"5":2,"6":45,"7":246,"8":114,"9":110,"10":148,"11":86,"12":66,"13":233,"14":181,"15":58,"16":143,"17":27,"18":179,"19":190,"20":152,"21":121,"22":225,"23":38,"24":93,"25":246,"26":53,"27":7,"28":90,"29":53,"30":45,"31":148,"32":145,"33":153}},"message":"Request has failed. Commit could not be loaded"}

Any ideas what the issue here might be?

I’d also appreciate any info on how the nodes are expected to run. If I am running a local node, should this data still be synced to the Ceramic network? I understand we’re to keep a node running for a local index of our composites, but it seems odd if the data is only stored on that node.

2

Can you share how you are running your Ceramic node? What CLI flags are you using and can you also share the contents of the config file (defaults to ~/.ceramic/daemon.config.json)

3

I’m running it like this:

CERAMIC_ENABLE_EXPERIMENTAL_INDEXING=true ceramic daemon

And my config file looks like this:

{
  "anchor": {},
  "http-api": {
    "cors-allowed-origins": [
      ".*"
    ]
  },
  "ipfs": {
    "mode": "bundled"
  },
  "logger": {
    "log-level": 2,
    "log-to-files": false
  },
  "metrics": {
    "metrics-exporter-enabled": false,
    "metrics-port": 9090
  },
  "network": {
    "name": "testnet-clay"
  },
  "node": {},
  "state-store": {
    "mode": "fs",
    "local-directory": "/Users/max/.ceramic/statestore/"
  },
  "indexing": {
    "db": "sqlite:///Users/max/.ceramic/indexing.sqlite",
    "allow-queries-before-historical-sync": true,
    "models": ["kjzl6hvfrbw6cajhydppvu1pek0zoygujchp1ax3i9yov23m28pi0wrhmk0y2r3","kjzl6hvfrbw6c6ysl0of2jl4zwhb0kqggilgbp7fs07hfgdcv8g7lwrkvhflkq9"]
  }
}

I think it’s just the default one and I added two models.

1 Like
4

Can you please send us your daemon startup logs all the way till it says “listening on port 7007”?

5

Yes this is the output:

❯ CERAMIC_ENABLE_EXPERIMENTAL_COMPOSE_DB=true ceramic daemon
[2022-09-12T07:35:21.615Z] IMPORTANT: Initializing SQLite connection
[2022-09-12T07:35:21.721Z] IMPORTANT: Starting Ceramic Daemon at version 2.6.1 with config:
{
  "anchor": {},
  "http-api": {
    "cors-allowed-origins": [
      ".*"
    ]
  },
  "ipfs": {
    "mode": "bundled"
  },
  "logger": {
    "log-level": 2,
    "log-to-files": false
  },
  "metrics": {
    "metrics-exporter-enabled": false,
    "metrics-port": 9090
  },
  "network": {
    "name": "testnet-clay"
  },
  "node": {},
  "state-store": {
    "mode": "fs",
    "local-directory": "/Users/max/.ceramic/statestore/"
  },
  "indexing": {
    "db": "sqlite:///Users/max/.ceramic/indexing.sqlite",
    "allow-queries-before-historical-sync": true,
    "models": [
      "kjzl6hvfrbw6cacfkv2unmdmcsa6z74nm2r25vhpa0yq9i8pq0d9730ypnk6edk",
      "kjzl6hvfrbw6c6fp3mk68xjfuj3jhr6x59kmtm68thdxf8wtr3nuomafju627qx",
      "kjzl6hvfrbw6c6mhbs6cpogcriqn3p7ekr766ga9wgbjsy63ocl3uj6ael992lh"
    ]
  }
}
[2022-09-12T07:35:21.753Z] IMPORTANT: Connecting to ceramic network 'testnet-clay' using pubsub topic '/ceramic/testnet-clay'
[2022-09-12T07:35:21.753Z] WARNING: Warning: indexing and query APIs are experimental and still under active development.  Please do not create Composites, Models, or ModelInstanceDocument streams, or use any of the new GraphQL query APIs on mainnet until they are officially released
[2022-09-12T07:37:33.343Z] IMPORTANT: Connected to anchor service 'https://cas-clay.3boxlabs.com' with supported anchor chains ['eip155:100']
[2022-09-12T07:37:35.279Z] IMPORTANT: Ceramic API running on 0.0.0.0:7007'
6

Huh, is there any output related to the IPFS node?

7

ComposeDB was not yet supported on the clay testnet 12 days ago, so anchor failures during that period would be expected. ComposeDB support was only released today. If you try again with new Models and new streams, do you still get errors anchoring?

8

I can’t see any output from IPFS. The process is running and I can see it using CPU/memory.

I re-created the models yesterday (I’ve installed newest versions of the libraries) and I’m still getting the errors, for example:

Sep 12 16:40:16 discourse-graph-demo ceramic[27835]: [2022-09-12T16:40:16.931Z] WARNING: Anchor failed for commit bagcqceratnzxosk5bdrwrqnk5qzhmwijn7hwcyokxe4g7hiv67twpacxb3tq of stream kjzl6kcym7w8ya52gslxoko5ve79dl0a7fa3z5rrhepo7ub4uftrhuwwjwhacs4: Request has failed. Commit could not be loaded
Sep 12 16:40:16 discourse-graph-demo ceramic[27835]: [2022-09-12T16:40:16.986Z] WARNING: Anchor failed for commit bagcqceraqubh3yi52ganlnas4o755k7x6plkqy73ufn3s4sb4httjnwszq6q of stream kjzl6kcym7w8y874lh32gffn8ogalwcystsyob2mpgwyoa4vl5o4qpeg3yokp4t: Request has failed. Commit could not be loaded

This is the full output since I added the new models and restarted the process yesterday: https://pastebin.com/raw/pra6mjae

1 Like
9

OK, thanks so much for the details. I’ll take a deeper look today.

10

Hey @nax, I just replied to your other thread here. Mind taking a look?

11

Hi @mohsin, thanks for that.

Are the issues related? I am still getting these messages on my node.

12

Hmm… I actually thought they were the same but on rereading your post, I think you’re right - this does seem to be different.

13

Can you run the following commands and let me know if you see something in the output of grep for each command?

The cURL command requests the list of swarm connected peers from your local IPFS daemon, and the peer IDs in the commands are from the peerlist that the Ceramic daemon will ask IPFS to swarm connect to on startup.

curl -X POST "http://127.0.0.1:5011/api/v0/swarm/peers" | grep QmWiY3CbNawZjWnHXx3p3DXsg21pZYTj4CRY1iwMkhP8r3
curl -X POST "http://127.0.0.1:5011/api/v0/swarm/peers" | grep QmSqeKpCYW89XrHHxtEQEWXmznp6o336jzwvdodbrGeLTk
curl -X POST "http://127.0.0.1:5011/api/v0/swarm/peers" | grep QmQotCKxiMWt935TyCBFTN23jaivxwrZ3uD58wNxeg5npi
curl -X POST "http://127.0.0.1:5011/api/v0/swarm/peers" | grep QmbeBTzSccH8xYottaYeyVX8QsKyox1ExfRx7T1iBqRyCd

I’m wondering if there is a connectivity issue between your local IPFS daemon and Clay nodes that is preventing the CAS from accessing commits created/stored locally.

14

Huh, I just started a local Ceramic/IPFS and was able to create a commit that’s accessible from the CAS without any additional steps besides updating to the latest RC (2.7.0-rc.0) and cleaning out the Ceramic state store and IPFS repo before starting the Ceramic daemon.

Can you check your network configuration? I wonder if your IPFS is unable to swarm connect properly with the Clay nodes.

Also, and this might be very verbose :grimacing: but can you add the --debug true flag to your Ceramic CLI command? If you see a bunch of pubsub messages, then things are probably networked correctly.

15

My locally created stream just got anchored on Clay too, not sure why your streams won’t… :confused:

16

Thanks for looking into this! :slight_smile:

I’m now running two nodes, one on a server that’s always running and another on my local machine, that I start on demand. When I run the cURL commands on the server I get some matches, when I run the commands on my local machine I don’t get any matches.

I agree maybe it’s to do with connectivity. For the server I’ve opened ports 80, 443, 7007, 4001, 5001 and 8080 (I looked up what ports are required for IPFS). On my local machine I’m not sure how the firewall is set up so it could be the case that some ports are being blocked from being accessed?

How are you able to check that the commits are accessible for CAS? It might be helpful for me if I create things on my local machine to be able to check that they are getting anchored.

I ran the Ceramic CLI on my local machine with the debug flag and I am seeing things like:

[Wed, 14 Sep 2022 08:08:54 GMT] service=pubsub peer=12D3KooWA4dq3YziSopypbhBiW5HXsqvt2cwWLnngMyqv9AD9kyc event=received topic=/ceramic/testnet-clay message.from=QmWiY3CbNawZjWnHXx3p3DXsg21pZYTj4CRY1iwMkhP8r3 message.seqno=17144b5540130aaa message.topicIDs.0=/ceramic/testnet-clay message.typ=0 message.doc=k2t6wyfsu4pfxxxgq2ueykz6bz62mz7d44u6631l7ha67u49gauqb4cn3jo5js message.stream=k2t6wyfsu4pfxxxgq2ueykz6bz62mz7d44u6631l7ha67u49gauqb4cn3jo5js message.tip=bafyreia6i7gmczt6q54gbo6zsrgbwogqvdt5mxpcwewl5sexhbkcw2ev5i

So that seems to be working?

Since running the daemon on a server that’s always running I’ve noticed it seems to be working better, so I think maybe the issue was running it on my local machine, and perhaps that my network is blocking some required traffic…

17

Yeah, I think both Windows and macOS block those ports by default and you have to explicitly open them up (I recall doing that for my Windows machine). This prevents updates from being synced across your local node and Clay nodes.

To check the anchoring status (or any status at all), you can query one of the Clay nodes for your stream like so (you might have to install curl and jq):

curl -X GET https://ceramic-clay.3boxlabs.com/api/v0/streams/k2t6wyfsu4pg0olld3hevk0xj67o5az70imph2jt2096hk5n5tmcpgs71sq7yh | jq
1 Like
18

Just being able to load your stream on one of the Clay nodes means that the stream was synced to them.

1 Like
19

We’re also experiencing this issue on our mainnet node. It’s been running consistently for a while with no changes. Our startup logs show that we’ve connected to the swarm and pubsub topic /ceramic/mainnet

20

I see, @zkTRUTH :frowning: Can you double-check your IPFS node? From the screenshot you posted in Discord, looks like commits could not be loaded from your IPFS node.

image
image1546×746 209 KB