Error while loading capability from IPFS with CID: TimeoutError: context deadline exceeded

Nodes Node Errors & Issues
1

We are having an issue on our testnet ceramic node. Our ceramic node cannot retrieve streams that it does not have. For example, if I create a stream using the public clay testnet node, then try to retrieve this stream from our node, we see errors like this:

Timeout error while loading CID bafyreigale3m6kycvlww5m5avdpfvmccqpjlbgvz6sooqmm2oex6jg26we from IPFS. 2 retries remain
Timeout error while loading CID bafyreigale3m6kycvlww5m5avdpfvmccqpjlbgvz6sooqmm2oex6jg26we from IPFS. 1 retries remain
Timeout error while loading CID bafyreigale3m6kycvlww5m5avdpfvmccqpjlbgvz6sooqmm2oex6jg26we from IPFS. 0 retries remain
ERROR: Error while loading CID bafyreigale3m6kycvlww5m5avdpfvmccqpjlbgvz6sooqmm2oex6jg26we from IPFS: TimeoutError: context deadline exceeded
ERROR: Error: Error while loading capability from IPFS with CID bafyreigale3m6kycvlww5m5avdpfvmccqpjlbgvz6sooqmm2oex6jg26we: TimeoutError: context deadline exceeded
1 Like
2

Hi @alexcarbon, can you please share the stream ID?

Also, how are you creating streams against the public Clay node? And how are you running your IPFS node?

1 Like
3

@mohsin
I just created a new stream using the ceramic-clay.3boxlabs.com node: kjzl6cwe1jw14aloenlkqstslm6rfhft50v91t4gxjeq1n6dn2lf34e18cmoih6

await TileDocument.create(
        this.ceramic,
        data,
        {
          controllers: [this.ceramic.did.parent],
          schema: schema_commit_id,
          tags: [tag],
        },
        { pin: true, }
      );

Then, we get the error I posted before when we attempt to get this stream from our node.

We are also recently seeing this error in our production node.

We’re using:
@ceramicnetwork/cli@2.20.1
kubo_v0.15.0_linux-amd64

I found our kubo version is old. Is v0.18.1 the recommended version?

4

Wanted to make sure I’ve got all the bits in place when I roll to new IPFS version.

@mohsin

  1. What version of ceramic should we move up to when rolling to IPFS/kubo v0.18.1?
  2. Do any settings need to be updated in ceramic config?
  3. I see the following IPFS settings referenced in github goipfs deamon repo. Any other settings I might want/need to be aware of (we have a baseline config - just going to update as needed)
    ipfs config Addresses.API “/ip4/0.0.0.0/tcp/$IPFS_API_PORT”
    ipfs config Addresses.Gateway “/ip4/0.0.0.0/tcp/$IPFS_GATEWAY_PORT”
    ipfs config --json Addresses.Swarm “["/ip4/0.0.0.0/tcp/$IPFS_SWARM_TCP_PORT", "/ip4/0.0.0.0/tcp/$IPFS_SWARM_WS_PORT/ws"]”
    ipfs config --json Pubsub.Enabled true
    ipfs config Pubsub.SeenMessagesTTL 10m
    ipfs config --json Swarm.RelayClient.Enabled false
5

Thanks for the information, @alexcarbon and @lee. I’ll investigate further.

Yes, please use at least v0.18.1. I would strongly suggest using our official Docker image, which already has settings optimized for use with Ceramic. Though, you’re already aware of the config the image uses.

From this file that you’ve already seen, I’d definitely take the migrate command as well as the bootstrap peers and peering command.

Since you’ll be going from an existing v0.15.0 node, you’ll need the migration, which is pretty trivial in this case (just a config file upgrade). Adding the bootstrap peers will ensure good connectivity with our nodes.

6

To answer your questions, @lee:

  • Please use the latest version of the Ceramic CLI (2.29.0)
  • You’re on an older version of the CLI so there are likely a few changes to the config since then. Mind pasting your Ceramic daemon config (after redacting any private keys, DIDs, etc.)? I can compare that against the latest Ceramic config file structure and let you know if you need any changes.

Also, @alexcarbon, I’m assuming that you’re getting a different CID in the error now, since this is a new stream. Mind sending me the latest errors?

7

@mohsin

Here’s daemon.config.json for our production ceramic node.

{
    "anchor": {
        "ethereum-rpc-url": "xxxxxx"
    },
    "http-api": {
        "cors-allowed-origins": [
            "ceramic.carbontitle.local"
        ]
    },
    "ipfs": {
        "mode": "remote",
        "host": "http://localhost:5001"
    },
    "logger": {
        "log-level": 2,
        "log-to-files": true
    },
    "network": {
        "name": "mainnet"
    },
    "node": {},
    "state-store": {
        "mode": "fs",
        "local-directory": "/mnt/ceramic/statestore"
    }
}

let me know what I need to change in order to roll to 2.29.0 – and feel free to reference any documentation that might give me the info we require. Thanks!

1 Like
8

Thanks, @lee.

Give me a little bit more time, and I’ll send over the latest instructions. Our docs have been updated for ComposeDB recently, but you’re not using it yet so you’ll need instructions for how to upgrade to newer Ceramic while not enabling ComposeDB.

By the way, have you looked at ComposeDB? It has much nicer and more performant interface for storing/indexing/querying Ceramic data.

9

@mohsin
Thanks for the update/info! Yes, we’re aware of ComposeDB and it is definitely in our long-range plans. For the time being, yes, any info you can provide re: upgrading to a newer ceramic without enabling ComposeDB would be much appreciated

1 Like
10

@lee, here’s a simplified, fresh Ceramic daemon file using the latest version.

Can you please add the empty node section as well as the indexing section as shown below, and let me know if it works?

I’ll send over the docs once they’re in a good state to share.

{
  "anchor": {
    "ethereum-rpc-url": "xxxxxx"
    },
  "http-api": {
    "cors-allowed-origins": [
      "ceramic.carbontitle.local"
    ]
  },
  "ipfs": {
    "mode": "remote",
    "host": "http://localhost:5001"
  },
  "logger": {
    "log-level": 2,
    "log-to-files": false
  },
  "network": {
    "name": "mainnet"
  },
  "node": {},
  "state-store": {
    "mode": "fs",
        "local-directory": "/mnt/ceramic/statestore"
  },
  "indexing": {
    "db": "",
    "allow-queries-before-historical-sync": true,
    "disable-composedb": true,
    "enable-historical-sync": false
  }
}
11

Hey @lee, @alexcarbon, looks like there have been issues during the bulk upload due to a high rate of anchoring requests from your Ceramic node.

In order to avoid these issues, I’d strongly suggest rate limiting your uploads as well as upgrading your Ceramic and IPFS nodes at your earliest convenience.

The newer version of Ceramic has optimizations w.r.t. anchoring that will reduce the load on IPFS and make it less susceptible to connectivity issues. The newer version of IPFS (particularly our Docker image, which tweaks some of the packages) has several important stability enhancements.