We are having an issue on our testnet ceramic node. Our ceramic node cannot retrieve streams that it does not have. For example, if I create a stream using the public clay testnet node, then try to retrieve this stream from our node, we see errors like this:
Timeout error while loading CID bafyreigale3m6kycvlww5m5avdpfvmccqpjlbgvz6sooqmm2oex6jg26we from IPFS. 2 retries remain
Timeout error while loading CID bafyreigale3m6kycvlww5m5avdpfvmccqpjlbgvz6sooqmm2oex6jg26we from IPFS. 1 retries remain
Timeout error while loading CID bafyreigale3m6kycvlww5m5avdpfvmccqpjlbgvz6sooqmm2oex6jg26we from IPFS. 0 retries remain
ERROR: Error while loading CID bafyreigale3m6kycvlww5m5avdpfvmccqpjlbgvz6sooqmm2oex6jg26we from IPFS: TimeoutError: context deadline exceeded
ERROR: Error: Error while loading capability from IPFS with CID bafyreigale3m6kycvlww5m5avdpfvmccqpjlbgvz6sooqmm2oex6jg26we: TimeoutError: context deadline exceeded
What version of ceramic should we move up to when rolling to IPFS/kubo v0.18.1?
Do any settings need to be updated in ceramic config?
I see the following IPFS settings referenced in github goipfs deamon repo. Any other settings I might want/need to be aware of (we have a baseline config - just going to update as needed)
ipfs config Addresses.API “/ip4/0.0.0.0/tcp/$IPFS_API_PORT”
ipfs config Addresses.Gateway “/ip4/0.0.0.0/tcp/$IPFS_GATEWAY_PORT”
ipfs config --json Addresses.Swarm “["/ip4/0.0.0.0/tcp/$IPFS_SWARM_TCP_PORT", "/ip4/0.0.0.0/tcp/$IPFS_SWARM_WS_PORT/ws"]”
ipfs config --json Pubsub.Enabled true
ipfs config Pubsub.SeenMessagesTTL 10m
ipfs config --json Swarm.RelayClient.Enabled false
Thanks for the information, @alexcarbon and @lee. I’ll investigate further.
Yes, please use at least v0.18.1. I would strongly suggest using our official Docker image, which already has settings optimized for use with Ceramic. Though, you’re already aware of the config the image uses.
From this file that you’ve already seen, I’d definitely take the migrate command as well as the bootstrap peers and peering command.
Since you’ll be going from an existing v0.15.0 node, you’ll need the migration, which is pretty trivial in this case (just a config file upgrade). Adding the bootstrap peers will ensure good connectivity with our nodes.
Please use the latest version of the Ceramic CLI (2.29.0)
You’re on an older version of the CLI so there are likely a few changes to the config since then. Mind pasting your Ceramic daemon config (after redacting any private keys, DIDs, etc.)? I can compare that against the latest Ceramic config file structure and let you know if you need any changes.
Also, @alexcarbon, I’m assuming that you’re getting a different CID in the error now, since this is a new stream. Mind sending me the latest errors?
Give me a little bit more time, and I’ll send over the latest instructions. Our docs have been updated for ComposeDB recently, but you’re not using it yet so you’ll need instructions for how to upgrade to newer Ceramic while not enabling ComposeDB.
By the way, have you looked at ComposeDB? It has much nicer and more performant interface for storing/indexing/querying Ceramic data.
Thanks for the update/info! Yes, we’re aware of ComposeDB and it is definitely in our long-range plans. For the time being, yes, any info you can provide re: upgrading to a newer ceramic without enabling ComposeDB would be much appreciated
Hey @lee, @alexcarbon, looks like there have been issues during the bulk upload due to a high rate of anchoring requests from your Ceramic node.
In order to avoid these issues, I’d strongly suggest rate limiting your uploads as well as upgrading your Ceramic and IPFS nodes at your earliest convenience.
The newer version of Ceramic has optimizations w.r.t. anchoring that will reduce the load on IPFS and make it less susceptible to connectivity issues. The newer version of IPFS (particularly our Docker image, which tweaks some of the packages) has several important stability enhancements.