Running Ceramic on Cloud using Azure Kubernetes

Nodes Node Errors & Issues
1

Hello. So I am facing issues with running my ceramic node on cloud. More specifically using Azure Kubernetes. So I basically took the instructions from the simpledeploy Github repository to run ceramic on cloud. The link to the repo is below. I followed the README, and successfully got the 5 pods up and running, as shown in the ceramic documentation. I then went and did a port-forward to my localhost to run the healthcheck, and also successfully got the message “Alive!”.

Before I continue, my Azure Kubernetes is correctly configured, and when I checked the context of the kubectl, it is connected to my azure cluster and all that. I could probably be wrong though, so if you suggest that this could be the issue, I could look into it.

So to go even a step further, I tried to actually interact with the node in my project. I used the ceramic http-client to connect to the link “http://127.0.0.1:7007”, and ran my code that adds data into the node. I got this “500 (Internal Server Error)” when doing so. More specifically I got the below error. I tried looking at the logs for the ceramic-one-0, and the js-ceramic-0 pods to see if any major errors were there, and It seemed like it was fine. Especially the fact that is showed “Ceramic API running on 0.0.0.0:7007”. I linked a picture to the log of js-ceramic-0. Maybe you can find something that I missed, but it looks fine. I mean I saw this warning that the http request to the ceramic-one-0:5101/ceramic/config/network failed. Could that be a warning?. All help would be very appreciated.

The error:

“Error: HTTP request to ‘http://127.0.0.1:7007/api/v0/streams’ failed with status ‘Internal Server Error’: {“error”:“Error while storing car to IPFS for stream kjzl6cwe1jw145wybr4kkmltnov7cajlbje8echnz7s8wx15o4l94dspbkoke6m: HTTPError: Not Found”}”

“Github simpledeploy repo”

image
image1920×401 139 KB

2

Hi @HadiBishr ,

It looks like the error is due to the node running without an authenticated DID. Have you went through the steps referenced in the logs, restarted your node, and retried? It’s unlikely that you’ll be able to recover the previous stream you tried to write, but future writes should work fine.

Additionally - it looks like your setup is on testnet-clay. Was this intentional? Or are you looking to set up a node on mainnet?

3

Hey @mzk,

I appreciate the reply. So I looked into what you said and I tried to setup up an Authenticated DID. And yes, I am trying to set up a node on the Mainnet. So I went to the documentation on Ceramic which shows you how to Access the Mainnet on Ceramic (I linked it below). So I started from the beginning, running the curl command with my email, did and all that. Once I completed the step of putting that One Time Code sent to my email into the last curl command, I got the expected output Ceramic says. But then when I went to reset my js-ceramic-0 pod in Kubernetes, I didn’t see the changes apply. Meaning, I didn’t see in the config for the daemon the “node”: {} populated with the privateSeedUrl. That said I went back to that same documentation I was looking at, and went to the section “Existing Node Operators”. My question is, should I do that section to apply the privateSeedURL. Do I have to apply the seed myself?

I also want to note, looking back at my setup and the documentation, I dont think that step would have made sense to do that “Existing Node Operators” part. Because in that part I am generating a private seed and a DID, which does not make sense since when I created these pods initially, it already came with a admin DID. Do I need this privateSeedURL to be filled. What do you suggest?

I also cant seem to edit my config file. When I went into the console for the js-ceramic-0 pod, and found the daemon config file, any changes I made there using vim would not apply to the ceramic daemon. Because I went to check the logs for the pod js-ceramic-0, and it was still using the old config file. What do you think?

And I apologize for all these questions. Thank you for the help in advance.

4

Your intuition is correct - you would update your config file with the corresponding private seed info outlined in the docs (just make sure they match the one you registered with).

If you’re using ComposeDB (looks like you are based on the logs and the fact you’re using the simpledeploy repo), your config file might be called composedb.config.json (I can’t remember if simpledeploy sets this up or retains the daemon-config.json naming).

Once you apply the changes to the config file, you will likely have to restart the pod for the changes to be applied

5

So I applied what I said previously. I successfully authenticated the admin DID specified in the config file I changed using the ComposeDB seed it generated. I also added the anchor, auth-method to did. I also set the privateSeedUrl to “inplace:ed25519#”. I set the network to mainnet too.

Once I did that, I deleted all my pods so that they can reset, and when I checked the logs of the js-ceramic-0 pod, the Ceramic API was successfully running on 0.0.0.0:7007. But then after that, there were errors of “ERROR: Error loading Model kjzl6hvfrbw6cb9pd0bl7zmm28h3qszh56ccpn50vsmrl7clroy4fvln00z7q6q used to publish Node Metrics: HTTPError: block was not found locally (offline)”

If I remember correctly, someone in another topic said this was an issue on Ceramic side, and the models they have available. Is this correct? I would also like to provide you the full log of the js-ceramic-0 pod, including the config file it was reading.

[2025-01-23T18:32:15.978Z] WARNING: Running on mainnet without providing an ethereumRpcUrl is not recommended. Using the default ethereum provided may result in your requests being rate limited
[2025-01-23T18:32:15.981Z] IMPORTANT: Starting Ceramic Daemon with @ceramicnetwork/cli package version 6.11.0, with js-ceramic repo git hash 4050fc1021e9a682d588fac662d3d7ef6f2eb73e, and with config:
{
“anchor”: {
“auth-method”: “did”
},
“http-api”: {
“cors-allowed-origins”: [
“.*”
],
“admin-dids”: [
“”
]
},
“ipfs”: {
“mode”: “remote”,
“host”: “http://ceramic-one-0:5101”,
“disable-peer-data-sync”: false
},
“logger”: {
“log-level”: 2,
“log-to-files”: false
},
“metrics”: {
“prometheus-exporter-enabled”: true,
“prometheus-exporter-port”: 9464,
“metrics-exporter-enabled”: false,
“metrics-publisher-enabled”: true
},
“network”: {
“name”: “mainnet”
},
“node”: {},
“state-store”: {
“mode”: “fs”,
“local-directory”: “/js-ceramic-data/statestore”
},
“indexing”: {
“db”: “postgres://ceramic:ceramic@postgres/ceramic”,
“allow-queries-before-historical-sync”: true,
“disable-composedb”: false,
“enable-historical-sync”: false
}
}
[2025-01-23T18:32:15.982Z] IMPORTANT: Connecting to IPFS node with version “ceramic-one/0.17.0” available as /ip4/127.0.0.1/tcp/4001/p2p/12D3KooWQdSftcFf6WUmXA2rcCwYkGNtXy52CJg5yZ44W2AJjqia, /ip4/192.168.1.86/tcp/4001/p2p/12D3KooWQdSftcFf6WUmXA2rcCwYkGNtXy52CJg5yZ44W2AJjqia
[2025-01-23T18:32:15.986Z] WARNING: No pubsub querier detected, won’t be able to load tips from the network
[2025-01-23T18:32:15.987Z] IMPORTANT: Initializing PostgreSQL connection with 10 max connections
[2025-01-23T18:32:16.079Z] IMPORTANT: Node DID set to ‘did:key:z6MktGj9164daKA6M7gjz1fARb7ZcYkdC9Za8VCWXgH3Jmoj’. This DID will be used to authenticate to the anchor service
[2025-01-23T18:32:16.080Z] IMPORTANT: Connecting to ceramic network ‘mainnet’ using ceramic-one with Recon for data synchronization.
[2025-01-23T18:32:16.341Z] IMPORTANT: Creating ComposeDB config table: ceramic_models
[2025-01-23T18:32:16.432Z] IMPORTANT: Creating ComposeDB config table: ceramic_model_implements
[2025-01-23T18:32:16.433Z] IMPORTANT: Creating ComposeDB config table: ceramic_config
[2025-01-23T18:32:16.579Z] WARNING: Recon: failed to verify network with error Error: HTTP request to ‘http://ceramic-one-0:5101/ceramic/config/network’ failed with status ‘Not Found’: . This is likely due to an older version of ceramic-one and you should upgrade.
[2025-01-23T18:32:16.584Z] IMPORTANT: Connecting to bootstrap peers for network mainnet
[2025-01-23T18:32:17.203Z] IMPORTANT: Connected to anchor service ‘https://cas.3boxlabs.com’ with supported anchor chains [‘eip155:1’]
[2025-01-23T18:32:17.204Z] WARNING: No pinned streams detected. This is expected if this is the first time this node has been run, but may indicate a problem with the node’s persistence setup if it should have pinned streams
[2025-01-23T18:32:17.214Z] IMPORTANT: Ceramic API running on 0.0.0.0:7007’
[2025-01-23T18:32:17.215Z] IMPORTANT: Waiting for Model kjzl6hvfrbw6cb9pd0bl7zmm28h3qszh56ccpn50vsmrl7clroy4fvln00z7q6q used to publish Node Metrics to be available locally
[2025-01-23T18:32:20.336Z] ERROR: Error loading Model kjzl6hvfrbw6cb9pd0bl7zmm28h3qszh56ccpn50vsmrl7clroy4fvln00z7q6q used to publish Node Metrics: HTTPError: block was not found locally (offline)
[2025-01-23T18:33:59.562Z] ERROR: Error loading Model kjzl6hvfrbw6cb9pd0bl7zmm28h3qszh56ccpn50vsmrl7clroy4fvln00z7q6q used to publish Node Metrics: HTTPError: block was not found locally (offline)
[2025-01-23T18:42:31.587Z] ERROR: Error loading Model kjzl6hvfrbw6cb9pd0bl7zmm28h3qszh56ccpn50vsmrl7clroy4fvln00z7q6q used to publish Node Metrics: HTTPError: block was not found locally (offline)
[2025-01-23T18:51:03.618Z] ERROR: Error loading Model kjzl6hvfrbw6cb9pd0bl7zmm28h3qszh56ccpn50vsmrl7clroy4fvln00z7q6q used to publish Node Metrics: HTTPError: block was not found locally (offline)

6

Hi @HadiBishr,
Sorry for the delayed response, things have been quite busy over here, we’ve got some big announcements coming soon.

Anyway looking at your logs, it looks like you’re running a very old version of ceramic-one:

[2025-01-23T18:32:16.579Z] WARNING: Recon: failed to verify network with error Error: HTTP request to ‘http://ceramic-one-0:5101/ceramic/config/network’ failed with status ‘Not Found’: . This is likely due to an older version of ceramic-one and you should upgrade.

Can you try upgrading to the newest version of ceramic-one and see if that fixes things?

7

Hey @spencer,

I appreciate the response, and sorry for the late reply.

Using the simpledeploy github repo as the ceramic documentation specified, it set the ceramic-one version to be 0.17.0 in the kustomization.yaml file. I am going to try setting the tag to latest and see how that goes and will follow up after this reply.

8

@spencer

I tried setting the tag to latest, and there were issues with running the ceramic-one pod. So I think the reason why the “simpledeploy” github repo put version “0.17.0” was because of this issue.

My alternative option is to just create my own build and everything on Kubernetes, not relying on a repo like “simpledeploy”

What do you think?

9

Yeah, I’m not sure how well maintained simpledeploy is. I suspect you may have better luck just running things yourself. CC @mzk