Could not connect to the IPFS API

nax · March 27, 2023, 12:06pm

I am trying to run a standalone IPFS node using the instructions here:

This is the command I’m using:

sudo docker run -p 0.0.0.0:8011:8011 -p 0.0.0.0:5001:5001  -v /home/ldg/statestore:/data/ipfs --name ipfs ceramicnetwork/go-ipfs-daemon

But it’s not working correctly, if I try go to the web UI I get:

And if I try point Ceramic to the node it errors. If I look in the inspector there’s various 404 errors and some 403 errors.

The instructions on the documentation make it seem like it should work out the box, but it doesn’t. I’ve experimented with various firewall ports being open, being quite liberal but nothing seems to make a difference.

Do you have any idea what could be causing these issues?

mohsin · March 28, 2023, 12:41am

Hi @nax,

Can you please paste the startup logs from when you start the IPFS Docker container?

nax · March 28, 2023, 9:08am

Sure:

ldg@discourse-graph-ipfs:~$ sudo docker run -p 0.0.0.0:8011:8011 -p 0.0.0.0:5001:5001  -v /home/ldg/statestore:/data/ipfs --name ipfs ceramicnetwork/go-ipfs-daemon
Changing user to ipfs
ipfs version 0.18.1
generating 2048-bit RSA keypair...done
peer identity: Qma719nmtd5WgP3F52w4jGDx9kKHVYMg7jbCFWGD8DPgH1
initializing IPFS node at /data/ipfs
to get started, enter:

	ipfs cat /ipfs/QmQPeNsJPyVWPFDVHb77w8G42Fvo15z4bG2X8D2GhfbSXc/readme

sh: true: unknown operand
Repo does not require migration.
Initializing daemon...
Kubo version: 0.18.1-675f8bd-dirty
Repo version: 13
System version: amd64/linux
Golang version: go1.19.1

Computing default go-libp2p Resource Manager limits based on:
    - 'Swarm.ResourceMgr.MaxMemory': "4.2 GB"
    - 'Swarm.ResourceMgr.MaxFileDescriptors': 524288

Applying any user-supplied overrides on top.
Run 'ipfs swarm limit all' to see the resulting limits.

2023/03/28 09:07:48 failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See https://github.com/lucas-clemente/quic-go/wiki/UDP-Receive-Buffer-Size for details.
Swarm listening on /ip4/127.0.0.1/tcp/4001
Swarm listening on /ip4/127.0.0.1/tcp/8081/ws
Swarm listening on /ip4/172.17.0.2/tcp/4001
Swarm listening on /ip4/172.17.0.2/tcp/8081/ws
Swarm listening on /p2p-circuit
Swarm announcing /ip4/127.0.0.1/tcp/4001
Swarm announcing /ip4/127.0.0.1/tcp/8081/ws
Swarm announcing /ip4/172.17.0.2/tcp/4001
Swarm announcing /ip4/172.17.0.2/tcp/8081/ws

Warning: You are using an RSA Peer ID, which was replaced by Ed25519
as the default recommended in Kubo since September 2020. Signing with
RSA Peer IDs is more CPU-intensive than with other key types.
It is recommended that you change your public key type to ed25519
by using the following command:

  ipfs key rotate -o rsa-key-backup -t ed25519

After changing your key type, restart your node for the changes to
take effect.

API server listening on /ip4/0.0.0.0/tcp/5001
WebUI: http://0.0.0.0:5001/webui
Gateway (readonly) server listening on /ip4/0.0.0.0/tcp/8080
Healthcheck server listening on port 8011
Daemon is ready

mohsin · March 28, 2023, 12:48pm

Thanks.

I’ve not seen this syntax before, can you try this instead?

-p 8011:8011 -p 5001:5001

nax · March 28, 2023, 1:14pm

I tried the other syntax too. This syntax just explicitly tells Docker to bind on 0.0.0.0. I was trying it out to make sure it would bind on the hosts IP. When I use the other syntax I get the same thing:

ldg@discourse-graph-ipfs:~$ sudo docker run -p 8011:8011 -p 5001:5001  -v /home/ldg/statestore:/data/ipfs --name ipfs ceramicnetwork/go-ipfs-daemon
Changing user to ipfs
ipfs version 0.18.1
Found IPFS fs-repo at /data/ipfs
Repo does not require migration.
Initializing daemon...
Kubo version: 0.18.1-675f8bd-dirty
Repo version: 13
System version: amd64/linux
Golang version: go1.19.1

Computing default go-libp2p Resource Manager limits based on:
    - 'Swarm.ResourceMgr.MaxMemory': "4.2 GB"
    - 'Swarm.ResourceMgr.MaxFileDescriptors': 524288

Applying any user-supplied overrides on top.
Run 'ipfs swarm limit all' to see the resulting limits.

2023/03/28 13:12:57 failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See https://github.com/lucas-clemente/quic-go/wiki/UDP-Receive-Buffer-Size for details.
Swarm listening on /ip4/127.0.0.1/tcp/4001
Swarm listening on /ip4/127.0.0.1/tcp/8081/ws
Swarm listening on /ip4/172.17.0.2/tcp/4001
Swarm listening on /ip4/172.17.0.2/tcp/8081/ws
Swarm listening on /p2p-circuit
Swarm announcing /ip4/127.0.0.1/tcp/4001
Swarm announcing /ip4/127.0.0.1/tcp/8081/ws
Swarm announcing /ip4/172.17.0.2/tcp/4001
Swarm announcing /ip4/172.17.0.2/tcp/8081/ws

Warning: You are using an RSA Peer ID, which was replaced by Ed25519
as the default recommended in Kubo since September 2020. Signing with
RSA Peer IDs is more CPU-intensive than with other key types.
It is recommended that you change your public key type to ed25519
by using the following command:

  ipfs key rotate -o rsa-key-backup -t ed25519

After changing your key type, restart your node for the changes to
take effect.

API server listening on /ip4/0.0.0.0/tcp/5001
WebUI: http://0.0.0.0:5001/webui
Gateway (readonly) server listening on /ip4/0.0.0.0/tcp/8080
Daemon is ready
Healthcheck server listening on port 8011

mohsin · March 28, 2023, 4:01pm

OK, I see, that’s strange. What errors do you see in the Ceramic logs?

Also, this image was pulled recently? Did you have it pulled previously by any chance?

nax · March 28, 2023, 4:31pm

I pulled the image on a fresh instance, so it must be the latest version.

Could it be some kind of firewall issue?

The logs from the node are as follows:

Mar 27 11:02:11 discourse-graph-demo-2 ceramic[375350]: [2023-03-27T11:02:11.069Z] WARNING:
Mar 27 11:02:11 discourse-graph-demo-2 ceramic[375350]:
Mar 27 11:02:11 discourse-graph-demo-2 ceramic[375350]:             This query may have incomplete results. Affected query: {"model":{"_type":2,"_cid":{"/":"bagcqcera6mecnpzjf3lcj233fedkud2axa7ygfyrrvnl5g6izsr233xmsq4q"}},"first":1000,"after":"eyJjcmVhdGVkX2F0IjoiMjAyMy0wMy0xNlQxNzoyMTo1NC45NTRaIn0"}
Mar 27 11:02:11 discourse-graph-demo-2 ceramic[375350]:             Please check that your state store is properly configured with strong persistence guarantees.
Mar 27 11:02:11 discourse-graph-demo-2 ceramic[375350]:             This may indicate a problem with data persistence of your state store, which can result in data loss.
Mar 27 11:02:11 discourse-graph-demo-2 ceramic[375350]:             Did not find stream state for streamid kjzl6kcym7w8y910ey9jph9b6518sxjs92rork0d0ow8q97o91i0mnvvha05lvx in our state store when serving an indexed query.
Mar 27 11:02:11 discourse-graph-demo-2 ceramic[375350]: [2023-03-27T11:02:11.069Z] WARNING:
Mar 27 11:02:11 discourse-graph-demo-2 ceramic[375350]:
Mar 27 11:02:11 discourse-graph-demo-2 ceramic[375350]:             This query may have incomplete results. Affected query: {"model":{"_type":2,"_cid":{"/":"bagcqcera6mecnpzjf3lcj233fedkud2axa7ygfyrrvnl5g6izsr233xmsq4q"}},"first":1000,"after":"eyJjcmVhdGVkX2F0IjoiMjAyMy0wMy0xNlQxNzoyMTo1NC45NTRaIn0"}
Mar 27 11:02:11 discourse-graph-demo-2 ceramic[375350]:             Please check that your state store is properly configured with strong persistence guarantees.
Mar 27 11:02:11 discourse-graph-demo-2 ceramic[375350]:             This may indicate a problem with data persistence of your state store, which can result in data loss.
Mar 27 11:02:11 discourse-graph-demo-2 ceramic[375350]:             Did not find stream state for streamid kjzl6kcym7w8y7brhcxjqmtq6apdq6lv5g8duhj3phbf8hfoiidwi313xqanhsx in our state store when serving an indexed query.
Mar 27 11:02:11 discourse-graph-demo-2 ceramic[375350]: [2023-03-27T11:02:11.069Z] WARNING:
Mar 27 11:02:11 discourse-graph-demo-2 ceramic[375350]:
Mar 27 11:02:11 discourse-graph-demo-2 ceramic[375350]:             This query may have incomplete results. Affected query: {"model":{"_type":2,"_cid":{"/":"bagcqcera6mecnpzjf3lcj233fedkud2axa7ygfyrrvnl5g6izsr233xmsq4q"}},"first":1000,"after":"eyJjcmVhdGVkX2F0IjoiMjAyMy0wMy0xNlQxNzoyMTo1NC45NTRaIn0"}
Mar 27 11:02:11 discourse-graph-demo-2 ceramic[375350]:             Please check that your state store is properly configured with strong persistence guarantees.
Mar 27 11:02:11 discourse-graph-demo-2 ceramic[375350]:             This may indicate a problem with data persistence of your state store, which can result in data loss.
Mar 27 11:02:11 discourse-graph-demo-2 ceramic[375350]:             Did not find stream state for streamid kjzl6kcym7w8y8x4okmfov25933n3avcl8dc2qkcvg44i5x428a90ulrjxjpkim in our state store when serving an indexed query.
Mar 27 11:02:11 discourse-graph-demo-2 ceramic[375350]: [2023-03-27T11:02:11.069Z] WARNING:
Mar 27 11:02:11 discourse-graph-demo-2 ceramic[375350]:
Mar 27 11:02:11 discourse-graph-demo-2 ceramic[375350]:             This query may have incomplete results. Affected query: {"model":{"_type":2,"_cid":{"/":"bagcqcera6mecnpzjf3lcj233fedkud2axa7ygfyrrvnl5g6izsr233xmsq4q"}},"first":1000,"after":"eyJjcmVhdGVkX2F0IjoiMjAyMy0wMy0xNlQxNzoyMTo1NC45NTRaIn0"}
Mar 27 11:02:11 discourse-graph-demo-2 ceramic[375350]:             Please check that your state store is properly configured with strong persistence guarantees.
Mar 27 11:02:11 discourse-graph-demo-2 ceramic[375350]:             This may indicate a problem with data persistence of your state store, which can result in data loss.
Mar 27 11:02:11 discourse-graph-demo-2 ceramic[375350]:             Did not find stream state for streamid kjzl6kcym7w8y5bwaxpw3h0tiv6m352g9szig8x23e2n4ucw4ku6h4nouey967i in our state store when serving an indexed query.
Mar 27 11:02:11 discourse-graph-demo-2 ceramic[375350]: [2023-03-27T11:02:11.069Z] WARNING:
Mar 27 11:02:11 discourse-graph-demo-2 ceramic[375350]:
Mar 27 11:02:11 discourse-graph-demo-2 ceramic[375350]:             This query may have incomplete results. Affected query: {"model":{"_type":2,"_cid":{"/":"bagcqcera6mecnpzjf3lcj233fedkud2axa7ygfyrrvnl5g6izsr233xmsq4q"}},"first":1000,"after":"eyJjcmVhdGVkX2F0IjoiMjAyMy0wMy0xNlQxNzoyMTo1NC45NTRaIn0"}
Mar 27 11:02:11 discourse-graph-demo-2 ceramic[375350]:             Please check that your state store is properly configured with strong persistence guarantees.
Mar 27 11:02:11 discourse-graph-demo-2 ceramic[375350]:             This may indicate a problem with data persistence of your state store, which can result in data loss.
Mar 27 11:02:11 discourse-graph-demo-2 ceramic[375350]:             Did not find stream state for streamid kjzl6kcym7w8y66akem429aknt3trrm5qtw3a06382i97gaftj8ldvxxfpfexxc in our state store when serving an indexed query.

Thanks!

mohsin · March 28, 2023, 4:36pm

Ahh, looks like the problem is different. I’m assuming this is a test setup and you can muck with it freely.

This error is because Ceramic is able to reach IPFS but doesn’t find commits corresponding to entries in its state store.

If you can, please delete the state store directory inside the .ceramic directory where your configuration is, and try again.

nax · March 29, 2023, 12:31pm

I copied the contents of statestore to the IPFS node and ran the server with:

sudo docker run -p 8011:8011 -p 5001:5001  -v /home/ldg/statestore:/data/ipfs --name ipfs ceramicnetwork/go-ipfs-daemon

Where statestore is the folder from the Ceramic node. Then on the Ceramic node I moved the statestore from .ceramic/statestore to .ceramic/statestore-old and updated the config to point to the IPFS node:

  "state-store": {
    "mode": "remote",
    "host": "http://INTERNAL_IP:5001"
  },

But when I try query anything I get the same errors:

Mar 29 12:29:04 discourse-graph-demo-2 ceramic[505577]: [2023-03-29T12:29:04.100Z] WARNING:
Mar 29 12:29:04 discourse-graph-demo-2 ceramic[505577]:             Did not find stream state for streamid kjzl6kcym7w8y4w47x50edddt9ndwdsfww2b7gr47gql58xpd5dbxfivbtbc3ia in our state store when serving an indexed query.
Mar 29 12:29:04 discourse-graph-demo-2 ceramic[505577]:             This may indicate a problem with data persistence of your state store, which can result in data loss.
Mar 29 12:29:04 discourse-graph-demo-2 ceramic[505577]:             Please check that your state store is properly configured with strong persistence guarantees.
Mar 29 12:29:04 discourse-graph-demo-2 ceramic[505577]:             This query may have incomplete results. Affected query: {"model":{"_type":2,"_cid":{"/":"bagcqcera6mecnpzjf3lcj233fedkud2axa7ygfyrrvnl5g6izsr233xmsq4q"}},"first":1000,"after":""}
Mar 29 12:29:04 discourse-graph-demo-2 ceramic[505577]:
Mar 29 12:29:04 discourse-graph-demo-2 ceramic[505577]: [2023-03-29T12:29:04.100Z] WARNING:
Mar 29 12:29:04 discourse-graph-demo-2 ceramic[505577]:             Did not find stream state for streamid kjzl6kcym7w8y918ownsbndlfnsjrmqwus3tqblmjqjzhalh9vw9gnzsp8u2a89 in our state store when serving an indexed query.
Mar 29 12:29:04 discourse-graph-demo-2 ceramic[505577]:             This may indicate a problem with data persistence of your state store, which can result in data loss.
Mar 29 12:29:04 discourse-graph-demo-2 ceramic[505577]:             Please check that your state store is properly configured with strong persistence guarantees.
Mar 29 12:29:04 discourse-graph-demo-2 ceramic[505577]:             This query may have incomplete results. Affected query: {"model":{"_type":2,"_cid":{"/":"bagcqcera6mecnpzjf3lcj233fedkud2axa7ygfyrrvnl5g6izsr233xmsq4q"}},"first":1000,"after":""}
Mar 29 12:29:04 discourse-graph-demo-2 ceramic[505577]:
Mar 29 12:29:04 discourse-graph-demo-2 ceramic[505577]: [2023-03-29T12:29:04.100Z] WARNING:
Mar 29 12:29:04 discourse-graph-demo-2 ceramic[505577]:             Did not find stream state for streamid kjzl6kcym7w8y9jn19zba1s0eny07kxnb1jvimjinxjnkaivyp7lfe3rnc24iw2 in our state store when serving an indexed query.
Mar 29 12:29:04 discourse-graph-demo-2 ceramic[505577]:             This may indicate a problem with data persistence of your state store, which can result in data loss.
Mar 29 12:29:04 discourse-graph-demo-2 ceramic[505577]:             Please check that your state store is properly configured with strong persistence guarantees.
Mar 29 12:29:04 discourse-graph-demo-2 ceramic[505577]:             This query may have incomplete results. Affected query: {"model":{"_type":2,"_cid":{"/":"bagcqcera6mecnpzjf3lcj233fedkud2axa7ygfyrrvnl5g6izsr233xmsq4q"}},"first":1000,"after":""}
Mar 29 12:29:04 discourse-graph-demo-2 ceramic[505577]:
Mar 29 12:29:04 discourse-graph-demo-2 ceramic[505577]: [2023-03-29T12:29:04.100Z] WARNING:
Mar 29 12:29:04 discourse-graph-demo-2 ceramic[505577]:             Did not find stream state for streamid kjzl6kcym7w8y5r3q86bm806wwmulaf58jej8zanfpcoeu5iwtcak9cbqrbmsyo in our state store when serving an indexed query.
Mar 29 12:29:04 discourse-graph-demo-2 ceramic[505577]:             This may indicate a problem with data persistence of your state store, which can result in data loss.
Mar 29 12:29:04 discourse-graph-demo-2 ceramic[505577]:             Please check that your state store is properly configured with strong persistence guarantees.
Mar 29 12:29:04 discourse-graph-demo-2 ceramic[505577]:             This query may have incomplete results. Affected query: {"model":{"_type":2,"_cid":{"/":"bagcqcera6mecnpzjf3lcj233fedkud2axa7ygfyrrvnl5g6izsr233xmsq4q"}},"first":1000,"after":""}
Mar 29 12:29:04 discourse-graph-demo-2 ceramic[505577]:

nax · March 29, 2023, 12:41pm

I have a feeling that maybe copying statestore from the Ceramic node and pointing the IPFS image to it isn’t correct.

I can see on the IPFS node that when it starts it creates various like:

ldg@discourse-graph-ipfs:~/statestore$ ls -alh
total 52K
drwxr-xr-x  9 ldg      ldg 4.0K Mar 29 12:34 .
drwxr-xr-x  7 ldg ldg 4.0K Mar 29 12:28 ..
drwxr-xr-x 25 ldg      ldg 4.0K Mar 29 12:34 blocks
-rw-------  1 ldg      ldg 6.2K Mar 29 12:28 config
drwxr-xr-x  2 ldg      ldg 4.0K Mar 29 12:28 datastore
-rw-------  1 ldg      ldg  190 Mar 29 12:25 datastore_spec
drwxr-xr-x  2 ldg      ldg 4.0K Mar 29 12:20 elp
drwxr-xr-x  2 ldg      ldg 4.0K Mar 29 12:20 elp-anchor-requests
drwx------  2 ldg      ldg 4.0K Mar 29 12:25 keystore
drwxr-xr-x  2 ldg      ldg 4.0K Mar 29 12:20 testnet-clay
drwxr-xr-x  2 ldg      ldg 4.0K Mar 29 12:20 testnet-clay-anchor-requests
-rw-r--r--  1 ldg      ldg    3 Mar 29 12:25 version

The files/folders:

blocks/ config datastore datastore_spec/ keystore/ version

Are created by the IPFS image but then the others are from the statestore on Ceramic node. They don’t seem to be compatible with one another?

My reason to do this is because I want to convert from using the fs IPFS to the remote. But I don’t want to lose data so I want to migrate from one to the other…

mohsin · March 29, 2023, 2:59pm

Looks like I misread the error. There is a similar error when not finding commits in IPFS for streams in the Ceramic state store, but this particular error is related to ComposeDB.

Are you using ComposeDB/indexing?

nax · March 29, 2023, 3:04pm

Oh I’m sorry, I should have mentioned that. Yes I am using ComposeDB and it’s indexing to a PostgreSQL DB.

mohsin · March 29, 2023, 3:48pm

No worries

Just so we have a working baseline, if you revert to your previous configuration, everything works correctly?

nax · March 30, 2023, 7:03am

Yes, if I change back to the fs statestore and point it to the right directory it works fine again.

Is it an issue that the IPFS node webui says that it can’t connect to the IPFS API? You mentioned before that it looked like the ComposeDB logs indicated that it was able to connect to the IPFS node, is this still true from the other things you’ve seen?

mohsin · March 30, 2023, 3:19pm

Ok great! That’s good to know.

I don’t believe so. I just ran this command on my machine and was able to navigate to http://127.0.0.1:5001/webui. Can you try that URL (or http://localhost:5001/webui) from your browser?

docker run -p 0.0.0.0:8011:8011 -p
 0.0.0.0:5001:5001 --name ipfs ceramicnetwork/go-ipfs-daemon

Also, I just realized a couple of things that might need to be adjusted in your configuration.

From the IPFS startup logs, it seems like the config and other data from the bundled IPFS are not available at /home/ldg/statestore. IIUC, they’re more likely located at ~/.ipfs. Would you mind confirming? If so, then you can map that path to /data/ipfs when starting up the Docker image, and all your IPFS data should then be available.

The other is that to run a standalone IPFS node, you’d also need to change the ipfs section of the Ceramic instead of the state-store section.

"ipfs": {
  "mode": "remote",
  "host": "http://INTERNAL_IP:5001"
},