Hello,
We’ve been using Firehose with great pleasure, but it occasionally crashes. Here’s the log we’re seeing on the client/consumer side:
“error Event { type: ‘error’, message: undefined }”
Image: ceramicnetwork/composedb:5.10.0
We’ve considered that this might be related to a connection keepalive limit or something similar, as suggested. However, restarting the consumer doesn’t fix the issue. The only solution so far has been to restart my Ceramic node.
Note:
I’m encountering these errors once per second.
And I’ve isolated the problem and tested the consumer by running it in the Ceramic container.
Hi,
Thanks for bringing attention to this and isolating the issue.
Just to discard some options, have you checked if you are hitting any node limitations such as CPU, memory, or file descriptor limits?
Could you tell more about the networking between the node and the client, please? I mean, there is a chance a proxy in front of a Ceramic node drops a connection.
Also, could you please elaborate on this statement: “And I’ve isolated the problem and tested the consumer by running it in the Ceramic container.” Does it mean you run the client and the node on the same container, and it fails the same?
Just in case, DMed you on Discord.
Welcome @julissa and thank you very much for the super fast response!
@ukstv. Yep same container and no proxy between them.
It could actually be a file descriptor limit. I increased the limits and tried, but it didn’t fix the issue. Now I’m migrating to EKS. If it reproduces there, I’ll write back again.
This issue was mysteriously resolved. If it happens again, I’ll revive this topic. Thanks for all the help.