Changes in the protocol last month?

So the costs for running my very-light-use ceramic node spiked from $0 in august to nearly $2000 in September. The vast majority of this is write to storage costs - even a low cost-per-write skyrockets when there are millions of them.

I haven’t done any work with ceramic over this period, so I’m not sure what has happened. There would have been (max) of about 20-50 writes to my apps streams (to the best of my knowledge).

Any idea what could be happening? I’ve had to turn off our Ceramic integration until I can figure out what the hell is going on.

Are your ceramic daemon logs being written to storage?

I found my boot drive where my Ceramic logs were written to was rapidly filling up with 10GB worth of logs every week under .ceramic/logs/http-access.log and .ceramic/logs/pubsub.log and causing a 502 error (Note: Please don’t brick the node if I can’t write logs)

My ceramic daemon has written 12 logs (6 each) every second since my logs start on September 5th.


that’s an excellent suggestion - yes we write logs to persisted storage, so I’ll look-see

great suggestion, thank you!

1 Like

So this seems to be the culprit (although I’ll have to wait a few days to get confirmation). The pubsub was writing 6+ lines per second. It seemed to be storing every request that came in.

In my case, I simply disabled log-to-files. Azure stores all your container output anyway, so all the interesting information (startup/config/error messages) are logged automatically.

I also tried raising the --log-level from 2 (important) to 3 (error) via cmd-line argument, but the container refused to start with

error: unknown option '--log-level'

@jsonMember(Number, { name: ‘log-level’ })