Changes in the protocol last month?

FrozenKiwi · October 4, 2022, 9:31pm

So the costs for running my very-light-use ceramic node spiked from $0 in august to nearly $2000 in September. The vast majority of this is write to storage costs - even a low cost-per-write skyrockets when there are millions of them.

I haven’t done any work with ceramic over this period, so I’m not sure what has happened. There would have been (max) of about 20-50 writes to my apps streams (to the best of my knowledge).

Any idea what could be happening? I’ve had to turn off our Ceramic integration until I can figure out what the hell is going on.

zkTRUTH · October 5, 2022, 12:01am

Are your ceramic daemon logs being written to storage?

I found my boot drive where my Ceramic logs were written to was rapidly filling up with 10GB worth of logs every week under .ceramic/logs/http-access.log and .ceramic/logs/pubsub.log and causing a 502 error (Note: Please don’t brick the node if I can’t write logs)

My ceramic daemon has written 12 logs (6 each) every second since my logs start on September 5th.

FrozenKiwi · October 5, 2022, 4:29pm

that’s an excellent suggestion - yes we write logs to persisted storage, so I’ll look-see

avi · October 5, 2022, 5:02pm

great suggestion, thank you!

FrozenKiwi · October 8, 2022, 9:00pm

So this seems to be the culprit (although I’ll have to wait a few days to get confirmation). The pubsub was writing 6+ lines per second. It seemed to be storing every request that came in.

In my case, I simply disabled log-to-files. Azure stores all your container output anyway, so all the interesting information (startup/config/error messages) are logged automatically.

I also tried raising the --log-level from 2 (important) to 3 (error) via cmd-line argument, but the container refused to start with

error: unknown option '--log-level'

github.com

ceramicnetwork/js-ceramic/blob/52a8c502ec1da7e920e1c83dfc0de2013fd09420/packages/cli/src/daemon-config.ts#L317


      
          export class DaemonLoggerConfig {
            /**
             * If 'logToFiles' is true, this contains the path on the local filesystem where log files will
             * be written.
             * When specifying in a config file, use the name 'log-directory'.
             */
            @jsonMember(String, { name: 'log-directory' })
            logDirectory?: string
          
          
  /**
             * Log level. Defaults to 2. Lower numbers are more verbose.
             * When specifying in a config file, use the name 'log-level'.
             */
            @jsonMember(Number, { name: 'log-level' })
            logLevel?: number
          
          
  /**
             * Controls whether logs get persisted to the file system.
             * When specifying in a config file, use the name 'log-to-files'.
             */
            @jsonMember(Boolean, { name: 'log-to-files' })

@jsonMember(Number, { name: ‘log-level’ })