Internal server error when reading from DIDDataStore

I am attempting to query a DIDDataStore and I seem to have an issue that is hard to debug. I am getting a fairly ambiguous error message:

{
	"level": 50,
	"time": 1662750809367,
	"pid": 28345,
	"hostname": "work-mbp.local",
	"err": {
		"type": "Error",
		"message": "HTTP request to 'https://ceramic-clay.3boxlabs.com/api/v0/streams' failed with status 'Internal Server Error': {\"error\":\"strict mode: unknown keyword: \\\"content\\\"\"}",
		"stack": "Error: HTTP request to 'https://ceramic-clay.3boxlabs.com/api/v0/streams' failed with status 'Internal Server Error': {\"error\":\"strict mode: unknown keyword: \\\"content\\\"\"}\n    at fetchJson (file:///Users/varunp/Documents/protocol/node_modules/@ceramicnetwork/common/lib/utils/http-utils.js:21:15)\n    at processTicksAndRejections (node:internal/process/task_queues:96:5)\n    at async Function.createFromGenesis (file:///Users/varunp/Documents/protocol/node_modules/@ceramicnetwork/http-client/lib/document.js:42:27)\n    at async CeramicClient.createStreamFromGenesis (file:///Users/varunp/Documents/protocol/node_modules/@ceramicnetwork/http-client/lib/ceramic-http-client.js:46:24)\n    at async TileLoader.create (file:///Users/varunp/Documents/protocol/node_modules/@glazed/tile-loader/dist/index.js:101:24)\n    at async DataModel.createTile (file:///Users/varunp/Documents/protocol/node_modules/@glazed/datamodel/dist/index.js:104:16)"
	},
	"msg": []
}

The line it is erroring at is:

store.get("backupIndex")

I initially thought it might be an issue of null initial value, but upon further investigation that does not appear to be the case. I have been unable to find any similar issues on the forum. Does anyone have any experience with this?

const modelAliases = {
  definitions: {
    backupIndex:
      "kjzl6cwe1jw1486yn4lzaisae3u7dx4nt2ubjsgxmv5bex7aotbsy194ackmvbx",
  },
  schemas: {
    Backup:
      "ceramic://k3y52l7qbv1frya9z15oyatn66xpixjcsb8zfp8hiyfdblffqyajv5qhrzmfri5mo",
    BackupIndex:
      "ceramic://k3y52l7qbv1frxqvdyuck1d7episbc6zoyup5q5bvdpwa8iso5jn2fflq8wz5a77k",
  },
  tiles: {},
};
  • The ceramic client has been initialized with a DID
  • The other relevant objects were initialized as follows:
this._loader = new TileLoader({ ceramic });
      this._dataModel = new DataModel({
        loader: this._loader,
        aliases: config.ceramicModelAliases,
      });
      this._dataStore = new DIDDataStore({
        ceramic,
        loader: this._loader,
        model: this._dataModel,
      });

@paul @anon94983028, any idea?

Can’t say much without more details, notably:

  • Does it work with a local Ceramic node?
  • Are all the used streams correctly published to Clay?
  • Can the streams be loaded using ceramic.loadStream()?
1 Like
  • It presents the same error using a local node.
  • The streams all appeared in the explorer (example: Cerscan | Explorer for the Ceramic Network).
  • I am able to load the streams using ceramic.loadStream(). This was the response for one of the streams:
{
  name: 'backupIndex',
  schema: 'ceramic://k3y52l7qbv1fryayp2rqp8kcask0mqycd7wp5p3tfqb4c2uqlhfzfabdwskw71d6o',
  description: 'SDL Data Wallet Backup Index'
}

Here is a full script demonstrating the error based off this code found linked in the ceramic projects on github: TheGame/create-model.mjs at f5295c324244f2973ea419c588e784c142a8091b · MetaFam/TheGame · GitHub. This is the most recent example project listed in the documentation, so this initialization flow generates aliases properly, but the initial error continues to persist.

import { CeramicClient } from "@ceramicnetwork/http-client";
import { DataModel } from "@glazed/datamodel";
import { ModelManager } from "@glazed/devtools";
import { DIDDataStore } from "@glazed/did-datastore";
import { TileLoader } from "@glazed/tile-loader";
import { DID } from "dids";
import { Ed25519Provider } from "key-did-provider-ed25519";
import { getResolver } from "key-did-resolver";
import { fromString } from "uint8arrays/from-string";

const CERAMIC_URL = process.env.CERAMIC_URL || "http://localhost:7007";

const BackupSchema = {
  $schema: "http://json-schema.org/draft-07/schema#",
  title: "DataWalletBackup",
  type: "object",
  properties: {
    header: {
      type: "object",
      properties: {
        hash: {
          type: "string",
        },
        timestamp: {
          type: "number",
        },
        signature: {
          type: "string",
        },
      },
    },
    blob: {
      type: "object",
      properties: {
        data: {
          type: "string",
        },
        initializationVector: {
          type: "string",
        },
      },
    },
  },
};

const BackupIndexSchema = {
  $schema: "http://json-schema.org/draft-07/schema#",
  title: "BackupIndex",
  type: "object",
  properties: {
    backups: {
      type: "array",
      items: {
        type: "object",
        properties: {
          id: {
            $ref: "#/definitions/CeramicStreamId",
          },
          timestamp: {
            type: "integer",
          },
        },
      },
    },
  },
  definitions: {
    CeramicStreamId: {
      type: "string",
      pattern: "^ceramic://.+(\\\\?version=.+)?",
      maxLength: 150,
    },
  },
};

async function run() {
  // The seed must be provided as an environment variable
  const seed = fromString(
    process.env.SEED ||
      "SEED",
    "base16",
  ); // node -e "console.log(require('crypto').randomBytes(32).toString('hex'))"
  // Connect to the local Ceramic node
  const ceramic = new CeramicClient(CERAMIC_URL);
  const provider = new Ed25519Provider(seed);
  const did = new DID({ provider, resolver: getResolver() });
  await did.authenticate();
  ceramic.did = did;

  // Create a manager for the model
  const manager = new ModelManager({ ceramic });

  // Publish the two schemas
  const [backupSchema, backupIndexSchema] = await Promise.all([
    manager.createSchema("DataWalletBackup", { content: BackupSchema }),
    manager.createSchema("BackupIndex", { content: BackupIndexSchema }),
  ]);

  // Create the definition using the created schema ID
  const backupsDefinition = await manager.createDefinition("backupIndex", {
    name: "backupIndex",
    description: "SDL Data Wallet Backup Index",
    schema: manager.getSchemaURL(backupIndexSchema),
  });

  // Write model to JSON file
  const modelAliases = await manager.deploy();
  console.log(modelAliases);

  // BEGIN TESTING
  const loader = new TileLoader({ ceramic });
  const dataModel = new DataModel({
    loader: loader,
    aliases: modelAliases,
  });
  const dataStore = new DIDDataStore({
    ceramic,
    loader,
    model: dataModel,
  });

  const index = await dataStore.get("backupIndex");
  console.log(index); // always null

  // internal server error
  const cid = await dataModel.createTile("DataWalletBackup", {
    header: { hash: "hash", timestamp: 41, signature: "sig" },
    blob: { data: "test", initializationVector: "initVector" },
  });

  // internal server error
  await dataStore.set("backupIndex", {
    backups: [{ id: cid, timestamp: 41 }],
  });

  // END TESTING
}

run().catch(console.error);

Updating to the following code for reading the data back, a new error has presented itself that I find to be quite odd:

 const modelAliases = await manager.deploy();
  console.log(modelAliases);
  const model = manager.toJSON();
  console.log(model);

  // BEGIN TESTING
  const clientManager = new ModelManager(ceramic);
  clientManager.addJSONModel(model);

  const cache = new Map();
  const loader = new TileLoader({ ceramic, cache });

  const store = new DIDDataStore({
    ceramic,
    loader,
    model: manager.model,
  });

  const index = await store.get("backupIndex");
  console.log(index); // always null

  // internal server error
  await store.set("backupIndex", {
    backups: [{ id: "id", timestamp: 41 }],
  });

Attempting to set the definition “backupIndex” in the above code results in the following message:

Error: Unable to decode multibase string "backupIndex", base36 decoder only supports inputs prefixed with k
    at Decoder.decode (file:///Users/varunp/Documents/protocol/node_modules/multiformats/esm/src/bases/base.js:30:15)
    at Codec.decode (file:///Users/varunp/Documents/protocol/node_modules/multiformats/esm/src/bases/base.js:75:25)
    at Function.fromStringNoThrow (file:///Users/varunp/Documents/protocol/node_modules/@ceramicnetwork/streamid/lib/commit-id.js:101:40)
    at Object.from (file:///Users/varunp/Documents/protocol/node_modules/@ceramicnetwork/streamid/lib/stream-ref.js:28:39)
    at TileLoader.keyToString [as _cacheKeyFn] (file:///Users/varunp/Documents/protocol/node_modules/@glazed/tile-loader/dist/index.js:55:26)
    at TileLoader.load (/Users/varunp/Documents/protocol/node_modules/dataloader/index.js:57:25)
    at TileLoader.load (file:///Users/varunp/Documents/protocol/node_modules/@glazed/tile-loader/dist/index.js:120:28)
    at DIDDataStore.getDefinition (file:///Users/varunp/Documents/protocol/node_modules/@glazed/did-datastore/dist/index.js:299:64)
    at DIDDataStore._setRecordOnly (file:///Users/varunp/Documents/protocol/node_modules/@glazed/did-datastore/dist/index.js:339:43)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)

This code is derived from the same example project here: https://github.com/MetaFam/TheGame/blob/b29b3b87f2be2a3b86a442dacafac96cab0d0b42/packages/backend/src/handlers/actions/idxCache/updateSingle.ts

The ModelManager is not meant to be used at runtime, looks like you’re misunderstanding the development process here. Please refer to Build a note-taking app with Glaze - Ceramic Developers for and end-to-end example of the process.

This is not the code we are actually running in our project. This was simply meant to demonstrate the issue that is described. If you were to actually look at the information provided you would see that this is clearly described as an example. Using the model manager to generate the exact model encoding expected should in theory isolate the logic to just the client. Even the documentation linked (which was heavily referenced in writing this) involves restoring from an encoding.

Yes the aliases generated by deploying the model need to be used at runtime, but for example this code you shared is not expected to work, the model used by the DIDDataStore needs to be a DataModel instance or the model aliases object, not models from the ModelManager instance:

Errors such as base36 decoder only supports inputs prefixed with k as you got usually mean a UTF-8 string is provided as a StreamID instead of an actual StreamID string, which in this case likely means your aliases are not properly setup.