Index DB consistency questions

Devtools ComposeDB
1

Hello,

We set up a service from our indexing db (sqlite) which takes the model data and syncs it to another service (elasticsearch). We figured that due to other mentions in the forum of using the DB as a place to query the data that this would make sense. However we came into some issues with the consistency of the database vs the data returned by ComposeDB.

There was one instance where a row was not in the indexing database, but was returned by Ceramic. Unfortunately (for debugging purposes) this row has in fact been added to the index. This didn’t happen in a matter of minutes though, it was not in the database for several hours whilst I was debugging the issue. I also don’t have the stream ID unfortunately so I’m not sure how much help this is for debugging.

We are also seeing inconsistencies with updates - there are currently 87 rows which have all been edited, but their stream_content has not beed updated yet in the index.

My questions regarding all this would then be;

  1. Is the problem here that things are only being updated or added to the index when they are anchored? And the anchoring is taking time for some reason.
  2. Is it recommended to use the index in this way to query it and synchronise with other services? I don’t see how it is if the data is different from the up to date data from Ceramic, because then it’s inconsistent with what’s displayed to the user.

Thanks!

1 Like
2

Hey Max, thanks for taking the time to capture these inconsistency issues - we’ll work with you to figure it out.

The team is actually traveling right now to our team retreat in Costa Rica - we might be a bit delayed in getting back to you if that’s okay. If you’re blocked I can bump up the urgency.

cc @spencer @anon94983028 @alex - fyi this is the CTO of lateral^

2 Likes
3

Hi @nax, sorry about the delay!
The issue with the stream_content not being updated in the indexing database was a bug that should be fixed in current versions.

As for whole streams not existing in the database but being returned by Ceramic, that is weirder and not something we’ve heard reported before. Unfortunately will be hard to debug if you don’t have it currently in that state or any way to reproduce. Definitely let us know if it happens again though!

4

Hey @spencer no worries!

I think we’ve experienced this issue with stream_content not being updated again. We’re using Ceramic daemon version 2.28.0.

On a request to https://ldg-node-mainnet.lateral.io/api/v0/collection with the payload:

{
    "model":"kjzl6hvfrbw6cb2jnkx7alocvot6qcl4351jl4p9jxoh4t71wy3j4m8mx82dj5l",
    "first":1000,
    "after":"eyJjcmVhdGVkX2F0IjoiMjAyMy0wMy0xNlQxNzoyMTo1NC45NTRaIn0"
}

We get from the list of edges in the response (I cherry picked one):

{
    "cursor": "eyJzdHJlYW1faWQiOiJranpsNmtjeW03dzh5OXI4Mnd0eGZ1Mzh0dDF5Mm5uOWRnOGliMmZ3ZDdrZGh0d2sydWd2amQwNDFmczkxeDEiLCJsYXN0X2FuY2hvcmVkX2F0IjoiMjAyMy0wMy0xNlQxNzo1NTo1OS4wMDBaIiwiY3JlYXRlZF9hdCI6IjIwMjMtMDMtMTZUMTc6MjM6MDcuOTIxWiJ9",
    "node": {
        "type": 3,
        "content": {
            "data": {
                "url": "",
                "label": "redacted",
                "childrenHidden": false
            },
            "type": "QuestionNode",
            "created": "2023-03-16T17:23:02.210Z",
            "position": {
                "x": 6249.009705321687,
                "y": 2081.686638932159
            },
            "lateralID": "bb602e5b-7b84-4d40-9137-08fbe4a18e9e",
            "projectID": "kjzl6kcym7w8y5ef48u98w02ivyeodzrat4ln7t4gwwiwm8gwxboulvg566rnq2",
            "deleted": true
        },
        "metadata": {
            "controllers": [
                "did:pkh:eip155:1:0x0d0a642ce2553c88a2e9b7fe5939d4b6cf69260e"
            ],
            "model": "kjzl6hvfrbw6cb2jnkx7alocvot6qcl4351jl4p9jxoh4t71wy3j4m8mx82dj5l"
        },
        "signature": 2,
        "anchorStatus": "FAILED",
        "log": [
            {
                "cid": "bagcqcerayodz32ecpkauh4ln7cs5u74pwcelyo3mbpnmphzhetqachw7xh2q",
                "type": 0,
                "expirationTime": 1686924574,
                "timestamp": 1678989359
            },
            {
                "cid": "bagcqcerae2p4v2cay7brlpmmon7at5yefpz5pbeih3qnh7vxvca3qrcwhxpq",
                "type": 1,
                "expirationTime": 1686924574,
                "timestamp": 1678989359
            },
            {
                "cid": "bafyreihr2lvtsumocyvhe63zyzzlfsqgxa52fywdhwohoa6ducabjlwkfi",
                "type": 2,
                "timestamp": 1678989359
            }
        ],
        "anchorProof": {
            "root": "bafyreiasa3lrgvwh3ufcgym2gj3san44uk4mdx7oqwh7tw2nwib4osi5gy",
            "txHash": "bagjqcgzarlm2zk5ajavbdz5npndc2felhs4jix2grn6hw6pf7pdyuqz2j3pa",
            "txType": "f(bytes32)",
            "chainId": "eip155:1"
        },
        "doctype": "MID"
    }
}

And in the database (copied it as JSON):

{
    "stream_id": "kjzl6kcym7w8y9r82wtxfu38tt1y2nn9dg8ib2fwd7kdhtwk2ugvjd041fs91x1",
    "controller_did": "did:pkh:eip155:1:0x0d0a642ce2553c88a2e9b7fe5939d4b6cf69260e",
    "stream_content": {
        "data": {
            "url": "",
            "label": "redacted",
            "childrenHidden": false
        },
        "type": "QuestionNode",
        "created": "2023-03-16T17:23:02.210Z",
        "position": {
            "x": 6249.009705321687,
            "y": 2081.686638932159
        },
        "lateralID": "bb602e5b-7b84-4d40-9137-08fbe4a18e9e",
        "projectID": "kjzl6kcym7w8y5ef48u98w02ivyeodzrat4ln7t4gwwiwm8gwxboulvg566rnq2"
    },
    "tip": "bagcqcerayodz32ecpkauh4ln7cs5u74pwcelyo3mbpnmphzhetqachw7xh2q",
    "last_anchored_at": "2023-03-16 17:55:59+00",
    "first_anchored_at": null,
    "created_at": "2023-03-16 17:23:07.921+00",
    "updated_at": "2023-03-16 18:37:04.285+00",
    "custom_projectID": "kjzl6kcym7w8y5ef48u98w02ivyeodzrat4ln7t4gwwiwm8gwxboulvg566rnq2"
}

As you can see, the "deleted": true property is missing from the database stream_content.

If I search the logs fro the stream ID I get:

Mar 22 12:25:47 discourse-graph-demo-2 ceramic[444]: [2023-03-22T12:25:47.217Z] INFO: Resumed running state for stream id: kjzl6kcym7w8y9r82wtxfu38tt1y2nn9dg8ib2fwd7kdhtwk2ugvjd041fs91x1
Mar 22 12:26:57 discourse-graph-demo-2 ceramic[444]: [2023-03-22T12:26:57.248Z] WARNING: Anchor failed for commit bafyreihr2lvtsumocyvhe63zyzzlfsqgxa52fywdhwohoa6ducabjlwkfi of stream kjzl6kcym7w8y9r82wtxfu38tt1y2nn9dg8ib2fwd7kdhtwk2ugvjd041fs91x1: Request doesn't exist

Any ideas what might be causing this? It prevents us from being able to rely on the indexing DB as accurate.

Also, we’re using PostgreSQL now.

5

Hey Nax,
So sorry about that, we did indeed have a regression slip in here. The new release going out later today will have a fix for this.

6

Release is published: Discord

Sorry about that @nax!

7

Great, thank you @spencer :slight_smile: