Index DB consistency questions


We set up a service from our indexing db (sqlite) which takes the model data and syncs it to another service (elasticsearch). We figured that due to other mentions in the forum of using the DB as a place to query the data that this would make sense. However we came into some issues with the consistency of the database vs the data returned by ComposeDB.

There was one instance where a row was not in the indexing database, but was returned by Ceramic. Unfortunately (for debugging purposes) this row has in fact been added to the index. This didn’t happen in a matter of minutes though, it was not in the database for several hours whilst I was debugging the issue. I also don’t have the stream ID unfortunately so I’m not sure how much help this is for debugging.

We are also seeing inconsistencies with updates - there are currently 87 rows which have all been edited, but their stream_content has not beed updated yet in the index.

My questions regarding all this would then be;

  1. Is the problem here that things are only being updated or added to the index when they are anchored? And the anchoring is taking time for some reason.
  2. Is it recommended to use the index in this way to query it and synchronise with other services? I don’t see how it is if the data is different from the up to date data from Ceramic, because then it’s inconsistent with what’s displayed to the user.


1 Like

Hey Max, thanks for taking the time to capture these inconsistency issues - we’ll work with you to figure it out.

The team is actually traveling right now to our team retreat in Costa Rica - we might be a bit delayed in getting back to you if that’s okay. If you’re blocked I can bump up the urgency.

cc @spencer @artur @alex - fyi this is the CTO of lateral^