How does data sharing between nodes work?

wings · January 10, 2023, 2:00pm

Hey there!

I’m working with a developer to build a media streaming application on ComposeDB + Ceramic, and we’ve had a lot of early success building a prototype.

One of the key features I’m looking to see from Ceramic is the ability to host a streaming platform that can “subscribe” to other sites running the same software, and vice versa. I was wondering how a scenario like this would work (context: Federation: Arbitrary subscriptions and site-to-site demonstration · Issue #4 · riffcc/ceramic-riff-web · GitHub).

My hope is that when you create a stream, other nodes unrelated to you can consume data from it, and possibly write to it in some way, but I haven’t quite understood Ceramic well enough to know how that works yet despite reading a fair bit of the documentation.

Any pointers/clarification much appreciated. Thanks!

spencer · January 11, 2023, 7:06pm

Yes, this is one of the core values of using Ceramic/ComposeDB, is that data is shareable between nodes and applications. With composeDB, all data written belongs to a given data model. Other nodes can index the same data model and then be able to read and write data into that data model, and all nodes on the network that have that model indexed will be able to discover the data.

For individual streams (which correspond to singular JSON ModelInstanceDocuments that belong to a Model), each stream has a DID that is the controller - only the end user who controls that DID can write into that Stream, but they can do so using any node or application on the network.

Does that help clarify things at all?

en0c-026 · January 13, 2023, 9:04pm

When a new node indexes a model that has already been indexed in another node, will the content already created from that other node’s tables be synchronized into the new node’s tables?
Because I’ve been testing it, and only the new content that is created appears in the new node and the content before the new node indexes the model no, I don’t know if that takes a while or doesn’t happen automatically.

spencer · January 13, 2023, 9:06pm

Currently only new data will be picked up by the indexer, but historical data sync is one of the major features that the team is working on right now. The plan is to have historical data sync ready before the mainnet launch of ComposeDB Beta next month.

avi · January 23, 2023, 9:33pm

Thanks for the great question! As Spencer mentioned, this is a big, much needed feature we’re pushing out Feb 15. Appreciate your patience

en0c-026 · January 25, 2023, 4:57am

That sound good! One question @avi @spencer. When a new document is added, eg a comment, with a relationship to another document, eg a post. creates a relationship where the comments appear in the comments field of the post document. Is there a way to break this relationship and unlink the comment given to a post? In other words it would be remove a comment from post

avi · January 25, 2023, 5:03pm

tagging in @paul - i think you might have to create a new composite Supported directives | ComposeDB on Ceramic

en0c-026 · January 25, 2023, 5:10pm

Yes, I am using that same example. I mean when creating a new comment it gets indexed into the comments field of a post given its StreamID, at this point is it possible to unlink a comment and not be indexed to the post it belonged to?

paul · January 25, 2023, 5:24pm

No this not currently supported. There may be more node-specific indexing primitives added over time to support such use-cases, but fundamentally the index acts as a cache to the data of the network.

spencer · January 26, 2023, 8:38pm

@paul could they not just set the field in the Comment that references the Post to null?

paul · January 27, 2023, 10:04am

Yes if the comment model has the post relation as optional then the relation could be removed by the author of the comment, but the use-case would be better served by completely removing the contents of the comment I think, as the comment is probably meaningless without the post relation?

en0c-026 · January 27, 2023, 11:01am

I have tried with the optional relation and I have not been able to make it work, I will try again and I will tell you.

en0c-026 · January 27, 2023, 3:08pm

Here the schema
Each Asset has an optional relationship with a User through the ownerID field.

type User @loadModel(id: "kjzl6hvfrbw6c63kw389nft0klxfoctc3x0112qid7fso2gnnmo5t9nrcasry1m") {
  id: ID!
}

type Asset @createModel(accountRelation: LIST, description: "A Asset") {
  ownerID: StreamID @documentReference(model: "User")
  owner: User @relationDocument(property: "ownerID")
  hidden: Boolean
  createdAt: String @string(maxLength: 100)
}

Then running updateAsset with an empty object, either with options.replace: true or not , I get the following errors.