ComposeDB Continued Support/Development

Hi!

I recently spoke with @paul and @mzk regarding some functionality in ComposeDB. Mark’s reply was that ComposeDB was no longer going to be actively maintained. Is this really true? What are the plans moving forward? I really feel the community ought to be updated on this? I like the composability and the tight integration with GraphQL including more advanced modeling features like complex N-layer arbitrary connections and traversal which is supported rather well currently and I am not sure that works out as well on orbisdb currently.

Please help us understand the current issues with ComposeDB especially with regards to the Ceramic-one transition. There is a community out here and we want to adopt the tech but news like these make us very hesitant. We have already spent a lot of effort adopting ComposeDB which seems like a very core product within the broader ceramic offering. If there is any development work required please advise how the community might be able to help moving forward!

Thank you and best regards,
Daniel

2 Likes

Hi Daniel,
Thank you for reaching out about this! I definitely understand why there may be some anxiety here about the future of Ceramic - no one wants to build on something that is going to disappear on them. Hopefully I can help clarify. The short answer is yes, ComposeDB and js-ceramic will soon be marked deprecated and will be sunsetted some time after that, in favor of ceramic-one instead. We are committed to making this a painless process however, and migrating to ceramic-one should be possible without any data loss.

For context, ComposeDB was designed to bring significantly more advanced data modeling capabilities to Ceramic - enabling developers to express more sophisticated data structures and relationships with Ceramic, by bringing a more database-like set of APIs and more sophisticated querying mechanisms. On that front I think ComposeDB was a great success, and the data modeling primitives built for ComposeDB (Model and ModelInstanceDocument Streams) are not going anywhere.

There were, however, some issues with ComposeDB as it was originally built. For one, it was added on to the existing legacy js-ceramic stack, inheriting a lot of its tech debt, performance and stability issues, as well as operational complexity to run. Perhaps the bigger issue though was the limitations around the p2p networking protocols it was using to synchronize data. At the time js-ceramic was built on top of ipfs/kubo, and used the unreliable gossipsub protocol to keep data in sync between nodes on the network. The lack of guaranteed message delivery and inability to synchronize old data using that system, led to use developing our own custom p2p protocol, Recon (CIP-124: Recon (tip synchronization protocol)).

As part of implementing Recon, we also took the opportunity to begin rewriting core pieces of the Ceramic. This is what gave birth to ceramic-one (C1), a new Ceramic implementation written in Rust. Initially C1 worked alongside js-ceramic and ComposeDB, bringing the Recon protocol to the existing Ceramic stack and tooling. Over time though, we have been able to push more and more of the core protocol down into C1. We also now have increased experience operating C1 in production under real workloads, and the results have been very promising. C1 is faster, more stable, and more powerful than js-ceramic, as well as serving as a much stronger foundation on which to continue developing and expanding what is possible with Ceramic going forward.

So far, ceramic-one has still lagged behind js-ceramic in terms of the feature set available. However we are fast approaching a point where C1 will matchs or exceed the existing ceramic/composedb stack for the most core and commonly utilized feature set. We have 2 remaining features we are still looking to bring forward to C1: schema validation and conflict resolution. Once those are released, we will likely officially deprecate js-ceramic and ComposeDB and recommend everyone switch over to using C1 on its own. We are committed to making this switch as seamless as possible. To that end, Data that was written with ComposeDB is fully compatible with ceramic-one and can be migrated into C1 without any data loss. The underlying IPLD data structures that underpin all Ceramic Streams are not changing, and so it is fully possible to use C1 to read data that was written with ComposeDB, and vice versa. There is a migration process that can move data from a ComposeDB deployment into ceramic-one and make it available to the C1 network.

While the underlying data is fully compatible and can come along to C1 from ComposeDB without issue, using C1 directly will require utilizing a new SDK and so may require some small changes to your application code. These changes should be fairly trivial and mechanical, and we’re happy to assist and provide guidance here to ease this process. Beyond that, if there are any features from ComposeDB that are important to you that are not present in C1, we’d love to hear about it and work with you to ensure your needs are met!

I hope this answers some of your questions and provides a bit more clarity. We’ve done a lot of work to ensure that no data from ComposeDB gets left behind in this transition to C1. If there are additional questions or concerns please reach out as we’d love to work with you to make the transition as smooth and seamless as possible so that you can take advantage of all the benefits that C1 will bring.

Cheers,
-Spencer

1 Like

Hi! Thanks Spencer!

So correct me for these ones:

  1. Model/ModelInstance are/will be natively supported on ceramic-one.
  2. As such present ComposeDB Data/Model Schemas will be the same or very similar
  3. New client SDK supports Models (at some point or now), this one?: GitHub - ceramicnetwork/ceramic-sdk
  4. Announcements for when/how to switch to new ceramic-one client side sdk for those interested in Models comes at some point later?

In terms of features, maybe I missed something (the n00b that I am not well versed enough on the core protocols) but interested in more ways of filtering in queries, like partial string match for instance, also of course simple “RBAC” (I know some possibilities with CACAO but have not been able to wrap my head around that for CDB).

Thanks again!

//Daniel

  • Model/ModelInstance are/will be natively supported on ceramic-one.

Yep!

  • As such present ComposeDB Data/Model Schemas will be the same or very similar

Indeed!

That’s right!

  • Announcements for when/how to switch to new ceramic-one client side sdk for those interested in Models comes at some point later?

That’s right, we should be getting those out soon. Looping in @mzk who has been working on the docs here.

As far as more advanced querying features, while C1 does have native querying capability, it will probably be a bit more limited compared to ComposeDB. OrbisDB, however, which will soon be updating to build on top of C1 directly, can provide more sophisticated query patterns and may be able to help serve some of those needs. CC @baptiste who leads OrbisDB development to comment on their querying capabilities. It is also possible with C1 to subscribe to a feed of updates and use that to maintain your own index in something like Postgres or ElasticSearch and then get whatever querying functionality you need that way as well.

More sophisticated access control is something we are definitely interested in and have done some thinking about, but is probably a bit further off as it’s a fairly complex feature.

Please let us know if there’s any other questions or concerns!

-Spencer