RFC: Enhanced Composite Management and Sharing in S3 Console

Devtools ComposeDB
1

Date: 2024-01-18
Author: US3R-NETWORK
Status: Draft

Abstract: This RFC proposes to integrate a comprehensive composite management service into the S3 Console, enhancing dApp developers’ ability to create, manage, share, and deploy composite data structures. The solution enables users to initially design and store composite schemas in an S3 centralized database and migrate them to the Ceramic network for sharing and reuse when projects mature.

1. Functional Requirements - Composite Creation & Management

1.1 Centralized Phase

  • Composite Editor: Offer a visual UI/UX for creating, editing, and saving composite schemas to the S3 centralized database.

  • Include a GraphQL Query Builder: Provide an integrated tool that allows developers to easily test GraphQL queries against their defined composite structures. This feature should enable developers to validate query correctness and preview the expected response format before deploying the schema in a production environment.

  • Generate and Download ComposeDB Development Scaffold: Introduce functionality to automatically generate and package runtime definitions compatible with various frameworks or libraries for the created composites. Developers should be able to download these artifacts directly from the S3 Console, including:

    • Runtime Definition Files: The generated files will encapsulate the necessary data models and mappings required by application runtimes.
    • TypeScript Type Definitions: Accompanying TypeScript type definitions (.d.ts files) to provide strong typing support during development within TypeScript-based projects. (v2)
    • ComposeDB Client Generation: Automate the creation and authorization of client-side code to interact with ComposeDB, the underlying data storage layer for composite structures. Developers will be able to quickly generate pre-configured client-side libraries that handle authentication and CRUD operations against the ComposeDB.(v2)
    • Standard CRUD Operations Boilerplate: Include ready-to-use code snippets or modules for performing standard Create, Read, Update, and Delete (CRUD) operations on composite data. This feature ensures developers have out-of-the-box functionality to manage their data without having to write custom logic from scratch. (v2)

    These additions will empower developers with practical tools for immediate validation and seamless integration of their composite schemas into their applications’ codebase, streamlining the development process and improving developer productivity.

  • Implement version control features to facilitate schema iteration and historical rollback. (v2)

1.2 Decentralized Migration & Publishing

  • Introduce a one-click publishing mechanism to securely upload composite schemas from S3 to the Ceramic network, obtaining a unique identifier (Stream ID).

    Composite Model Schema

    type Composite 
@createModel(accountRelation: LIST, description: "Composite information of ComposeDB")
@createIndex(fields: [{ path: "name" }])
@createIndex(fields: [{ path: "description" }])
@createIndex(fields: [{ path: "models" }])
@createIndex(fields: [{ path: "revoke" }])
@createIndex(fields: [{ path: "createdAt" }])
@createIndex(fields: [{ path: "modifiedAt" }]) {
  creator: DID! @documentAccount // Creator's DID identifier
  version: CommitID! @documentVersion // Current version's Commit ID
  name: String! @string(minLength: 3, maxLength: 100) // Composite name
  description: String @string(maxLength: 10000) // Description of the composite
  models: [StreamID!]! // List of base model Stream IDs composing this composite
  schema: String! @string(maxLength: 100000) // Graphql Schema for the composite
	composite: String! @string(maxLength: 100000) // EncodedComposite for the composite
  runtimeDefinition: String! @string(maxLength: 100000) // Runtime definition possibly including specific logic for parsing, validating, and processing composite data
  revoke: Boolean // Whether to revoke the public availability of this composite model
  createdAt: DateTime // Creation timestamp
  modifiedAt: DateTime // Last modification timestamp
}

  • Support setting permissions for schemas published on the Ceramic network and provide options for review processes. (v2)

2. Functional Requirements - Composite Discovery & Deployment

2.1 Composite Explore Page

  • Develop a Composite Explore page featuring search tools and detailed listings that help developers find and filter out high-quality, published composite schemas.
  • Incorporate rating systems and popular recommendations to promote discovery and usage of top-tier composites.

2.2 Private Ceramic Node Deployment

  • Add a one-click deployment feature allowing selected composite schemas to be directly synced to a developer’s private Ceramic node.
  • Configure and authenticate private node information to ensure secure deployment.

3. Security and Privacy Protection

  • Ensure security measures and privacy protection are implemented throughout all composite operations. (v2)

4. Compatibility and Usability

  • Ensure new features are compatible with existing S3 Console services without impacting their stability or user experience.
  • Provide thorough usage guides and tutorials to assist developers in making the most of the composite management and service functionalities.

5. References

This RFC focuses on improving the management and sharing of composite data structures within the S3 Console, providing a more convenient and efficient development environment for dApp developers. We welcome community members to engage in discussions and refine this proposal.

1 Like
2

@Mark @baptiste @msena @spencer would love any feedback from the community

1 Like
3

Thanks for sharing @0xA25532B1287dEe6501 and @0xEE3CA4dd4CeB341691! This is going to be super helpful towards making it easier for developer to discover composites directly from the network instead of their out of band means today. I’ll leave it to @spencer @mzk @dbcfd and @paul on the technical/schema side but from a flow perspective I think this roughly makes sense. Looking forward to testing it out.

4

Hello @0xEE3CA4dd4CeB341691 !

Thanks so much for sharing and tagging me. This appears to echo what we’ve discussed over the past two standups - I see everything from the social validation we discussed to usage guides and tutorials.

I particularly like the idea you proposed around TypeScript Type Definitions - I think little features like this have great potential to significantly accelerate workflow for devs in our community.

One question I have - for users using both the centralized and decentralized phases of this project, how will they know what ComposeDB versions the nodes are running? For example, just yesterday, I tried to deploy a set of models that included interfaces. console.s3 rejected this (which I assume is because the versioning has not been updated yet). Will the user be able to choose which versions their one-click nodes will run?

Another idea (not sure what usage would actually look like, but it would be fun as a social experiment/potential stretch feature) - along with the rating system, it might be interesting to allow users to comment on certain artifacts (models, composites, etc). Certainly not as important as what you’ve outlined above, but it might be fun to promote additional ways to allow community members to engage with each other.

1 Like
5

This seems like a super convenient tool!

A question on the composite DM you are suggesting. Why are you defining indexes on almost all fields? In particular,

  • description, will anyone ever be able to know the full description of a composite so that they can query based on it
  • models, not sure if it’s possible to index on arrays, if it is you might have to include all models in your query to match. This is an interesting problem because it would be nice to be able to find all composites that includes a specific model
1 Like
6

I’m pretty sure ComposeDB does not support indexing or querying on array fields currently.

1 Like
7

So the models field can be removed, as it was originally intended for searching purposes, and information about the models can be extracted from within the runtimeDefinition.

8

The description field is indexed to support fuzzy search; however, if LIKE op for strings is currently not supported, then there’s no need to index this field.

9

Hey, sounds great to have composites discovery using a model like this!
I think there are potential issues with the suggested design though:

1. models list

As mentioned by Joel and Spencer, the models array couldn’t be indexed if they are embedded in the document, however using another model to store the relation would enable discovery, for example:

type CompositeModel @createModel(description: "Composite and Model relation", accountRelation: SET, accountRelationFields: ["compositeID", "modelID"]) {
  compositeID: StreamID! @documentReference(model: "Node")
  # View to access the referenced Composite document
  composite: Node! @relationDocument(property: "compositeID")
  modelID: StreamID!
}

type Composite
# ... model directives
{
  # View to access the CompositeModel documents using this composite
  compositeModels: [CompositeModel]! @relationFrom(model: "CompositeModel", property: "compositeID")
  # ... other model fields
}

Note that here using the SET account relation would ensure that the composite/model relation would be unique for a given account, which would avoid having duplicate models when querying for the relations created by the account controlling the composite.
The SET account relation is not available yet, but is a work in progress and should be shipped in the coming weeks.

2. revoke field

I think the use-case for the revoke field could be covered by the “unindex” feature we are currently working on to indicate a document should be removed from indexers, so it shouldn’t be discoverable from indexer queries (though it would still be loadable directly by stream ID)?

If so, this is also a functionality that is not available yet but is being worked on and should be shipped in the coming weeks along with the SET account relation.

3. runtimeDefinition and schema

Both the ComposeDB runtime definition and schema depend on the composite definition, but also the version of the ComposeDB runtime being used. As such, they should always be generated by the ComposeDB tools to make sure they work for the target runtime.
Having these be stored in a document and used directly would most likely tie to a specific runtime version and cause problems when attempting to update.

4. composite string

The encoded composite basically contains 2 things: the JSON-encoded commits for the models and additional metadata about the composite, such as its version, common embeds and additional views.

Unlike the runtimeDefinition and schema, it should be safe to store and reuse the encoded definition as its format shouldn’t change much and the current format is likely to continue being supported by the ComposeDB devtools, but storing the models commits here seems a bit wasteful as ideally the models themselves should be accessible on the Ceramic node and loadable directly?

1 Like
10

Very happy to hear about SET account relation and “unindex” feature!

11

using another model to store the relation of models and composites make sense.

12

we would remvoe runtimeDefinition field.
Does ComposeDB dev-tools provide a tool/function to convert composite definitions into their original graphql schema format(includeing composeDB directives/types)?
We store the schema primarily for the purpose of sharing it with other developers later, allowing them to make slight modifications as needed to suit their requirements.

1 Like
13

If by “original graphql schema format” you mean the composite definition schema containing notably the @createModel directive then no, the devtools don’t provide a way to generate it from a composite.

Basically, the composite definition schema can be seen as a factory for the composite, once the composite is created it can be manipulated directly (combined with other composites, encoded to/from JSON, compiled to a runtime definition, etc.), but the definition schema itself is an abstraction for the composite, not its canonical structure, if that makes sense?

1 Like
14

What’s the convention/canonical structure to share composite, the encoded json?

1 Like
15

To share as a file, yes.

2 Likes