Treat this proposal as a first draft that will evolve based on your feedback, not a final version to rubber stamp. Please let us know your thoughts as comments below!
Problem
ComposeDB currently supports loading lists of documents by their model type, their controlling account, or their related documents. However, ComposeDB lacks the ability to filter and/or order the returned documents by their fields. This has been a heavily requested feature by the community on this forum and in direct conversations.
Solution
This RFC introduces a possible solution to add simple filtering and ordering primitives to GraphQL queries as arguments on GraphQL connections, in addition to the existing pagination arguments. As a requirement for filtering and ordering, this proposal also defines how users specify the fields they want indexed.
Approach
This RFC is written with the following in mind:
- MVP â Provide an intentionally minimum set of functionalities; let the community ask for more
- UX â Queries should be easy to setup and use
- Performance â Queries should perform efficiently
- Security â Feature should be secure and introduce no vulnerabilities to the node
Example Composite
For discussion in this RFC, we will be using the following composite:
enum PublicationStatus {
DRAFT
PUBLISHED
ARCHIVED
}
type Image {
src: URI!
alt: String
}
type Post @createModel(...) {
text: String!
status: PublicationStatus!
publishedAt: Date
image: Image!
tags: [String!]
}
Indexing
To properly support query filtering and sorting, we need to define a format for indexed fields. In our approach, fields can be set to be indexed using the @createIndices directive in the composite, as seen below:
enum PublicationStatus {
DRAFT
PUBLISHED
ARCHIVED
}
type Image {
src: URI!
alt: String
}
type Post @createModel(...) {
text: String!
status: PublicationStatus!
publishedAt: Date
image: Image!
tags: [String!]
}
@createIndices(model: "Post",
indices: [
{"fields":["status"]},
{"fields":["publishedAt"]}
])
We would have the following definition for the indices field in the directive:
{
"fields": [array of field names, with nested fields specified using dot notations],
}
Supported Indices
The following types of fields will be supported for creating indices:
- Scalar (including Enum) - Top Level
- Scalar - On nested objects, such as
altinimage
The following types fields will NOT be supported for creating indices:
- Arrays - Arrays will not be indexed
For indices, we will support both single value indices and multiple value (e.g. multi-column) indices.
Design Considerations
We chose this approach to specifying indices separately in the composite file (vs directly in the models) because it separates indices from the models.
Benefits
- Multiple apps can reuse the same models, but each app can set their own indices based on their particular query patterns. This is great for model reuse and composability.
Tradeoffs
- Users could create too many indices which would negatively impact the performance of their ComposeDB deployment.
- Devs who deploy a Model to their app will need to remember to add indices themselves to support whatever queries they want to perform - the indices wonât automatically come with installing the Model.
- This is a slightly different approach than developers might be used to with popular GraphQL data tooling such as Prisma or Hasura, which uses an
@indexdirective directly in the GraphQL model.
Filtering
This section describes value filters, object filters, logic filters, and contains an example.
Value filters
Value filters apply to individual fields in a document. They are object containing a set of supported keys, but a single key must be used per filter.
The following input describes the simple boolean value filter, allowing to filter by a specific value or is the field value is null:
input BooleanValueFilterInput {
isNull: Boolean
equalTo: Boolean
}
For other scalars, the GraphQL runtime would generate the following input object for all supported scalars. The Float scalar is used here as example, and would be replaced by the specific scalar (Int, String, Date, URIâŚ) in every instance:
input FloatValueFilterInput {
isNull: Boolean
equalTo: Float
notEqualTo: Float
in: [Float!]
notIn: [Float!]
lessThan: Float
lessThanOrEqualTo: Float
greaterThan: Float
greaterThanOrEqualTo: Float
}
Enums present in the schema also get value filters generated:
enum PublicationStatus {
DRAFT
PUBLISHED
ARCHIVED
}
input PublicationStatusValueFilterInput {
isNull: Boolean
equalTo: PublicationStatus
notEqualTo: PublicationStatus
in: [PublicationStatus!]
notIn: [PublicationStatus!]
}
Object filters
Object filters contain the mapping of indexed fields to value filters in a document. This is the simple filtering we would support when this is released.
For example, our Post document described above would have the following object filter generated:
input PostObjectFilterInput {
status: PublicationStatusValueFilterInput
publishedAt: DateValueFilterInput
}
Logical filters
At a higher level, it is possible to use logical conditions to refine the filtering on a document, for example based on our PostObjectFilterInput above:
input PostFilterInput {
doc: PostObjectFilterInput
and: [PostFilterInput!]
or: [PostFilterInput!]
not: PostFilterInput
}
As with value filters, only one key/value pair must be present in the filter object.
Example Filter
The filter being recursive, conditions can be nested, for example in a GraphQL JSON input:
// Filter posts that are drafts or published in April 2023
{
"filter": {
"and": [
"or": [
{ "doc": { "status": { "equalTo": "DRAFT" } } },
{
"and": [
{ "doc": { "status": { "equalTo": "PUBLISHED" } } },
{ "doc": { "publishedAt": { "greaterThanOrEqualTo": "2023-04-01" } } },
{ "doc": { "publishedAt": { "lessThanOrEqualTo": "2023-04-30" } } }
]
}
]
]
}
}
Ordering
Ordering can be specified on an indexed field, in ascending or descending order, with null values last:
enum SortOrder {
ASC
DESC
}
As with filtering, an input type is automatically generated for each document having indexed fields.
For example, our Post document described above as:
type Post @createModel(...) {
text: String! @string(...)
status: PublicationStatus!
publishedAt: Date
image: Image!
tags: [String!]
}
@createIndices(model: "Post",
indices: [
{"fields":["status"]},
{"fields":["publishedAt"]}
])
Would have the following object generated:
input PostOrderByInput {
status: SortOrder
publishedAt: SortOrder
}
Querying
filter and orderBy arguments can be provided in all GraphQL connections, in addition to the first, last, after and before arguments used for pagination already supported by ComposeDB:
query {
postIndex(
# Existing pagination arguments
first: Int,
last: Int,
after: String,
before: String,
# Added filtering argument
filter: PostFilterInput,
# Added ordering argument
orderBy: PostOrderByInput
)
}
Downsides, Limitations, and Tradeoffs
- This proposal provides a set of operators which would work with ComposeDB, but may not include all operators that users would want. We would seek to implement the operators that would cover the majority of the ComposeDB query use cases.
- This approach requires users to learn our GraphQL operators, and how to combine them to achieve the results they want.
- Queries cannot be prepared ahead of time, so could be rejected at runtime if malformed or poorly performing.
Possible Future Features
Custom Queries
In this approach, developers specify the fields they want indexed and then write custom functions for querying data based on those indices. This approach offers greater query customization and power, however it is more complex and requires developers to interact with the specified query language. Note, for this example we use SQL, but we could support any query language.
Given a composite:
type Ball {
red: Number!
green: Number!
blue: Number!
position: Position!
}
@createIndices(model: "Ball",
indices: [
{"fields":["red"]},
{"fields":["green"]},
{"fields":["blue"]}
])
Users could then write raw queries in some query language that is decided upon by the ComposeDB team (SQL, Cypher, or custom):
input BallQuery {
desiredRed: Number!
neededGreen: Number!
wantedBlue: Number!
}
query {
@query(def: "SELECT * FROM Ball WHERE red = $desiredRed AND blue = $wantedBlue AND green = $neededGreen")
getBallsByColor(input: BallQuery!): [Ball!]
@query(def: "SELECT * FROM Ball WHERE red > 0")
getRedBalls(): [Ball!]
}
The raw queries would be prepared on the node, and the input would be passed to the prepared statement. An example usage would be:
query GetBallsByColorQuery($input: BallQuery) {
getBallsByColor(input: $input) {
position
}
}
Note that in this system users would need to learn the query language specified. Additionally, queries would be evaluated when creating composites, and composite creation could fail due to queries being malformed or poorly performing.
Inspiration
For this proposal, we have been inspired by the following data products:
- Hasura
- Prisma
- Dgraph
- Neo4J
- TypeORM
Feedback
Please comment below. Based on your feedback, we will start finalizing designs and begin implementation. Hereâs some of the things weâre interested in learning:
- How satisfied are you with this implementation?
- Does this meet all of your filtering / ordering needs and use cases?
- Which ones does it meet? Where does it fall short?
- How would you improve it?
- Do you need more complex filtering capabilities like joins and aggregations? What do you want to use them for?
- Are your query needs so specific that they can only be satisfied by custom resolvers (Custom Queries)? If so, which query language would you prefer to use (e.g. SQL, Cypher, MongoDB, custom)?
We at denoted (
