devtools.Composite.fromModels does not support filter

0xA25532B1287dEe6501 · October 13, 2023, 3:04am

In order to support filter and order in our dapp, we need to get encoded composite definition from already existing model stream ids.
but calling devtools.Composite.fromModels dose NOT return indices Field.

you could reproduce this using following model
https://s3.xyz/models/modelview/kjzl6hvfrbw6c5sfh4zjss0pmc5x99sjdjbcu7zz8up5t7k0yvsybwvn2701o6k?network=MAINNET

after run cli command

composedb composite:from-model  kjzl6hvfrbw6c5sfh4zjss0pmc5x99sjdjbcu7zz8up5t7k0yvsybwvn2701o6k kjzl6hvfrbw6c6vwzcigxgsnskmtkwodxnxzy0fzasxs6apumpv67q9sip974ry

you will get composite schema, which missing indices

{
    "version": "1.0",
    "models": {
        "kjzl6hvfrbw6c5sfh4zjss0pmc5x99sjdjbcu7zz8up5t7k0yvsybwvn2701o6k": [
            {
                "jws": {
                    "payload": "AXESIG6P7pKKzNQCveq2jabKCyt5biJLz_SiaaqZAG8iVDU6",
                    "signatures": [
                        {
                            "signature": "9pW3prqQ7_jjHT9mV8rvmOtsQxCf_6KUcH3NSAomc6Y48ENfzA1fetMmkYBem88HsG9EE4atTkIbylfG8C6dCw",
                            "protected": "eyJhbGciOiJFZERTQSIsImtpZCI6ImRpZDprZXk6ejZNa21tSzlHVndLZkVtWDRtSkhjblFmaUQ3NlNTQ3k2Mk41NGJZWHNkRDVyTGVHI3o2TWttbUs5R1Z3S2ZFbVg0bUpIY25RZmlENzZTU0N5NjJONTRiWVhzZEQ1ckxlRyJ9"
                        }
                    ],
                    "link": "bafyreidor7xjfcwm2qbl32vwrwtmuczlpfxces6p6srgtkuzabxsevbvhi"
                },
                "linkedBlock": "omRkYXRhp2RuYW1lZ0Jpb0xpbmtldmlld3OjZ2NyZWF0b3KhZHR5cGVvZG9jdW1lbnRBY2NvdW50Z3Byb2ZpbGWjZHR5cGVwcmVsYXRpb25Eb2N1bWVudGVtb2RlbHg/a2p6bDZodmZyYnc2YzZ2d3pjaWd4Z3Nuc2ttdGt3b2R4bnh6eTBmemFzeHM2YXB1bXB2NjdxOXNpcDk3NHJ5aHByb3BlcnR5aXByb2ZpbGVJRGd2ZXJzaW9uoWR0eXBlb2RvY3VtZW50VmVyc2lvbmZzY2hlbWGmZHR5cGVmb2JqZWN0ZSRkZWZzoW9DZXJhbWljU3RyZWFtSUSjZHR5cGVmc3RyaW5nZXRpdGxlb0NlcmFtaWNTdHJlYW1JRGltYXhMZW5ndGgYZGckc2NoZW1heCxodHRwczovL2pzb24tc2NoZW1hLm9yZy9kcmFmdC8yMDIwLTEyL3NjaGVtYWhyZXF1aXJlZINocGxhdGZvcm1maGFuZGxlaXByb2ZpbGVJRGpwcm9wZXJ0aWVzpWRkYXRhomR0eXBlZnN0cmluZ2ltYXhMZW5ndGgaAAGGoGZoYW5kbGWiZHR5cGVmc3RyaW5naW1heExlbmd0aBhkZ25ldHdvcmuiZHR5cGVmc3RyaW5naW1heExlbmd0aBhkaHBsYXRmb3JtomR0eXBlZnN0cmluZ2ltYXhMZW5ndGgYZGlwcm9maWxlSUShZCRyZWZ3Iy8kZGVmcy9DZXJhbWljU3RyZWFtSUR0YWRkaXRpb25hbFByb3BlcnRpZXP0Z3ZlcnNpb25jMS4waXJlbGF0aW9uc6FpcHJvZmlsZUlEomR0eXBlaGRvY3VtZW50ZW1vZGVseD9ranpsNmh2ZnJidzZjNnZ3emNpZ3hnc25za210a3dvZHhueHp5MGZ6YXN4czZhcHVtcHY2N3E5c2lwOTc0cnlrZGVzY3JpcHRpb254HXNvY2lhbCBsaW5rcyBmb3IgdXMzci1uZXR3b3Jrb2FjY291bnRSZWxhdGlvbqFkdHlwZWRsaXN0ZmhlYWRlcqNjc2VwZW1vZGVsZW1vZGVsUs4BBAFxcQsACWhtb2RlbC12MWtjb250cm9sbGVyc4F4OGRpZDprZXk6ejZNa21tSzlHVndLZkVtWDRtSkhjblFmaUQ3NlNTQ3k2Mk41NGJZWHNkRDVyTGVH"
            }
        ],
        "kjzl6hvfrbw6c6vwzcigxgsnskmtkwodxnxzy0fzasxs6apumpv67q9sip974ry": [
            {
                "jws": {
                    "payload": "AXESIOHbHVmGyD2Swa5Xkdx1zHr9rXv31KztltQxTWR9RRue",
                    "signatures": [
                        {
                            "signature": "K_qD3g7x4UnQGkc_ePptu-uG2ds1EAAD5Sk61e4Fo2Mo6E5kukQmsq-yrJ3fCog6ztDAxqGGOtQYhDCPA-06AA",
                            "protected": "eyJhbGciOiJFZERTQSIsImtpZCI6ImRpZDprZXk6ejZNa21tSzlHVndLZkVtWDRtSkhjblFmaUQ3NlNTQ3k2Mk41NGJZWHNkRDVyTGVHI3o2TWttbUs5R1Z3S2ZFbVg0bUpIY25RZmlENzZTU0N5NjJONTRiWVhzZEQ1ckxlRyJ9"
                        }
                    ],
                    "link": "bafyreihb3movtbwihwjmdlsxshohltd27wwxx56uvtwznvbrjvsh2ri3ty"
                },
                "linkedBlock": "omRkYXRhp2RuYW1lZ1Byb2ZpbGVldmlld3OhZ3ZlcnNpb26hZHR5cGVvZG9jdW1lbnRWZXJzaW9uZnNjaGVtYaZkdHlwZWZvYmplY3RlJGRlZnOiZldhbGxldKVkdHlwZWZvYmplY3RldGl0bGVmV2FsbGV0aHJlcXVpcmVkg2VjaGFpbmdhZGRyZXNzZ3ByaW1hcnlqcHJvcGVydGllc6NlY2hhaW6hZCRyZWZxIy8kZGVmcy9DaGFpblR5cGVnYWRkcmVzc6NkdHlwZWZzdHJpbmdpbWF4TGVuZ3RoGDxpbWluTGVuZ3RoFGdwcmltYXJ5oWR0eXBlZ2Jvb2xlYW50YWRkaXRpb25hbFByb3BlcnRpZXP0aUNoYWluVHlwZaNkZW51bYVjRVZNZlNPTEFOQWVBUFRPU2NTVUlkQVRPTWR0eXBlZnN0cmluZ2V0aXRsZWlDaGFpblR5cGVnJHNjaGVtYXgsaHR0cHM6Ly9qc29uLXNjaGVtYS5vcmcvZHJhZnQvMjAyMC0xMi9zY2hlbWFocmVxdWlyZWSBZG5hbWVqcHJvcGVydGllc6VjYmlvo2R0eXBlZnN0cmluZ2ltYXhMZW5ndGgYyGltaW5MZW5ndGgDZG5hbWWjZHR5cGVmc3RyaW5naW1heExlbmd0aBhkaW1pbkxlbmd0aANkdGFnc6NkdHlwZWVhcnJheWVpdGVtc6JkdHlwZWZzdHJpbmdpbWF4TGVuZ3RoGDxobWF4SXRlbXMUZmF2YXRhcqJkdHlwZWZzdHJpbmdpbWF4TGVuZ3RoGQPoZ3dhbGxldHOjZHR5cGVlYXJyYXllaXRlbXOhZCRyZWZuIy8kZGVmcy9XYWxsZXRobWF4SXRlbXMUdGFkZGl0aW9uYWxQcm9wZXJ0aWVz9Gd2ZXJzaW9uYzEuMGlyZWxhdGlvbnOga2Rlc2NyaXB0aW9ueBhwcm9maWxlIGZvciB1czNyLW5ldHdvcmtvYWNjb3VudFJlbGF0aW9uoWR0eXBlZnNpbmdsZWZoZWFkZXKjY3NlcGVtb2RlbGVtb2RlbFLOAQQBcXELAAlobW9kZWwtdjFrY29udHJvbGxlcnOBeDhkaWQ6a2V5Ono2TWttbUs5R1Z3S2ZFbVg0bUpIY25RZmlENzZTU0N5NjJONTRiWVhzZEQ1ckxlRw=="
            }
        ]
    },
    "indices": {},
    "aliases": {},
    "views": {
        "account": {},
        "root": {},
        "models": {}
    }
}

the right composite schema (saved while create model) is

{
    "version": "1.0",
    "models": {
        "kjzl6hvfrbw6c6vwzcigxgsnskmtkwodxnxzy0fzasxs6apumpv67q9sip974ry": [
            {
                "jws": {
                    "payload": "AXESIOHbHVmGyD2Swa5Xkdx1zHr9rXv31KztltQxTWR9RRue",
                    "signatures": [
                        {
                            "signature": "K_qD3g7x4UnQGkc_ePptu-uG2ds1EAAD5Sk61e4Fo2Mo6E5kukQmsq-yrJ3fCog6ztDAxqGGOtQYhDCPA-06AA",
                            "protected": "eyJhbGciOiJFZERTQSIsImtpZCI6ImRpZDprZXk6ejZNa21tSzlHVndLZkVtWDRtSkhjblFmaUQ3NlNTQ3k2Mk41NGJZWHNkRDVyTGVHI3o2TWttbUs5R1Z3S2ZFbVg0bUpIY25RZmlENzZTU0N5NjJONTRiWVhzZEQ1ckxlRyJ9"
                        }
                    ],
                    "link": "bafyreihb3movtbwihwjmdlsxshohltd27wwxx56uvtwznvbrjvsh2ri3ty"
                },
                "linkedBlock": "omRkYXRhp2RuYW1lZ1Byb2ZpbGVldmlld3OhZ3ZlcnNpb26hZHR5cGVvZG9jdW1lbnRWZXJzaW9uZnNjaGVtYaZkdHlwZWZvYmplY3RlJGRlZnOiZldhbGxldKVkdHlwZWZvYmplY3RldGl0bGVmV2FsbGV0aHJlcXVpcmVkg2VjaGFpbmdhZGRyZXNzZ3ByaW1hcnlqcHJvcGVydGllc6NlY2hhaW6hZCRyZWZxIy8kZGVmcy9DaGFpblR5cGVnYWRkcmVzc6NkdHlwZWZzdHJpbmdpbWF4TGVuZ3RoGDxpbWluTGVuZ3RoFGdwcmltYXJ5oWR0eXBlZ2Jvb2xlYW50YWRkaXRpb25hbFByb3BlcnRpZXP0aUNoYWluVHlwZaNkZW51bYVjRVZNZlNPTEFOQWVBUFRPU2NTVUlkQVRPTWR0eXBlZnN0cmluZ2V0aXRsZWlDaGFpblR5cGVnJHNjaGVtYXgsaHR0cHM6Ly9qc29uLXNjaGVtYS5vcmcvZHJhZnQvMjAyMC0xMi9zY2hlbWFocmVxdWlyZWSBZG5hbWVqcHJvcGVydGllc6VjYmlvo2R0eXBlZnN0cmluZ2ltYXhMZW5ndGgYyGltaW5MZW5ndGgDZG5hbWWjZHR5cGVmc3RyaW5naW1heExlbmd0aBhkaW1pbkxlbmd0aANkdGFnc6NkdHlwZWVhcnJheWVpdGVtc6JkdHlwZWZzdHJpbmdpbWF4TGVuZ3RoGDxobWF4SXRlbXMUZmF2YXRhcqJkdHlwZWZzdHJpbmdpbWF4TGVuZ3RoGQPoZ3dhbGxldHOjZHR5cGVlYXJyYXllaXRlbXOhZCRyZWZuIy8kZGVmcy9XYWxsZXRobWF4SXRlbXMUdGFkZGl0aW9uYWxQcm9wZXJ0aWVz9Gd2ZXJzaW9uYzEuMGlyZWxhdGlvbnOga2Rlc2NyaXB0aW9ueBhwcm9maWxlIGZvciB1czNyLW5ldHdvcmtvYWNjb3VudFJlbGF0aW9uoWR0eXBlZnNpbmdsZWZoZWFkZXKjY3NlcGVtb2RlbGVtb2RlbFLOAQQBcXELAAlobW9kZWwtdjFrY29udHJvbGxlcnOBeDhkaWQ6a2V5Ono2TWttbUs5R1Z3S2ZFbVg0bUpIY25RZmlENzZTU0N5NjJONTRiWVhzZEQ1ckxlRw=="
            }
        ],
        "kjzl6hvfrbw6c5sfh4zjss0pmc5x99sjdjbcu7zz8up5t7k0yvsybwvn2701o6k": [
            {
                "jws": {
                    "payload": "AXESIG6P7pKKzNQCveq2jabKCyt5biJLz_SiaaqZAG8iVDU6",
                    "signatures": [
                        {
                            "signature": "9pW3prqQ7_jjHT9mV8rvmOtsQxCf_6KUcH3NSAomc6Y48ENfzA1fetMmkYBem88HsG9EE4atTkIbylfG8C6dCw",
                            "protected": "eyJhbGciOiJFZERTQSIsImtpZCI6ImRpZDprZXk6ejZNa21tSzlHVndLZkVtWDRtSkhjblFmaUQ3NlNTQ3k2Mk41NGJZWHNkRDVyTGVHI3o2TWttbUs5R1Z3S2ZFbVg0bUpIY25RZmlENzZTU0N5NjJONTRiWVhzZEQ1ckxlRyJ9"
                        }
                    ],
                    "link": "bafyreidor7xjfcwm2qbl32vwrwtmuczlpfxces6p6srgtkuzabxsevbvhi"
                },
                "linkedBlock": "omRkYXRhp2RuYW1lZ0Jpb0xpbmtldmlld3OjZ2NyZWF0b3KhZHR5cGVvZG9jdW1lbnRBY2NvdW50Z3Byb2ZpbGWjZHR5cGVwcmVsYXRpb25Eb2N1bWVudGVtb2RlbHg/a2p6bDZodmZyYnc2YzZ2d3pjaWd4Z3Nuc2ttdGt3b2R4bnh6eTBmemFzeHM2YXB1bXB2NjdxOXNpcDk3NHJ5aHByb3BlcnR5aXByb2ZpbGVJRGd2ZXJzaW9uoWR0eXBlb2RvY3VtZW50VmVyc2lvbmZzY2hlbWGmZHR5cGVmb2JqZWN0ZSRkZWZzoW9DZXJhbWljU3RyZWFtSUSjZHR5cGVmc3RyaW5nZXRpdGxlb0NlcmFtaWNTdHJlYW1JRGltYXhMZW5ndGgYZGckc2NoZW1heCxodHRwczovL2pzb24tc2NoZW1hLm9yZy9kcmFmdC8yMDIwLTEyL3NjaGVtYWhyZXF1aXJlZINocGxhdGZvcm1maGFuZGxlaXByb2ZpbGVJRGpwcm9wZXJ0aWVzpWRkYXRhomR0eXBlZnN0cmluZ2ltYXhMZW5ndGgaAAGGoGZoYW5kbGWiZHR5cGVmc3RyaW5naW1heExlbmd0aBhkZ25ldHdvcmuiZHR5cGVmc3RyaW5naW1heExlbmd0aBhkaHBsYXRmb3JtomR0eXBlZnN0cmluZ2ltYXhMZW5ndGgYZGlwcm9maWxlSUShZCRyZWZ3Iy8kZGVmcy9DZXJhbWljU3RyZWFtSUR0YWRkaXRpb25hbFByb3BlcnRpZXP0Z3ZlcnNpb25jMS4waXJlbGF0aW9uc6FpcHJvZmlsZUlEomR0eXBlaGRvY3VtZW50ZW1vZGVseD9ranpsNmh2ZnJidzZjNnZ3emNpZ3hnc25za210a3dvZHhueHp5MGZ6YXN4czZhcHVtcHY2N3E5c2lwOTc0cnlrZGVzY3JpcHRpb254HXNvY2lhbCBsaW5rcyBmb3IgdXMzci1uZXR3b3Jrb2FjY291bnRSZWxhdGlvbqFkdHlwZWRsaXN0ZmhlYWRlcqNjc2VwZW1vZGVsZW1vZGVsUs4BBAFxcQsACWhtb2RlbC12MWtjb250cm9sbGVyc4F4OGRpZDprZXk6ejZNa21tSzlHVndLZkVtWDRtSkhjblFmaUQ3NlNTQ3k2Mk41NGJZWHNkRDVyTGVH"
            }
        ]
    },
    "indices": {
        "kjzl6hvfrbw6c6vwzcigxgsnskmtkwodxnxzy0fzasxs6apumpv67q9sip974ry": [],
        "BioLink": [
            {
                "fields": [
                    {
                        "path": [
                            "platform"
                        ]
                    }
                ]
            },
            {
                "fields": [
                    {
                        "path": [
                            "network"
                        ]
                    }
                ]
            },
            {
                "fields": [
                    {
                        "path": [
                            "handle"
                        ]
                    }
                ]
            }
        ]
    },
    "aliases": {},
    "views": {
        "account": {},
        "root": {},
        "models": {
            "kjzl6hvfrbw6c6vwzcigxgsnskmtkwodxnxzy0fzasxs6apumpv67q9sip974ry": {}
        }
    }
}

mohsin · October 14, 2023, 2:24pm

Hi @0xA25532B1287dEe6501,

Can you please confirm the dev tools package version?

0xA25532B1287dEe6501 · October 16, 2023, 1:12am

it is 0.5.0

mohsin · October 16, 2023, 2:07pm

OK, thank you.

Hey @paul, any idea what’s happening here? Is this a missing feature perhaps?

paul · October 16, 2023, 3:25pm

No it’s by design, indices information is stored in composites, not models.
When loading existing models, indices should be added explicitly to the composite.

mohsin · October 16, 2023, 3:27pm

Ah gotcha, ty!

0xA25532B1287dEe6501 · October 17, 2023, 2:02am

This design make sence!
BUT, if I wanted to reuse someone’s model, I do NOT have the composite json file, how could i use filter and order?
could @createIndex also be used with @loadModel?

dbcfd · October 18, 2023, 2:29pm

@createIndex can be used with @loadModel as in

type Ball @loadModel(id: "kjzl6hvfrbw6c8v02f9kocdsfvtz4cl60yjkvw0ir13ouqse879c0m17eelj9qy")
  @createIndex(fields: [{path: "red"}])
{
  id: ID!
}

However, a current limitation of the indexing is that this can only be done once per model per node. If you re-use the same model in multiple composites on the same node, they will need the same indices.

0xA25532B1287dEe6501 · October 19, 2023, 6:34am

I tried to use @createIndex with @loadModel on a model, which is not created on the node
then i get the following error:

Schema verification failed for index: kjzl6hvfrbw6c5k700w9ff9o7y243g2wifpx9bdmdrvnjteu3annxfd3kljtp6w. Please make sure latest migrations have been applied.

then i dig the source a little and found this

it check whether the field is indexed in mid table of the node’s pg.

so, @createIndex must be used on the same node the model is created, and @createIndex is applied to the same field while @createModel?
or if we startIndex of a model(NOT created on this node), and manually set the index field of mid table, then @createIndex would work?

I think, when using @createIndex with @loadModel, if the field was NOT indexed in the mid table, setup the index(as create model does), might be an better option?

dbcfd · October 23, 2023, 2:07pm

Looking into this more. The ideal user experience is that when you use loadModel, the MID table is created properly. Calling startIndex for the model might be a workaround (haven’t tested), but it shouldn’t be needed.

0xEE3CA4dd4CeB341691 · November 8, 2023, 5:15am

Ser any update on this?

0xEE3CA4dd4CeB341691 · November 8, 2023, 5:17am

Our use case is in data model explorer like https://s3.xyz/models/modelview/kjzl6hvfrbw6c5k700w9ff9o7y243g2wifpx9bdmdrvnjteu3annxfd3kljtp6w?network=TESTNET we could like to make it easier for other app developers to re-use existing models not created by themselves. This means to index models from new nodes.

0xEE3CA4dd4CeB341691 · November 8, 2023, 5:35am

As far as reuse is concerned, is it possible to reuse “composite” not just the models? It seems that the composite information is lost or have to be shared & indexed offchain, e.g. via github? @paul

dbcfd · November 8, 2023, 3:09pm

Yes, composites are not reusable, without sharing through something like github. They are particular to your application, and that is where indices are stored.

I have a possible fix for your issue, I am verifying it and determining possible impacts.

0xEE3CA4dd4CeB341691 · November 9, 2023, 3:58pm

Awesome. Looking forward to it!

0xEE3CA4dd4CeB341691 · November 15, 2023, 7:36am

Ser any update on this?

dbcfd · November 17, 2023, 4:17pm

So after some back and forth on one implementation, I think we’re going to handle this at the composedb layer, with a couple differences

composite:create will no longer index models, but you can pass the --deploy flag to do this
composite:deploy will index models, but you can pass the --no-index flag to prevent this
This will allow you to repeatedly create or deploy models, without adjusting indices as desired.

PR for this behavior: feat: expose flag to determine whether a composite is indexed by dbcfd · Pull Request #196 · ceramicstudio/js-composedb · GitHub

That PR is against the newer version of ComposeDB which works with Ceramic 3.X.

0xEE3CA4dd4CeB341691 · November 23, 2023, 4:57am

Does this PR support

type Ball @loadModel(id: "kjzl6hvfrbw6c8v02f9kocdsfvtz4cl60yjkvw0ir13ouqse879c0m17eelj9qy")
  @createIndex(fields: [{path: "red"}])
{
  id: ID!
}
```?

0xEE3CA4dd4CeB341691 · November 23, 2023, 5:13am

or is this more of an workaround for this limitation?

However, a current limitation of the indexing is that this can only be done once per model per node. If you re-use the same model in multiple composites on the same node, they will need the same indices.

dbcfd · November 27, 2023, 4:13pm

More of a workaround.

When first creating a model, you will run composite:create then composite:deploy. When updating a model, you will run composite:create, and optionally, composite:deploy --no-index.

If reusing a model multiple times with composite:deploy use the --no-index flag.