I’m trying to load the history of changes for a given stream. However, I only seem to get the same data back for every commit loaded from allCommitIds. I’ve tried loading directly from the log but TileLoader throws errors so I guess those CID’s aren’t what I’m looking for.
How can I retrieve all prior commits for a stream? This is how it’s implemented currently:
export async function getHistory(address: string, { ceramic, dataModel, dataStore }: CeramicCommon) : Promise<TileDocument<JWE>[]|null>{
const link = await getLink(address, ceramic);
const did = link.did; //await core.getAccountDID(link.id);
if (!did) {
log.debug("No DID found for account");
return null;
};
log.trace(`Loading history for ${did}`);
const definitionId = dataModel.getDefinitionID("AccountDetails");
// The recordId is the stream defined by definition for this DID
const streamId = await dataStore.getRecordID(definitionId!, did);
if (!streamId)
return null;
const stream = await TileDocument.load(ceramic, streamId);
const ids = stream.allCommitIds;
// const ids = stream.state.log
// .map(l => new CommitID(l.type, l.cid))
// .filter(c => c.type === TILE_TYPE_ID);
log.trace(`Found ${ids.length} commits`);
return Promise.all(ids.map(id => TileDocument.load<JWE>(ceramic, id)));
}
I’ve also tried using ceramic.loadStreamCommits and ceramic.loadStream(id) without getting where I need to go.
log.trace(`Found ${ids.length} commits`);
const commits = await Promise.all(ids.map(id => ceramic.loadStream(id)));
return commits.map(c => c.content as TileDocument<JWE>);
The code in your main code block looks correct as best as I can tell. What are you seeing when you print the content of the documents returned?
What are you seeing when you print the content of the documents returned?
Every document is identical. I’ve knocked up a stand-alone script here that you can see for yourself:
import { Core } from '@self.id/core'
import { TileDocument } from '@ceramicnetwork/stream-tile'
import type { JWE } from 'did-jwt';
const did = "did:3:kjzl6cwe1jw146vyxh3bf1l325qcb3b4rxnns181whfw1mezrpmgyia2mm2otzm";
const aliases = JSON.parse(`{
"definitions": {
"AccountDetails": "kjzl6cwe1jw149gym1m5qjzzqjdlupnsxivgeef9z6xjkc14ldbkgwag1cssryr"
},
"schemas": {
"jwe": "ceramic://k3y52l7qbv1frxwvp66vqpcd046ju9ocvimye6htwucpg00h08nltj1fkns6p9ji8"
},
"tiles": {}
}`)
const core = new Core({ ceramic: 'testnet-clay', aliases});
export async function getOtherHistory() : Promise<JWE[]|null>{
console.log(`Loading history for ${did}`);
const { ceramic, dataModel, dataStore } = core;
const definitionId = dataModel.getDefinitionID("AccountDetails");
// The recordId is the stream defined by definition for this DID
const streamId = await dataStore.getRecordID(definitionId!, did);
if (!streamId)
return null;
const stream = await TileDocument.load(ceramic, streamId);
const ids = stream.allCommitIds;
console.log("All Ids:\n", ids.map(id => id.toString()).join('\n'));
console.log(`Found ${ids.length} commits`);
// TODO: do this with a multi-query instead
const commits = await Promise.all(ids.map(id => ceramic.loadStream(id)));
return commits.map(c => c.content as JWE);
}
const all = await getOtherHistory();
if (all) {
for (const commit of all) {
console.log(commit.ciphertext);
}
}
You should be able to simply run node --loader ts-node/esm testScript.ts and you’ll see the following output:
Loading history for did:3:kjzl6cwe1jw146vyxh3bf1l325qcb3b4rxnns181whfw1mezrpmgyia2mm2otzm
All Ids:
kjzl6cwd29gy4fzehcbt0sac31b0umlq6ofdfpc3ztfrked5an5cb31pw22hu68
k6zn3rc3iwgugav5uw7ftgbmsfggipk4996nvsfd02p8t2z4pitqlszaioefk5wkbwyl43yexij4v95hapel1qy6l28z69cbr0peof5ih0odq4b93oe4op2
k6zn3rc3iwgugav5uw7ftgbmsfggipk4996nvsfd02p8t2z4pitqlszaioefk5wkbwyl44c1sg11sbkkroon11ycbtm1qyyp4ir7n3yvxrm8x5741dv70ee
kzdxps4zufva6109enjntuvw4f92mpw3juqzeonrkmoau8bcl7k1xvyo02129tvyyo44893qc7y77amt29guf570a8xwge4a47grg7wg6j6p4psf7msz1
k6zn3rc3iwgugav5uw7ftgbmsfggipk4996nvsfd02p8t2z4pitqlszaioefk5wkbwyl41y3vd7e21x25rur36mlxrj27ugnmv99t5ycb06oyyma1ka0aet
kzdxps4zufva6109enjntuvw4f92mpw3juqzeonrkmoau8bcl7k1xvyo02129tvyyo461qdpfhszeyhtw5t27hh8amkqr7z9qjqs6wvf6d5o3br1u29ur
k6zn3rc3iwgugav5uw7ftgbmsfggipk4996nvsfd02p8t2z4pitqlszaioefk5wkbwyl45gv7j09xb9r3o2k5eyk4pginewa1p4g5u9iui2bih9hzxodtl0
k6zn3rc3iwgugav5uw7ftgbmsfggipk4996nvsfd02p8t2z4pitqlszaioefk5wkbwyl42ia4jcyc840ed3jn153cnz3qcmw3ocezs9xlsmah119djrfqpn
kzdxps4zufva6109enjntuvw4f92mpw3juqzeonrkmoau8bcl7k1xvyo02129tvyyo43fkzq32v00axb8ovmoxpnmgo317g6t0knpufih7090me9p3hr8
k6zn3rc3iwgugav5uw7ftgbmsfggipk4996nvsfd02p8t2z4pitqlszaioefk5wkbwyl44v7uw4v5lw2ixi05tjpvi2uomxyy64yl1mp7gzd05opnf7rv83
k6zn3rc3iwgugav5uw7ftgbmsfggipk4996nvsfd02p8t2z4pitqlszaioefk5wkbwyl43evr2l5txazgmc1ej0lopjlomoh08qnri482ma1ryvppkobu11
k6zn3rc3iwgugav5uw7ftgbmsfggipk4996nvsfd02p8t2z4pitqlszaioefk5wkbwyl40wjt2eutparomr7x5v7gz81ckpnsv5cp3xsx1thmxb8ioyoxl7
k6zn3rc3iwgugav5uw7ftgbmsfggipk4996nvsfd02p8t2z4pitqlszaioefk5wkbwyl3zng97ksuo1vs1c5w8h3y83mp0m4lblgbtyhoacetggfyfk76mr
kzdxps4zufva6109enjntuvw4f92mpw3juqzeonrkmoau8bcl7k1xvyo02129tvyyo44pj9jhjtjsrlpbkfxgrpkfqj7ijrmh5pv3jmr1da418dj0nwib
k6zn3rc3iwgugav5uw7ftgbmsfggipk4996nvsfd02p8t2z4pitqlszaioefk5wkbwyl41w4fbg4j279dglyhj313aex7dcftv5p873qsmo603wfeqyk2f5
k6zn3rc3iwgugav5uw7ftgbmsfggipk4996nvsfd02p8t2z4pitqlszaioefk5wkbwyl454srgbdsy6uwl11paqkk23gl5o4ntfkydmug6kn1dcuidzunhi
kzdxps4zufva6109enjntuvw4f92mpw3juqzeonrkmoau8bcl7k1xvyo02129tvyyo49jhzevfzrtr4qx7bkp4tkl2x4ug0c5ya2w6mpcdsqhdjxm66sz
Found 17 commits
UM_yM0hWibBNri2oAqB4aaB61MLCyn3ZNTOR8HkR5mD48Lbsh_xSLl0-vydjgAv1iAOvGcQ5j1wauAWB7EI0GkOco5oRLy9m7bGMW0FbJnDhMQlwBtSxTI3izqUFeJxj-y2gcW6IwwZYIhxxZDjwGMNXgp7zZWE7IB8So0AcxkaCv2R33fmzXMmfU3nfzvdfLuj8_Gf7ZuSyR3fwMF0QOlyBW10e147ZjG67QI1gwUqvT9Jyn_uxyZE0wNT_bRwQIetGt_NhHphS_2LAPx8ZQWJ_MoNXbs2K
UM_yM0hWibBNri2oAqB4aaB61MLCyn3ZNTOR8HkR5mD48Lbsh_xSLl0-vydjgAv1iAOvGcQ5j1wauAWB7EI0GkOco5oRLy9m7bGMW0FbJnDhMQlwBtSxTI3izqUFeJxj-y2gcW6IwwZYIhxxZDjwGMNXgp7zZWE7IB8So0AcxkaCv2R33fmzXMmfU3nfzvdfLuj8_Gf7ZuSyR3fwMF0QOlyBW10e147ZjG67QI1gwUqvT9Jyn_uxyZE0wNT_bRwQIetGt_NhHphS_2LAPx8ZQWJ_MoNXbs2K
UM_yM0hWibBNri2oAqB4aaB61MLCyn3ZNTOR8HkR5mD48Lbsh_xSLl0-vydjgAv1iAOvGcQ5j1wauAWB7EI0GkOco5oRLy9m7bGMW0FbJnDhMQlwBtSxTI3izqUFeJxj-y2gcW6IwwZYIhxxZDjwGMNXgp7zZWE7IB8So0AcxkaCv2R33fmzXMmfU3nfzvdfLuj8_Gf7ZuSyR3fwMF0QOlyBW10e147ZjG67QI1gwUqvT9Jyn_uxyZE0wNT_bRwQIetGt_NhHphS_2LAPx8ZQWJ_MoNXbs2K
UM_yM0hWibBNri2oAqB4aaB61MLCyn3ZNTOR8HkR5mD48Lbsh_xSLl0-vydjgAv1iAOvGcQ5j1wauAWB7EI0GkOco5oRLy9m7bGMW0FbJnDhMQlwBtSxTI3izqUFeJxj-y2gcW6IwwZYIhxxZDjwGMNXgp7zZWE7IB8So0AcxkaCv2R33fmzXMmfU3nfzvdfLuj8_Gf7ZuSyR3fwMF0QOlyBW10e147ZjG67QI1gwUqvT9Jyn_uxyZE0wNT_bRwQIetGt_NhHphS_2LAPx8ZQWJ_MoNXbs2K
UM_yM0hWibBNri2oAqB4aaB61MLCyn3ZNTOR8HkR5mD48Lbsh_xSLl0-vydjgAv1iAOvGcQ5j1wauAWB7EI0GkOco5oRLy9m7bGMW0FbJnDhMQlwBtSxTI3izqUFeJxj-y2gcW6IwwZYIhxxZDjwGMNXgp7zZWE7IB8So0AcxkaCv2R33fmzXMmfU3nfzvdfLuj8_Gf7ZuSyR3fwMF0QOlyBW10e147ZjG67QI1gwUqvT9Jyn_uxyZE0wNT_bRwQIetGt_NhHphS_2LAPx8ZQWJ_MoNXbs2K
UM_yM0hWibBNri2oAqB4aaB61MLCyn3ZNTOR8HkR5mD48Lbsh_xSLl0-vydjgAv1iAOvGcQ5j1wauAWB7EI0GkOco5oRLy9m7bGMW0FbJnDhMQlwBtSxTI3izqUFeJxj-y2gcW6IwwZYIhxxZDjwGMNXgp7zZWE7IB8So0AcxkaCv2R33fmzXMmfU3nfzvdfLuj8_Gf7ZuSyR3fwMF0QOlyBW10e147ZjG67QI1gwUqvT9Jyn_uxyZE0wNT_bRwQIetGt_NhHphS_2LAPx8ZQWJ_MoNXbs2K
UM_yM0hWibBNri2oAqB4aaB61MLCyn3ZNTOR8HkR5mD48Lbsh_xSLl0-vydjgAv1iAOvGcQ5j1wauAWB7EI0GkOco5oRLy9m7bGMW0FbJnDhMQlwBtSxTI3izqUFeJxj-y2gcW6IwwZYIhxxZDjwGMNXgp7zZWE7IB8So0AcxkaCv2R33fmzXMmfU3nfzvdfLuj8_Gf7ZuSyR3fwMF0QOlyBW10e147ZjG67QI1gwUqvT9Jyn_uxyZE0wNT_bRwQIetGt_NhHphS_2LAPx8ZQWJ_MoNXbs2K
UM_yM0hWibBNri2oAqB4aaB61MLCyn3ZNTOR8HkR5mD48Lbsh_xSLl0-vydjgAv1iAOvGcQ5j1wauAWB7EI0GkOco5oRLy9m7bGMW0FbJnDhMQlwBtSxTI3izqUFeJxj-y2gcW6IwwZYIhxxZDjwGMNXgp7zZWE7IB8So0AcxkaCv2R33fmzXMmfU3nfzvdfLuj8_Gf7ZuSyR3fwMF0QOlyBW10e147ZjG67QI1gwUqvT9Jyn_uxyZE0wNT_bRwQIetGt_NhHphS_2LAPx8ZQWJ_MoNXbs2K
UM_yM0hWibBNri2oAqB4aaB61MLCyn3ZNTOR8HkR5mD48Lbsh_xSLl0-vydjgAv1iAOvGcQ5j1wauAWB7EI0GkOco5oRLy9m7bGMW0FbJnDhMQlwBtSxTI3izqUFeJxj-y2gcW6IwwZYIhxxZDjwGMNXgp7zZWE7IB8So0AcxkaCv2R33fmzXMmfU3nfzvdfLuj8_Gf7ZuSyR3fwMF0QOlyBW10e147ZjG67QI1gwUqvT9Jyn_uxyZE0wNT_bRwQIetGt_NhHphS_2LAPx8ZQWJ_MoNXbs2K
UM_yM0hWibBNri2oAqB4aaB61MLCyn3ZNTOR8HkR5mD48Lbsh_xSLl0-vydjgAv1iAOvGcQ5j1wauAWB7EI0GkOco5oRLy9m7bGMW0FbJnDhMQlwBtSxTI3izqUFeJxj-y2gcW6IwwZYIhxxZDjwGMNXgp7zZWE7IB8So0AcxkaCv2R33fmzXMmfU3nfzvdfLuj8_Gf7ZuSyR3fwMF0QOlyBW10e147ZjG67QI1gwUqvT9Jyn_uxyZE0wNT_bRwQIetGt_NhHphS_2LAPx8ZQWJ_MoNXbs2K
UM_yM0hWibBNri2oAqB4aaB61MLCyn3ZNTOR8HkR5mD48Lbsh_xSLl0-vydjgAv1iAOvGcQ5j1wauAWB7EI0GkOco5oRLy9m7bGMW0FbJnDhMQlwBtSxTI3izqUFeJxj-y2gcW6IwwZYIhxxZDjwGMNXgp7zZWE7IB8So0AcxkaCv2R33fmzXMmfU3nfzvdfLuj8_Gf7ZuSyR3fwMF0QOlyBW10e147ZjG67QI1gwUqvT9Jyn_uxyZE0wNT_bRwQIetGt_NhHphS_2LAPx8ZQWJ_MoNXbs2K
UM_yM0hWibBNri2oAqB4aaB61MLCyn3ZNTOR8HkR5mD48Lbsh_xSLl0-vydjgAv1iAOvGcQ5j1wauAWB7EI0GkOco5oRLy9m7bGMW0FbJnDhMQlwBtSxTI3izqUFeJxj-y2gcW6IwwZYIhxxZDjwGMNXgp7zZWE7IB8So0AcxkaCv2R33fmzXMmfU3nfzvdfLuj8_Gf7ZuSyR3fwMF0QOlyBW10e147ZjG67QI1gwUqvT9Jyn_uxyZE0wNT_bRwQIetGt_NhHphS_2LAPx8ZQWJ_MoNXbs2K
UM_yM0hWibBNri2oAqB4aaB61MLCyn3ZNTOR8HkR5mD48Lbsh_xSLl0-vydjgAv1iAOvGcQ5j1wauAWB7EI0GkOco5oRLy9m7bGMW0FbJnDhMQlwBtSxTI3izqUFeJxj-y2gcW6IwwZYIhxxZDjwGMNXgp7zZWE7IB8So0AcxkaCv2R33fmzXMmfU3nfzvdfLuj8_Gf7ZuSyR3fwMF0QOlyBW10e147ZjG67QI1gwUqvT9Jyn_uxyZE0wNT_bRwQIetGt_NhHphS_2LAPx8ZQWJ_MoNXbs2K
UM_yM0hWibBNri2oAqB4aaB61MLCyn3ZNTOR8HkR5mD48Lbsh_xSLl0-vydjgAv1iAOvGcQ5j1wauAWB7EI0GkOco5oRLy9m7bGMW0FbJnDhMQlwBtSxTI3izqUFeJxj-y2gcW6IwwZYIhxxZDjwGMNXgp7zZWE7IB8So0AcxkaCv2R33fmzXMmfU3nfzvdfLuj8_Gf7ZuSyR3fwMF0QOlyBW10e147ZjG67QI1gwUqvT9Jyn_uxyZE0wNT_bRwQIetGt_NhHphS_2LAPx8ZQWJ_MoNXbs2K
UM_yM0hWibBNri2oAqB4aaB61MLCyn3ZNTOR8HkR5mD48Lbsh_xSLl0-vydjgAv1iAOvGcQ5j1wauAWB7EI0GkOco5oRLy9m7bGMW0FbJnDhMQlwBtSxTI3izqUFeJxj-y2gcW6IwwZYIhxxZDjwGMNXgp7zZWE7IB8So0AcxkaCv2R33fmzXMmfU3nfzvdfLuj8_Gf7ZuSyR3fwMF0QOlyBW10e147ZjG67QI1gwUqvT9Jyn_uxyZE0wNT_bRwQIetGt_NhHphS_2LAPx8ZQWJ_MoNXbs2K
UM_yM0hWibBNri2oAqB4aaB61MLCyn3ZNTOR8HkR5mD48Lbsh_xSLl0-vydjgAv1iAOvGcQ5j1wauAWB7EI0GkOco5oRLy9m7bGMW0FbJnDhMQlwBtSxTI3izqUFeJxj-y2gcW6IwwZYIhxxZDjwGMNXgp7zZWE7IB8So0AcxkaCv2R33fmzXMmfU3nfzvdfLuj8_Gf7ZuSyR3fwMF0QOlyBW10e147ZjG67QI1gwUqvT9Jyn_uxyZE0wNT_bRwQIetGt_NhHphS_2LAPx8ZQWJ_MoNXbs2K
UM_yM0hWibBNri2oAqB4aaB61MLCyn3ZNTOR8HkR5mD48Lbsh_xSLl0-vydjgAv1iAOvGcQ5j1wauAWB7EI0GkOco5oRLy9m7bGMW0FbJnDhMQlwBtSxTI3izqUFeJxj-y2gcW6IwwZYIhxxZDjwGMNXgp7zZWE7IB8So0AcxkaCv2R33fmzXMmfU3nfzvdfLuj8_Gf7ZuSyR3fwMF0QOlyBW10e147ZjG67QI1gwUqvT9Jyn_uxyZE0wNT_bRwQIetGt_NhHphS_2LAPx8ZQWJ_MoNXbs2K
The commits are encrypted, but you can see that the cyphertext is the same for every commit, so the decrypted document is the same
Dammit, 15 mins after posting that I figured it out…
The issue is the simultaneous calls to loadStream
const commits = await Promise.all(ids.map(id => ceramic.loadStream(id)));
Internally, streams are cached and the call to loadStream syncs stream to the given ID. Doing them all at once means that once complete, the single cache is still sync’ed on a single commit, and we get back X documents all referencing that single (tip) commit.
Given that it takes kinda a long time to query a document, it might pay to make this cache either re-entrant, or put a simple guard & warn if trying to update it while someone else has the ‘lock’.
So… last question: It takes a long time to load a stream with more than a few commits, and I definitely expect that to happen fairly soon. it’s recommended to use ceramic.multiquery to batch requests, is there any docs on constructing manual queries?
Glad you were able to figure it out! Please feel free to file issues on github (or better yet open a PR!) for any improvements that you think should be made to the product to make this clearer.
Please note that loading a stream at a specific historical CommitID is a much less optimized codepath than loading the current state of the stream, so I’d definitely expect some performance problems if that is something you will be doing often.
The docs for multiQuery are here: CeramicClient | Ceramic Typescript Implementation (and the MultiQuery type it depends on is here: MultiQuery | Ceramic Typescript Implementation)