New CAS anchor-retry logic problem

[zkTRUTH]

Last night I upgraded our ceramic node to the latest release as per the recommendation to @ceramicnetwork/cli@2.8.0. Today we are having problems with the CAS:

[2022-10-07T12:14:23.631Z] ERROR: Error: Error connecting to CAS while attempting to anchor k2t6wyfsu4pfwyk74vomr9xqlg5bb3gnkvbrej275bo3te00pg4zusf6adqa72 at commit bafyreiak6g2wgaexlyn4m46widt4u3il4qrae6vxg7m3dnhmgpb743tldy: HTTP request to 'https://cas.3boxlabs.com/api/v0/requests' failed with status 'Bad Request': {"error":"Creating request with streamId k2t6wyfsu4pfwyk74vomr9xqlg5bb3gnkvbrej275bo3te00pg4zusf6adqa72 and commit CID bafyreiak6g2wgaexlyn4m46widt4u3il4qrae6vxg7m3dnhmgpb743tldy failed: Cannot read properties of undefined (reading 'cid')"}

Obviously our requestAnchor call has a streamID and a cid or it wouldn’t be able to log them.

The error points directly at the new retry logic that was added in the most recent release: feat: Ceramic asks CAS to anchor indefinitely until some ok response … · ceramicnetwork/js-ceramic@18150a9 · GitHub

[mohsin]

Hi @zkTRUTH, we added more logging around this on Friday and figured out what’s happening.

Seems like issues with loading at specific commits strikes again As Spencer mentioned a while back, this is one of our more unoptimized flows since it’s not very frequently used, though I understand how/why Cambrian uses it. Sorry you’ve had to be the one to discover these latent issues

Here are @StephH’s notes about what’s happening:

Each anchor commit is associated to a request.

For a stream, if i make commit A then commit B than anchor, commit B will be the one that is anchored. The anchor commit is then associated with commit B. 

When we try to get the data of commit A, the CAS will see that the request for commit A is COMPLETED. It will then look for the associated anchor commit. But this anchor commit does not exist because the one created is only linked to commit B (not commit A).

We’re discussing how to fix this, and will keep you posted.