TerraForm Issues: "getaddrinfo ENOTFOUND go-ipfs-ceramic-prod-4-1-internal.XXX.com"

Hello,

I am having issues facilitating the communication between the IPFS and Ceramic instances with the terraform project.

Issue
Running terraform apply -var-file="vars.tfvars" “successfully” deploys the infrastructure to AWS. VPC configured as mentioned in this thread. IPFS container deploys and reaches stable state in ECS. However, the Ceramic ECS container endlessly fails on startup with the following error:

[2022-09-26T18:54:06.562Z] WARNING: 'Error loading our PeerID from IPFS: FetchError: request to https://go-ipfs-ceramic-prod-4-1-internal.example.com:5011/api/v0/id failed, reason: getaddrinfo ENOTFOUND go-ipfs-ceramic-prod-4-1-internal.example.com. Skipping connection to bootstrap peers'
/js-ceramic/node_modules/rxjs/dist/cjs/internal/util/reportUnhandledError.js:13
            throw err;
            ^
FetchError: request to https://go-ipfs-ceramic-prod-4-1-internal.example.com:5011/api/v0/id failed, reason: getaddrinfo ENOTFOUND go-ipfs-ceramic-prod-4-1-internal.example.com
    at ClientRequest.<anonymous> (/js-ceramic/node_modules/node-fetch/lib/index.js:1461:11)
    at ClientRequest.emit (node:events:513:28)
    at ClientRequest.emit (node:domain:489:12)
    at TLSSocket.socketErrorListener (node:_http_client:481:9)
    at TLSSocket.emit (node:events:513:28)
    at TLSSocket.emit (node:domain:489:12)
    at emitErrorNT (node:internal/streams/destroy:157:8)
    at emitErrorCloseNT (node:internal/streams/destroy:122:3)
    at processTicksAndRejections (node:internal/process/task_queues:83:21) {
  type: 'system',
  errno: 'ENOTFOUND',
  code: 'ENOTFOUND'
}

The VPC seems properly setup by AWS automatically, ACM stores the SSL certificate for example.com, and Route53 both owns/ hosts example.com and contains the ACM entry for it. Any assistance greatly appreciated!

Huh, is go-ipfs-ceramic-prod-4-1-internal, specifically, set up correctly in Route53? ENOTFOUND means that DNS lookup could not find this sub-domain.

You’ll need to have a CNAME entry in Route53 that maps go-ipfs-ceramic-prod-4-1-internal.example.com to the IPFS ALB URL.

I did have a public dns hosted zone with a record for for go-ipfs-ceramic-prod-4-1-internal.example.com. Spent a chunk of time researching the issue given your response. I was able to resolve this specific issue by setting up a Private DNS hosted zone for the VPC and adding the record there. You have to deploy with terraform, let the ceramic container initially reach a looping failed state where it can’t find the ipfs container, then you can associate the DNS.

I am going to write out the full steps I used (see this thread on VPC config if you need prior context):

  • Create a public hosted zone ceramic.example.com (must own domain on route53)
    • assuming you have a separate hosted zone for example.com, add a NS record to the example.com hosted zone with the value being the nameservers found in the ceramic.example.com hosted zone
    • Create ACM certificate for ceramic.example.com (I had *.ceramic.example.com as secondary domain on SSL certificate but not sure if it matters) and add it to the ceramic.example.com hosted domain in route53 (button press)
  • terraform apply
  • Create private hosted zone ceramic.example.com
    • Create A record go-ipfs-ceramic-prod-4-1-internal.ceramic.example.com (numbers may or may not change for you), pick alias > application/ classic load balancer > dualstack.internal-ceramic-prod-4-1-ipfs-nd-in-XXXXXXXX.us-east-1.elb.amazonaws.com.

There are still some issues so I don’t bother with adding the routes for the public ceramic.example.com hosted zones but it is the same process pointed to the public ipfs and ceramic ALB’s I assume. If it is not there will be another thread :mask:

1 Like

This is very helpful!! Thanks for taking the time to write out the list of steps! :pray:t4:

1 Like