Proposal - IEKit - A modular toolkit for building impact evaluators

Grant/Project Name:

Impact Evaluator Toolkit (IEKit)

Proposer’s contact info:

The main point of contact on Valory’s side is Oaksprout. Reach out to discuss or clarify. Valory is the core builder in the Autonolas ecosystem. Autonolas has a tech stack for building decentralized and DAO-owned off-chain systems.

Grant Category:

Apps, Tooling

ELI5 Project Summary:

We propose to build and demonstrate a toolkit for developers in the Ceramic ecosystem to build early Impact Evaluators (IEs).

Although the scope is limited to a MVP, the proposed toolkit opens the door to permissionless, self-improving contribution evaluation systems which can be customized to the requirements of any organization.

This project will benefit Ceramic because it could highlight Ceramic’s central role in pushing forward the state of the art of IEs.

Project Description:

Introduction

As mentioned in Joel’s introduction and his talk on the topic, IEs are not new. In fact, one might argue that IEs’ objective functions - taking a measure of the world and distributing rewards accordingly - are crucial to the entire web3 project. Although instances of IEs are common, especially for measuring contribution in DAOs, most are relatively static and single purpose. Dedicated and programmable infrastructure for building them is yet to surface.

We aim to set the foundations upon which such infrastructure can be built. By combining the Autonolas stack with dynamic data streams on Ceramic, we plan to develop a simple impact evaluator, which tracks 1 type of contribution - @-mentions on Twitter. The scores for each contributing participant are stored on Ceramic.

This project will provide and demonstrate a way for Ceramic projects – and Web3 projects more broadly – to do community contribution tracking using datastreams. By developing the off-chain components of the IE with Autonolas, this toolkit lays the groundwork for the creation of an open, modular and self-improving framework for customizing how impact is evaluated in any decentralized organization.

Once delivered, our solution would provide the most advanced and extensible toolkit for pushing Impact Evaluators forward, cementing Ceramic’s role at the center of this promising technology. It also has potential to become a primitive for projects to drive data incentives, boosting utilization of the Ceramic network.

To begin with we describe main benefits to the Ceramic ecosystem. We then discuss the high level architecture of the MVP. Following this we go into a bit more detail with a technical specification. Then we explore how this product could be extended and grown far beyond the MVP. Finally, we offer more information about the Autonolas tech stack, the team and other projects we are working on.

Benefits to Ceramic

  • Cement Ceramic as the place to build IEs. This MVP represents a milestone on the path towards a full concrete IE with Ceramic at the heart of its implementation. The 3box and Valory teams can point to it as a beachhead with a clear path forward for the future of IEs.
  • Immediate, high Total Addressable Market. By tracking twitter mentions, even as an MVP the service is readily useful to all organizations with a twitter account.

High level architecture

We aim to:

  • Deliver a toolkit for building simple IEs
  • Demonstrate a live implementation of the toolkit as a decentralized, off-chain service powered by Autonolas

IEKit will be made up of a set of code, written in Autonolas’ open-source framework called Open Autonomy. Developers will be able to fork this and add new actions. From there they can run it as a single agent on centralized hosting, or immediately publish it to the Autonolas protocol and run it as a decentralized, off-chain service.

This project also aims to implement a canonical example of IEKit, to give a tangible demonstration of its capabilities. The demo aims to incentivize community members in the Autonolas community to tag the project on Twitter. This represents a simple incentivized action which we see as the core component of an Impact Evaluator.

The demo will run in rounds. Each round has 3 stages:

  1. data retrieval
  2. scoring
  3. writing

Data retrieval stage: the IE service retrieves all Autonolas @mentions from the Twitter API

Scoring stage: for every post that an individual account makes mentioning the Autonolas account, the IE service allocates 200 points. This happens in memory.

Writing stage: points are written to a Ceramic stream via the IE service’s DID.

Technical specification

We have designed a simple architecture for an IE to keep track of contributions to a community’s growth by mentioning the Autonolas handle in Twitter posts. Then, those contributions are assigned a score and the scores are written to a Ceramic stream.

Autonolas Service Specification

Core FSM

Autonomous services that use Autonolas technology are implemented as Finite State Machines where the logic is split across independent components: rounds define the rules to transition across different states and behavior, and implement the actual business logic. At the end of each round, agents agree on the round’s data output.

  • ReadData: agents pull data from the Twitter API. They retrieve all the posts which mention the Autonolas user account

  • Score: agents assign a score to each user according to how many posts have contributed.

  • WriteScores: the service writes the scores to a Ceramic stream. If the Safe DID is mature enough, this data write could be done in a decentralized way. All the agents can be set as owners of a Gnosis Safe contract and interact with the data stream through it. If Safe DID is not mature, agents can take turns randomly to write to the stream.

Figure 1: diagram of the core FSM

Keep in mind that the FSM diagram just accounts for the happy path and that error handling will be added once the complete technical requirements are known.

Chained FSM diagram

Pre-existing functionality, like agent registration and reset and pause rounds, will be chained to the core FSM (Figure 1) to form a complete FSM (Figure 2). The FinishedWriting round in the previous diagram represents a chaining point and will be removed during chaining.

Figure 2: diagram of the chained FSM

Roles and responsibilities

  • Service developer: implements the rounds and its business logic, API integrations, FSM chainings, chain integrations and any other capability needed for the service to run.
  • Agent: single instance of the service logic that runs the rounds.
  • Agent operator: runs one or more agent instances as specified by the service owner. Updates to new versions when required by the service.
  • Service owner: ultimately responsible for the service. Decides the service needs, the number of agents and operators and registers the service in the Autonolas protocol.

Other service details

  • Number of operators: depending on the level of decentralization we could increase or decrease this number. We recommend starting the service with a set of one operator running four agents and scale it up over time.

  • Number of API integrations: 1

  • Who runs the autonomous service: independent operators would run this service in a best-case scenario. Of course, at the beginning, it may need to be run internally. Once we have a tested MVP, we can seek independent operators.

  • Who pays for gas: gas fees, as well as infrastructure costs, are both part of the operational costs and service owners are responsible for covering them. In this case, Autonolas would be the service owner.

Extension

We aimed to fit the proposal into a relatively low budget. As a result we weren’t able to include several compelling features related to make a full-fledged Impact Evaluator:

Add Token Distribution Module

An immediate extension to the writing phase would be the pushing of an on chain transaction, which would distribute rewards directly to contributors. Depending on the organizations’ requirements one could airdrop the tokens or make them claimable via a Merkle tree.

Data mining incentives

Liquidity mining led to a step change in the levels of liquidity in DeFi, even after the bear market. Applying incentives to attract quality data could create an equally explosive outcome for the dataverse. This MVP could be extended to provide a technical architecture for incentivizing Ceramic actions.

Expand data input types

In future iterations of the service there is no limit to which types of contribution could be tracked. Data on members’ actions could be retrieved from on-chain events, or via any API. The contribution being tracked could even be the output of a Coordinape round and/or SourceCred round.

Expand scoring algorithms and Impact Evaluation

While the MVP simply counts contributions, it could be extended to evaluate them in more complex ways. It could, for example, employ Gitcoin Passport as a Sybil resistance check. Indeed, as the pool of actions being tracked grows because of the new data inputs added in the data retrieval stage, the methodologies by which actions are evaluated could become as arbitrarily complex as a machine learning or training model. These could optimize incentives to get the most value from the community. This bring us on one of the most compelling extensions.

Sophisticated Self-improvement

With the possibility of extending the MVP to autonomously distribute rewards according to any arbitrarily complex methodologies that has access to vast amounts of data inputs, including the data streams which power the impact evaluator itself, the stage is set for a self improving and self training impact evaluators to emerge. Even further beyond that, it would be possible for someone to create a more generalized, monolithic service which optimized over and aggregated all the IEs contributed to the framework.

An alternative way that the system could self improve is by outsourcing the training of the model to communities of token holders who could add and remove actions, and modify aggregation methodologies via gauge mechanisms. This would parallel the voting we already see on protocols like Curve. Although, rather than voting on LP rewards, these token holders vote on how and what gets rewarded. This system would therefore optimize impact evaluation via a market of incentivized actors.

Growth

To see maximum benefit to the Ceramic ecosystem, we have designed this MVP to be forked and composed in order to proliferate the number of IEs. Developers will be able to fork this code, develop new IEs and redeploy them via the Autonolas on-chain protocol.

Take a simple example. Say another DAO likes what they see from this running MVP, but feels that the current metrics are not relevant to their ecosystem. They would like to use different metrics and aggregate them to have more granular weighting for each metric. To get this going they would simply need to:

  • Fork the open-source code generated during this project
  • Swap out the existing APIs with new APIs
  • Make any necessary changes to the data aggregation algorithm
  • Register their new code on the Autonolas Protocol
  • Operators will come in to run their code

This process could be replicated to create a host of different IEs. As developers get more adventurous, they will be able to add increasingly rich data sources which can also be directly imported as components via the protocol. They will also be able to create more and more sophisticated aggregation algorithms which could even use things like predictive ML models.

We have constantly improving documentation on the entire stack. We also help developers in their learning journey with our academy. Finally, through the composability the stack offers, developers can reuse many existing code components.

About Autonolas

Autonolas is a stack for building and running sophisticated, decentralized off-chain services, called Autonomous Services.

Autonomous services on Autonolas are made of software agents. Agents are a bit like bots but unlike bots, they are social and can work together. With autonomous services, agents work together to deliver a service. There can be just a few agents in a service or many. Agents are open-source software.

A different individual or company runs each agent. This makes autonomous services decentralized – no single agent can control the service. It also makes them robust, and capable of managing large amounts of capital. If one agent goes down, the rest can continue.

Autonomous services run continuously. Agents can keep a service running indefinitely. This means services can spring into action when something happens off the blockchain. This is something smart contracts cannot do.

Services pull in data from any source a developer chooses. For example

  • an AWS storage bucket
  • a crypto-native data source like IPFS, Ceramic or The Graph,
  • or any other API.

Autonomous services can do sophisticated computation, even machine learning.

These services are reliable because agents must agree on what the service should do, via a Tendermint consensus engine.

Once the agents have come to consensus about what the service should do, they can take action. This action could be submitting a transaction on Ethereum, Avalanche, a Cosmos chain, Solana, etc. Services can also take action on a traditional API. This means that they are not limited to acting in the world of Web3. Autonomous services can be a bridge between the traditional tech world and crypto.

Autonomous services are transparent because the service is registered on-chain and the code is referenced via IPFS hashes. Each operator fetches the code by pointing to this service instance registered on-chain. Only if operators run compatible code will the application work. That means, as long as less than one third of operators are malicious or faulty, the integrity of the service is preserved. This limits the scope of misbehavior in v1. We are already exploring improvements to our stack which bring cryptographic guarantees to what happens inside an autonomous service.

Each autonomous service has a service owner who configures the parameters of the service, maintains it and decides on the incentives for operators to run it. It is worth reiterating for the current MVP, that because the service is run by multiple independent operators, it can perform all of its functionality in a way that is robust, trust-minimized and transparent.

The Autonolas Team

The team was founded and is led by:

  • David Minarsch (CEO) – Ph.D. University of Cambridge in Applied Game Theory. Led and authored first blockchain-focused multi-agent system framework at Fetch.ai.
  • David Galindo – Ph.D. Polytechnical University of Catalunya in Cryptography. Previously Head of Cryptography at Fetch.ai and Associate Professor of Cybersecurity at University of Birmingham.
  • Oaksprout the Tan – 10+ years experience in Web2 product roles. Held recognized community contribution roles at Balancer, THORChain and Ceramic.

The company behind Autonolas, Valory, raised a seed round last year, and is made up of 14 team members on top of the co-founders. Learn more.

Related Autonolas Projects

Dynamic Community Recognition NFTs

Dynamic Community Recognition NFTs update their image according to external data points. The Autonolas community is already testing this primitive with NFTs that update their image according to the points they have collected on the community leaderboard. This could integrate naturally with the impact evaluator to reflect the score of participating contributors in the images of community member NFTs.

SMPs

An SMP is a service that autonomously manages a Balancer Managed Pool according to predefined conditions. It can take in data from any API and aggregate it in any arbitrarily complex manner, including advanced statistics and machine learning. It then uses the aggregated data to make decisions on how to reallocate the positions in a given pool. Finally, it executes the on-chain transactions necessary to make the appropriate changes to the pool.
The primitive enable asset/treasury managers to automate portfolio management according to off and on- chain signals.

Valory received a grant from Balancer to build an MVP of this product.

The Autonolas Leaderboard

Despite not having a token, Autonolas has already built a basic IE. It is incentivizing contributions to its ecosystem via an off-chain point system. The system tracks and aggregates certain data inputs for each contributor. Seeing as each contributor holds a community NFT, their points can be reflected dynamically in the NFT via the ddNFT service.

Oracle service

Autonolas has an oracle service running in production on Polygon. As well as a price oracle, we also run an oracle which pulls in data about different Uniswap v2 contract DEXes, using a ML model which we developed to come up with APY predictions per pool. This further exemplifies running services, but is mostly relevant in that it illustrates the incredible complexity that an autonomous service can handle, and points towards the rich strategies developers will be able to build to attract users to impact evaluators on Ceramic.

The code behind these services would likely be heavily reused for this proposed project. The same code could be used to pull in data from APIs.

Read more about the technical architecture of this oracle service.

Other Services

El Collectooorr is an autonomous NFT collector. It watches for drops on Art Blocks, and then uses an algorithm to make a decision about how to take part in the minting auction. This also illustrates the usage of sophisticated, proactive, off-chain services to deliver novel asset management products. El Collectooorr was built by community developers, illustrating the ability for external developers to take our framework and extend it into novel use cases.

In addition to these services built by our team and by independent community developers, we have also attracted developers from various crypto project teams. Most notably here, teams such as Brahma are using the stack for robustly managing the off-chain requirements of their protocols. Other earlier stage teams have been trialling the stack for rich community incentivization and coordination, as well as novel asset management use cases. As Open Autonomy – our framework for developing autonomous services – and the first major part of our protocol are live, we expect many more services to be in production soon.

Relevant links:

  • Valory team website:https://www.valory.xyz/
  • Autonolas website: https://autonolas.network
  • Twitter: https://twitter.com/autonolas
  • Github: Valory · GitHub
  • Previous projects the team has meaningfully contributed to:
    • Fetch
    • Ceramic – Oaksprout won a Ceramic hackathon prize for Tiles – the browser for open, composable data. He is also on Ceramic’s community contribution team, Team Terracotta.
    • Balancer – Oaksprout received a grant to develop Baller, a heavily-used tool for calculating impermanent loss on Balancer’s sophisticated liquidity pools. He later joined Balancer’s community contribution team, the Ballers.
    • THORChain
  • Comparable projects or peers:
    • Fetch

Ceramic Ecosystem Value Proposition:

- What is the problem statement this proposal hopes to solve for the Ceramic ecosystem?

Despite the diversity of communities and organizations that compensate members for their contributions, impact evaluation is in its infancy. Solutions exist, but they are inflexible and only serve a subset of contributions types, leading to inefficient rewards distributions that do not necessarily push the needle towards the organization’s goals. Considering that there currently are over 2000 active DAOs and counting and Ceramic is an ideal way to track contribution there is a currently untapped opportunity here to significantly expand the Ceramic ecosystem.

- How does your proposal offer a value proposition solving the above problem?

By combining the Autonolas stack with Ceramic data this proposal offers a first glimpse of a modular and autonomous framework for customizing impact evaluators to the specific news of any organization. This granular composability is a key competitive advantage that could attract a large proportion of the market in the future.

- Why will this solution be a source of growth for the Ceramic ecosystem?

See Benefits to Ceramic above.

Funding requested (DAI/USDC): $15,000

Milestones:

  • Milestone #1: Development and testing of IE Kit (1 month) - $10,000 - 100 Hours @ 100 $/ Hr
    • Build the Core FSM as specified above
      • As we have an already existing component to build upon, the bulk of the work here will be to create the data pull and processing logic, integrate the required API and the logic to prepare the transactions.
    • Test coverage
    • External comms
  • Milestone #2: Demonstration by integrating with existing Autonolas contribution service and dynamic community recognition NFTs (1 month) - $5,000 - 50 Hours @ 100 $/ Hr
    • Component registration on Autonolas protocol
    • Progressive decentralization to multiple agents
    • Testnet operations
    • Mainnet deployment
    • Frontend support
    • External comms

I accept the 3Box Labs Grants terms and conditions: Yes
I understand that I will be required to provide additional KYC information to the 3Box Labs to receive this grant: Yes

1 Like

Hi @Oaksprout, thank you for your proposal! We will be in touch with an update once we have completed our initial review (1-2 weeks).

1 Like

Thanks! Do reach out if there’s anything I can help clarify.

Congratulations @Oaksprout, I’m delighted to inform you that your grant proposal has been accepted! :tada:

We would like to award you a Ceramic Builders Grant.

We will follow up shortly with more details via email.

1 Like

Brilliant! Thanks to you and Ceramic – looking forward to having this up and running to showcase a new primitive for evaluating impact and coordating DAOs.

1 Like

Grant Update

We have implemented our first FSM component (the one that reads mentions from the Twitter API) and its unit tests (100% coverage).

While starting the implementation of the other FSM component which writes to Ceramic, we hit a few stumbling blocks. The conversation is ongoing here. Understandably there are a few gaps in documentation, and all existing libraries are implemented in JS, whereas our agents are implemented in Python. We are now creating our own Python implementation that communicates with the HTTP-API, but this has been a bit of speed bump. Regardless, we are ahead of schedule.

In summary, our code is virtually complete but payload encoding and signing is not working yet. We’ve scheduled a call with the 3box DevRel team on Tuesday to dig in further, and @zfer has been very helpful.

Bonus note: we have a number of open conversations with teams interested in using Autonolas Contribute. Contribute will use the code developed during this grant, so anybody who implements it will inherit our Ceramic implementation out of the box.

Grant Update

Our focus the past couple of weeks has been on implementing a simple client in Python for creating and updating streams. It was pretty tricky! You can see some of the technical back and forth we had here:

Thanks again to @zfer for all the help and back and forth. And on our side, well done to @davidv for pushing through.

On top of that, we have also finished the second finite state machine (FSM) component and its unit tests. This means that almost all the code is now in place, the only remaining work is on chaining the FSM components together.

Due to this unforeseen issue with commit signing and the Python client, our completion plans have been pushed back by about two weeks. Fingers crossed we’ll still have some form of demo, even if only running in a centralised way, to show the community at ETHDenver!

2 Likes

Grant Update

We have now finished chaining the FSM components, so all the code is in place.

We have also implemented the e2e tests. This means that we are able to run the service in a test environment where the Ceramic and Twitter APIs are mocked. The next step is to run the service on a real environment.

From next Monday, Twitter API access will be significantly restricted so we are evaluating how to proceed on that front. Our current plan is to simply purchase a basic plan, but this does limit external developers’ ability to simply deploy the toolkit and have it launch. We’ll need to consider where to go from here.

Grant Update – Milestone #1 Completed

We’re happy to announce that we’ve completed the first of our 2 milestones during these past two weeks.

Milestone #1 revolved around creating the code that coordinates and rewards contributors in a DAO. That’s now complete and the code behind IEKit can be found in this repo:

Please see the following video demo’ing it:

https://drive.google.com/file/d/1MQH6yBav7iXcDkFwyE3_CGN3wY8jiRf-/view?usp=sharing

The demo video is divided into 2 main parts:

  1. Technical description of the parts and flows of the system

  2. A walk through of logs showing the output and showing evidence of the working system

Completion of Milestone #1 involved a surprisingly heavy lift in terms of building out a Python client, so we’re really happy to see the fundamentals working in terms of the Autonolas tech.

The emphasis of Milestone #2 is really to demonstrate to the Ceramic and Impact Evaluator communities what can be done here. We’ve already started work on that and are excited to show it as soon as it’s ready.

1 Like

Love it. Great work on Milestone #1.

Tagging in DevRel on the demo @Justina

Update

Great meeting you all in person at ETH Denver.

Short update on our side this week – we’re putting the finishing touches on milestone #2 to finalise the backend, get a frontend in place, prepare a demo and put together comms to help the Impact Evaluator community and Ceramic ecosystem understand what we’ve built here. Launch is imminent!

IEKit is available, and the demo is live!

We’re happy to announce that we’ve now completed both milestones, and IEKit has been launched.

image

Check out the landing page.

Demo

image

Video walkthrough:

image

I’ll keep it short and sweet here because there’s a fair bit of content out there about IEKit now:

Looking forward to seeing what the Impact Evaluator community builds with this.

p.s. for now, the content we’ve put out here is shaped for people interested in Impact Evaluators. But with time we’d also like to share more about how this work benefits Ceramic’s developers and ecosystem, e.g. in repurposing the code for generalised, DAO-owned co-processors for Ceramic, and the work we did on a Python client

1 Like

Amazing! Love to see IEKit going live.

Congratulations on completing the final milestone of this grant. The team will do a quick review and will let you know any questions or comments.

cc @Justina for visibility on that awesome IEKit demo!

1 Like

Fantastic. Let us know if there’s anything we can clarify.

As mentioned, we’ll also be following up with some materials about how what we’ve built here is more broadly applicable to Ceramic devs, beyond the Impact Evaluator use case.

Do reach out @Justina - would be great to sync up on how to get the word out.

Looks great!
Do you have a link pointing to the composite being used please?

Also, it seems you made a Python implementation of the Ceramic client as part of this project, do you have any plan to release it as a standalone project other members of the community could use please?
Thanks!

1 Like

We didn’t actually use a composite in this case. Would it have been beneficial to?

As a follow up to this project, we would like to survey the community to gauge interest in the Python client. Currently it satisfies the basic requirements around stream updates, but we’re keen to help develop a client if there’s demand from the community.

It’s not a strict requirement to use ComposeDB but as it is our current focus, we expect most of the community will build with it going forwards, so the more projects creating reusable composites the better.

Sounds great, thanks!

1 Like