Grant/Project Name:
Impact Evaluator Toolkit (IEKit)
Proposer’s contact info:
The main point of contact on Valory’s side is Oaksprout. Reach out to discuss or clarify. Valory is the core builder in the Autonolas ecosystem. Autonolas has a tech stack for building decentralized and DAO-owned off-chain systems.
Grant Category:
Apps, Tooling
ELI5 Project Summary:
We propose to build and demonstrate a toolkit for developers in the Ceramic ecosystem to build early Impact Evaluators (IEs).
Although the scope is limited to a MVP, the proposed toolkit opens the door to permissionless, self-improving contribution evaluation systems which can be customized to the requirements of any organization.
This project will benefit Ceramic because it could highlight Ceramic’s central role in pushing forward the state of the art of IEs.
Project Description:
Introduction
As mentioned in Joel’s introduction and his talk on the topic, IEs are not new. In fact, one might argue that IEs’ objective functions - taking a measure of the world and distributing rewards accordingly - are crucial to the entire web3 project. Although instances of IEs are common, especially for measuring contribution in DAOs, most are relatively static and single purpose. Dedicated and programmable infrastructure for building them is yet to surface.
We aim to set the foundations upon which such infrastructure can be built. By combining the Autonolas stack with dynamic data streams on Ceramic, we plan to develop a simple impact evaluator, which tracks 1 type of contribution - @-mentions on Twitter. The scores for each contributing participant are stored on Ceramic.
This project will provide and demonstrate a way for Ceramic projects – and Web3 projects more broadly – to do community contribution tracking using datastreams. By developing the off-chain components of the IE with Autonolas, this toolkit lays the groundwork for the creation of an open, modular and self-improving framework for customizing how impact is evaluated in any decentralized organization.
Once delivered, our solution would provide the most advanced and extensible toolkit for pushing Impact Evaluators forward, cementing Ceramic’s role at the center of this promising technology. It also has potential to become a primitive for projects to drive data incentives, boosting utilization of the Ceramic network.
To begin with we describe main benefits to the Ceramic ecosystem. We then discuss the high level architecture of the MVP. Following this we go into a bit more detail with a technical specification. Then we explore how this product could be extended and grown far beyond the MVP. Finally, we offer more information about the Autonolas tech stack, the team and other projects we are working on.
Benefits to Ceramic
- Cement Ceramic as the place to build IEs. This MVP represents a milestone on the path towards a full concrete IE with Ceramic at the heart of its implementation. The 3box and Valory teams can point to it as a beachhead with a clear path forward for the future of IEs.
- Immediate, high Total Addressable Market. By tracking twitter mentions, even as an MVP the service is readily useful to all organizations with a twitter account.
High level architecture
We aim to:
- Deliver a toolkit for building simple IEs
- Demonstrate a live implementation of the toolkit as a decentralized, off-chain service powered by Autonolas
IEKit will be made up of a set of code, written in Autonolas’ open-source framework called Open Autonomy. Developers will be able to fork this and add new actions. From there they can run it as a single agent on centralized hosting, or immediately publish it to the Autonolas protocol and run it as a decentralized, off-chain service.
This project also aims to implement a canonical example of IEKit, to give a tangible demonstration of its capabilities. The demo aims to incentivize community members in the Autonolas community to tag the project on Twitter. This represents a simple incentivized action which we see as the core component of an Impact Evaluator.
The demo will run in rounds. Each round has 3 stages:
- data retrieval
- scoring
- writing
Data retrieval stage: the IE service retrieves all Autonolas @mentions from the Twitter API
Scoring stage: for every post that an individual account makes mentioning the Autonolas account, the IE service allocates 200 points. This happens in memory.
Writing stage: points are written to a Ceramic stream via the IE service’s DID.
Technical specification
We have designed a simple architecture for an IE to keep track of contributions to a community’s growth by mentioning the Autonolas handle in Twitter posts. Then, those contributions are assigned a score and the scores are written to a Ceramic stream.
Autonolas Service Specification
Core FSM
Autonomous services that use Autonolas technology are implemented as Finite State Machines where the logic is split across independent components: rounds define the rules to transition across different states and behavior, and implement the actual business logic. At the end of each round, agents agree on the round’s data output.
-
ReadData: agents pull data from the Twitter API. They retrieve all the posts which mention the Autonolas user account
-
Score: agents assign a score to each user according to how many posts have contributed.
-
WriteScores: the service writes the scores to a Ceramic stream. If the Safe DID is mature enough, this data write could be done in a decentralized way. All the agents can be set as owners of a Gnosis Safe contract and interact with the data stream through it. If Safe DID is not mature, agents can take turns randomly to write to the stream.
Figure 1: diagram of the core FSM
Keep in mind that the FSM diagram just accounts for the happy path and that error handling will be added once the complete technical requirements are known.
Chained FSM diagram
Pre-existing functionality, like agent registration and reset and pause rounds, will be chained to the core FSM (Figure 1) to form a complete FSM (Figure 2). The FinishedWriting round in the previous diagram represents a chaining point and will be removed during chaining.
Figure 2: diagram of the chained FSM
Roles and responsibilities
- Service developer: implements the rounds and its business logic, API integrations, FSM chainings, chain integrations and any other capability needed for the service to run.
- Agent: single instance of the service logic that runs the rounds.
- Agent operator: runs one or more agent instances as specified by the service owner. Updates to new versions when required by the service.
- Service owner: ultimately responsible for the service. Decides the service needs, the number of agents and operators and registers the service in the Autonolas protocol.
Other service details
-
Number of operators: depending on the level of decentralization we could increase or decrease this number. We recommend starting the service with a set of one operator running four agents and scale it up over time.
-
Number of API integrations: 1
-
Who runs the autonomous service: independent operators would run this service in a best-case scenario. Of course, at the beginning, it may need to be run internally. Once we have a tested MVP, we can seek independent operators.
-
Who pays for gas: gas fees, as well as infrastructure costs, are both part of the operational costs and service owners are responsible for covering them. In this case, Autonolas would be the service owner.
Extension
We aimed to fit the proposal into a relatively low budget. As a result we weren’t able to include several compelling features related to make a full-fledged Impact Evaluator:
Add Token Distribution Module
An immediate extension to the writing phase would be the pushing of an on chain transaction, which would distribute rewards directly to contributors. Depending on the organizations’ requirements one could airdrop the tokens or make them claimable via a Merkle tree.
Data mining incentives
Liquidity mining led to a step change in the levels of liquidity in DeFi, even after the bear market. Applying incentives to attract quality data could create an equally explosive outcome for the dataverse. This MVP could be extended to provide a technical architecture for incentivizing Ceramic actions.
Expand data input types
In future iterations of the service there is no limit to which types of contribution could be tracked. Data on members’ actions could be retrieved from on-chain events, or via any API. The contribution being tracked could even be the output of a Coordinape round and/or SourceCred round.
Expand scoring algorithms and Impact Evaluation
While the MVP simply counts contributions, it could be extended to evaluate them in more complex ways. It could, for example, employ Gitcoin Passport as a Sybil resistance check. Indeed, as the pool of actions being tracked grows because of the new data inputs added in the data retrieval stage, the methodologies by which actions are evaluated could become as arbitrarily complex as a machine learning or training model. These could optimize incentives to get the most value from the community. This bring us on one of the most compelling extensions.
Sophisticated Self-improvement
With the possibility of extending the MVP to autonomously distribute rewards according to any arbitrarily complex methodologies that has access to vast amounts of data inputs, including the data streams which power the impact evaluator itself, the stage is set for a self improving and self training impact evaluators to emerge. Even further beyond that, it would be possible for someone to create a more generalized, monolithic service which optimized over and aggregated all the IEs contributed to the framework.
An alternative way that the system could self improve is by outsourcing the training of the model to communities of token holders who could add and remove actions, and modify aggregation methodologies via gauge mechanisms. This would parallel the voting we already see on protocols like Curve. Although, rather than voting on LP rewards, these token holders vote on how and what gets rewarded. This system would therefore optimize impact evaluation via a market of incentivized actors.
Growth
To see maximum benefit to the Ceramic ecosystem, we have designed this MVP to be forked and composed in order to proliferate the number of IEs. Developers will be able to fork this code, develop new IEs and redeploy them via the Autonolas on-chain protocol.
Take a simple example. Say another DAO likes what they see from this running MVP, but feels that the current metrics are not relevant to their ecosystem. They would like to use different metrics and aggregate them to have more granular weighting for each metric. To get this going they would simply need to:
- Fork the open-source code generated during this project
- Swap out the existing APIs with new APIs
- Make any necessary changes to the data aggregation algorithm
- Register their new code on the Autonolas Protocol
- Operators will come in to run their code
This process could be replicated to create a host of different IEs. As developers get more adventurous, they will be able to add increasingly rich data sources which can also be directly imported as components via the protocol. They will also be able to create more and more sophisticated aggregation algorithms which could even use things like predictive ML models.
We have constantly improving documentation on the entire stack. We also help developers in their learning journey with our academy. Finally, through the composability the stack offers, developers can reuse many existing code components.
About Autonolas
Autonolas is a stack for building and running sophisticated, decentralized off-chain services, called Autonomous Services.
Autonomous services on Autonolas are made of software agents. Agents are a bit like bots but unlike bots, they are social and can work together. With autonomous services, agents work together to deliver a service. There can be just a few agents in a service or many. Agents are open-source software.
A different individual or company runs each agent. This makes autonomous services decentralized – no single agent can control the service. It also makes them robust, and capable of managing large amounts of capital. If one agent goes down, the rest can continue.
Autonomous services run continuously. Agents can keep a service running indefinitely. This means services can spring into action when something happens off the blockchain. This is something smart contracts cannot do.
Services pull in data from any source a developer chooses. For example
- an AWS storage bucket
- a crypto-native data source like IPFS, Ceramic or The Graph,
- or any other API.
Autonomous services can do sophisticated computation, even machine learning.
These services are reliable because agents must agree on what the service should do, via a Tendermint consensus engine.
Once the agents have come to consensus about what the service should do, they can take action. This action could be submitting a transaction on Ethereum, Avalanche, a Cosmos chain, Solana, etc. Services can also take action on a traditional API. This means that they are not limited to acting in the world of Web3. Autonomous services can be a bridge between the traditional tech world and crypto.
Autonomous services are transparent because the service is registered on-chain and the code is referenced via IPFS hashes. Each operator fetches the code by pointing to this service instance registered on-chain. Only if operators run compatible code will the application work. That means, as long as less than one third of operators are malicious or faulty, the integrity of the service is preserved. This limits the scope of misbehavior in v1. We are already exploring improvements to our stack which bring cryptographic guarantees to what happens inside an autonomous service.
Each autonomous service has a service owner who configures the parameters of the service, maintains it and decides on the incentives for operators to run it. It is worth reiterating for the current MVP, that because the service is run by multiple independent operators, it can perform all of its functionality in a way that is robust, trust-minimized and transparent.
The Autonolas Team
The team was founded and is led by:
- David Minarsch (CEO) – Ph.D. University of Cambridge in Applied Game Theory. Led and authored first blockchain-focused multi-agent system framework at Fetch.ai.
- David Galindo – Ph.D. Polytechnical University of Catalunya in Cryptography. Previously Head of Cryptography at Fetch.ai and Associate Professor of Cybersecurity at University of Birmingham.
- Oaksprout the Tan – 10+ years experience in Web2 product roles. Held recognized community contribution roles at Balancer, THORChain and Ceramic.
The company behind Autonolas, Valory, raised a seed round last year, and is made up of 14 team members on top of the co-founders. Learn more.
Related Autonolas Projects
Dynamic Community Recognition NFTs
Dynamic Community Recognition NFTs update their image according to external data points. The Autonolas community is already testing this primitive with NFTs that update their image according to the points they have collected on the community leaderboard. This could integrate naturally with the impact evaluator to reflect the score of participating contributors in the images of community member NFTs.
SMPs
An SMP is a service that autonomously manages a Balancer Managed Pool according to predefined conditions. It can take in data from any API and aggregate it in any arbitrarily complex manner, including advanced statistics and machine learning. It then uses the aggregated data to make decisions on how to reallocate the positions in a given pool. Finally, it executes the on-chain transactions necessary to make the appropriate changes to the pool.
The primitive enable asset/treasury managers to automate portfolio management according to off and on- chain signals.
Valory received a grant from Balancer to build an MVP of this product.
The Autonolas Leaderboard
Despite not having a token, Autonolas has already built a basic IE. It is incentivizing contributions to its ecosystem via an off-chain point system. The system tracks and aggregates certain data inputs for each contributor. Seeing as each contributor holds a community NFT, their points can be reflected dynamically in the NFT via the ddNFT service.
Oracle service
Autonolas has an oracle service running in production on Polygon. As well as a price oracle, we also run an oracle which pulls in data about different Uniswap v2 contract DEXes, using a ML model which we developed to come up with APY predictions per pool. This further exemplifies running services, but is mostly relevant in that it illustrates the incredible complexity that an autonomous service can handle, and points towards the rich strategies developers will be able to build to attract users to impact evaluators on Ceramic.
The code behind these services would likely be heavily reused for this proposed project. The same code could be used to pull in data from APIs.
Read more about the technical architecture of this oracle service.
Other Services
El Collectooorr is an autonomous NFT collector. It watches for drops on Art Blocks, and then uses an algorithm to make a decision about how to take part in the minting auction. This also illustrates the usage of sophisticated, proactive, off-chain services to deliver novel asset management products. El Collectooorr was built by community developers, illustrating the ability for external developers to take our framework and extend it into novel use cases.
In addition to these services built by our team and by independent community developers, we have also attracted developers from various crypto project teams. Most notably here, teams such as Brahma are using the stack for robustly managing the off-chain requirements of their protocols. Other earlier stage teams have been trialling the stack for rich community incentivization and coordination, as well as novel asset management use cases. As Open Autonomy – our framework for developing autonomous services – and the first major part of our protocol are live, we expect many more services to be in production soon.
Relevant links:
- Valory team website:https://www.valory.xyz/
- Autonolas website: https://autonolas.network
- Twitter: https://twitter.com/autonolas
- Github: Valory · GitHub
- Previous projects the team has meaningfully contributed to:
- Fetch
- Ceramic – Oaksprout won a Ceramic hackathon prize for Tiles – the browser for open, composable data. He is also on Ceramic’s community contribution team, Team Terracotta.
- Balancer – Oaksprout received a grant to develop Baller, a heavily-used tool for calculating impermanent loss on Balancer’s sophisticated liquidity pools. He later joined Balancer’s community contribution team, the Ballers.
- THORChain
- Comparable projects or peers:
- Fetch
Ceramic Ecosystem Value Proposition:
- What is the problem statement this proposal hopes to solve for the Ceramic ecosystem?
Despite the diversity of communities and organizations that compensate members for their contributions, impact evaluation is in its infancy. Solutions exist, but they are inflexible and only serve a subset of contributions types, leading to inefficient rewards distributions that do not necessarily push the needle towards the organization’s goals. Considering that there currently are over 2000 active DAOs and counting and Ceramic is an ideal way to track contribution there is a currently untapped opportunity here to significantly expand the Ceramic ecosystem.
- How does your proposal offer a value proposition solving the above problem?
By combining the Autonolas stack with Ceramic data this proposal offers a first glimpse of a modular and autonomous framework for customizing impact evaluators to the specific news of any organization. This granular composability is a key competitive advantage that could attract a large proportion of the market in the future.
- Why will this solution be a source of growth for the Ceramic ecosystem?
See Benefits to Ceramic above.
Funding requested (DAI/USDC): $15,000
Milestones:
-
Milestone #1: Development and testing of IE Kit (1 month) - $10,000 - 100 Hours @ 100 $/ Hr
- Build the Core FSM as specified above
- As we have an already existing component to build upon, the bulk of the work here will be to create the data pull and processing logic, integrate the required API and the logic to prepare the transactions.
- Test coverage
- External comms
- Build the Core FSM as specified above
-
Milestone #2: Demonstration by integrating with existing Autonolas contribution service and dynamic community recognition NFTs (1 month) - $5,000 - 50 Hours @ 100 $/ Hr
- Component registration on Autonolas protocol
- Progressive decentralization to multiple agents
- Testnet operations
- Mainnet deployment
- Frontend support
- External comms
I accept the 3Box Labs Grants terms and conditions: Yes
I understand that I will be required to provide additional KYC information to the 3Box Labs to receive this grant: Yes
)