Co-authored by Fraser Edwards and Eduardo Hotta.
We always planned on recapping how the cheqd airdrops went but in addition to that, we need to eat some humble pie and share the lessons we’ve learned in case they are of any use to projects executing future airdrops. Some of these are airdrop mechanics, some technical, and some communications and community building.
We’ve split this out into The Good, The Bad and The Ugly since that seems to bucket things nicely. For once, we’re not space-themed. Unfortunately (or fortunately) no one has made a TGTBATU in space.
- The Good: We blew way past our original expectations and really had to HODL (Hold On for Dear Life — the meaning being from the fictional origin story of HODL)
- The Bad: We tried some new technical approaches for the airdrop and these didn’t go exactly to plan. We also attempted to prevent and call out gaming of the airdrop but since our execution was off, we were throwing stones in glass houses.
- The Ugly: During Mission One we found significant gaming which could have exposed the network to a similar issue to the one which has split the Juno community. We’d like to mention that we respect and like Juno very much, so this is not in any way a criticism of their airdrop. Some of this we got right, some we got wrong.
Scene setting
Before we get into the action, we need to set the scene. Like most projects planning airdrops, we were aiming to widen our participation, awareness and people’s ability to build, especially as we were pretty unknown in the Cosmos ecosystem, having only founded the company in April 2021, launched in November 2021 and mainly focusing our marketing efforts on the Self-Sovereign Identity (SSI) world. We’re aiming to give people worldwide their data privacy, control and ownership back so improving participation and ownership fits well with our mission.
Before the airdrop, our community metrics were:
- Telegram: 12k
- Twitter: 25k
- Discord: 30 (not ks, 30!)
- Accounts: ~4.5k
Our original barometer for success was to increase at least our Telegram and Twitter communities by 5,000, which would have been a ~38% increase on Telegram, and 25% on Twitter.
We blew past these substantially. So let’s introduce the good.
The good
As we’ve hinted above, we shot past the internal targets we had set ourselves. Where we ended up was:
- Telegram: ~23k (~27k at peak) — 91.7% increase
- Twitter: ~48k (~50k at peak) — 94.4% increase
- Discord: ~11k (~15k at peak) — 36,566.7% increase (albeit from a very low level)
- Accounts: ~74k — 1,544.4% increase
This was almost entirely driven by Mission One where the aim was improving awareness through rewards for following us on Twitter and joining our communities on Telegram and Discord amongst others. We also wanted Mission One to allow more people to participate in Mission Two which was focused around staking and liquidity pooling (LPing).
As well as pure numbers, we’ve had many messages from members of communities we didn’t know existed (e.g. the Bangalore Web3 Cosmos community) to say we’re on their radar. Given some of our best contributors come from these communities this is exactly what we were hoping for and we hope to return their support!
Another fantastic thing that happened was the support and the encouraging words we received from our community. This airdrop experience created a strong bond between community members and also with cheqd.
The scale and pick-up caught us by surprise and led to the introduction of The Bad.
rewards.cheqd.io website traffic by country
The bad
Comms
We always aim to provide clear instructions and communicate in the simplest way possible so our messages could be understood by a wide group of people.
Unfortunately, we failed to provide a clear message in at least three instances
The first being that we stated in our first blog post that people entering the airdrop had to provide a cheqd wallet. However, more than 700 entries provided non-cheqd wallet addresses (cosmos, osmo, juno, regen, umee, etc.), which we decided to convert into cheqd wallets, so more people could participate in the airdrop.
The second time is the following:
A lot of people did not understand the phrase ‘week commencing’ and thought the claim website would be live on Monday 14th March 2022.
Why are we sharing this?
We took longer than we anticipated to start distribution due to the time needed to convert non-cheqd to cheqd wallets.
The second one created a situation where people started complaining that the claim page wasn’t live on Monday. This snowballed into people spreading the misinformation that we ‘delayed or cancelled’ the distribution as early as it was morning in Asia (and our team mostly follows UK hours).
Finally, the third instance happened when we closed the claim webpage as we went through a network upgrade. We had created a ‘maintenance’ page based on the design of the ‘Your Geography / IP address range is blocked’ page that was being displayed in affected regions.
Unfortunately, the ‘maintenance’ page took too long to propagate on the network, therefore people visiting the claim webpage during that time saw a ‘Your Geography / IP address range is blocked’, which created a bit of confusion that eventually led to a heavy flow of messages in our groups.
Oh, and one last thing… we were also already on high alert…
Our sensitivity to anything worrying was heightened after a project started using our ticker ($CHEQ) and had all the signs that it would be a rugpull during Mission One. This project copied and used a lot of the same things we do in our community (e.g. saying ‘cheq this out’ and their community members being called ‘cheqmates’) and also ran their own airdrop.
Without spending too much time on this, the project was indeed a rugpull and used our brand and airdrop to confuse people in the Cosmos and Juno community. Luckily, they rebranded and changed a lot of things before their rugpull, which meant they didn’t harm our brand that much. We thank our community for making them rebrand in a very short space of time!
Now going back to the gaming of the airdrop…
We’ll cover it properly in The Ugly but we saw a substantial amount of gaming of the airdrop. Enough to threaten the entire integrity of it.
This meant we had to review our original reward tiers and the amount of the rewards itself. In a nutshell, we decided to reduce the staking and increase the liquidity pool rewards.
Whilst we knew we hadn’t protected ourselves fully for Mission Two (you never can with gaming, it’s always a game of cat and mouse), we did want to call this out to other projects that were about to execute airdrops so they could avoid what we were going through.
Where we got it terribly wrong was not having our own house in order. We do wonder if we would have received the same reaction if claims had been working perfectly for those who were eligible, but alas, we’ll never know, and we have to own it. Thank you to those who did fight our corner, we really appreciated it, and it made a big difference to us.
Another thing is that we didn’t have enough manpower to reply to every tweet, support ticket, and message on Telegram and Discord. Most of the complaints, feedback, and messages came on a Saturday. Our team was so exhausted that we had to rest during that day. In hindsight, we should have bought a lot of energy drinks and answered some of the questions being brought up.
We hope this detailed breakdown goes some way towards an act of repentance.
execution
Our Head of Marketing reviewing this section of the blog
Originally, the criteria for Mission Two were intended to be CHEQ “AND” others rather than CHEQ “OR” others as it became, to use logic gate terms. The first would have meant only people holding CHEQs would be eligible, whereas the second meant anyone who fit any of the criteria was eligible regardless of prior interaction with CHEQs.
However, during Mission One, we thought that it would be a good idea to give more people the opportunity to join Mission Two. The idea was simple: allow ATOM, OSMO, and JUNO stakers to claim an initial CHEQ reward that they could then stake or participate in a Liquidity Pool.
Thus, we split the claim in two stages, the first one for ATOM, OSMO and JUNO stakers, and the second for CHEQ stakers and LPs.
But…
We started getting feedback from the community saying that they preferred to have just one claim, as two would be too complicated. We listened and went back to having just one claim, but since we had already announced that it was “OR” instead of “AND”, we didn’t change that back.
This meant the universe of eligible addresses increased from ~30k (4.5 existing, max of 25k new due to limit) to 204k. However, our approach and processes had been designed around ~30k addresses, which we quickly found issues with.
snapshots
One of the major issues that resulted from this was that our snapshot execution was off at this new scale. The typical way to take snapshots for an airdrop is by running a full archive node for the target chains. For large chains (e.g. OSMO), this requires running what can be quite heavy infra, waiting to catch-up then being able to work with this so it can be time consuming. It also means that all of the logic for the airdrop is in code and is effectively a black-box to non-devs, putting the burden for execution and support resolution with the dev team.
We attempted to circumvent this by making API calls using Google App Scripts (similar to those used by block explorers) to pull results then use either Excel or Sheets to process these ready for distribution. This would have provided a lightweight execution and allowed the development team to focus on executing the core roadmap as well as allowing non-dev members of the team to test and support.
The first snag introduced by 204k accounts was that App Scripts was limited to 100k requests per day. Since we were checking across ATOM, OSMO and JUNO chains, we were well beyond that. After hitting rate limit alerts, we split the requests, so they were made from multiple accounts, however, we still hit the limits but without sufficient alerts in place with hindsight. This is the root cause of the majority of the genuine support tickets we have received.
The second was stitching together the results, ensuring they were unique and calculating rewards. Excel and Sheets were adequate tools for 25k accounts but were beyond their limits when tasked with 204k records requiring blends of computationally intense formulas (at least on this scale) such as vlookups, indexes and matches to combine them into a cohesive list. This meant every time we identified an issue, investigated, resolved and rolled it out, the processing took a colossal amount of time. Again, this would have been no problem at all at 30k, but hugely time consuming at 204k.
snapshots
After seeing issues in the speed of distribution for Mission One, for Mission Two, we used a CosmJS-based Cloudflare Worker, which is a highly scalable/fast way of doing distributions, using six distribution wallets to include multiple distribution transactions per block.
rewards.cheqd.io website traffic spike
However, one of the assumptions we built into this model was being able to use Cloudflare Works Unbound (up to 30 seconds CPU time) versus Cloudfare Workers Bundled. Unfortunately, Unbound is only possible with a custom enterprise contract which isn’t highlighted in the documentation. Although we had six distribution wallets, when we had a flood of transactions, each worker was only able to run for 100ms max before being throttled, forcing us to drop down to one distribution account whilst we refactored.
CloudFlare Workers comparison
Over the weekend, we refactored to use three Workers (with two accounts each) that could fit comfortably within the execution time for a Bundled Cloudflare Worker, which comfortably cleared the queue down extremely quickly. During this refactoring, we spotted an issue where it was possible to claim multiple times. As we had assumed at worst, a small distribution queue, we were checking whether there had already been a distribution to an address when anyone attempted to claim. This meant if you submitted multiple claims before the first distribution was made, there would be multiple distributions. We, therefore, shifted the check to the point of distribution to prevent multiple claims.
The other issue was that we stored the list in a key-value pair called Cloudflare Workers KV (key, value; example below), which sorts the keys (in our case, wallet addresses) alphabetically. Originally, this would not be a problem since the speed of distribution using the Unbound usage model would prevent any backlog of addresses.
Key: cheqd1…xxx Value: {“amount”:50,”timestamp”:1649853749658}
Unfortunately, until this was resolved, it meant that anyone low down the list of addresses alphabetically was constantly being pushed to the end by new requests. We fixed this by appending a randomly-assigned “queue-X” prefix in front of the key.
New functionality required
One major downside of not getting things right the first time is that it often introduces the need for new features which weren’t originally required.
As a result of the issues in the snapshots, we had a blend of:
- People who had not claimed
- People who had successfully claimed against a correct allocation
- People who had successfully claimed against a partially correct allocation
Since our distribution was built around a single claim/distribution, we had to amend this to allow those in category three to claim any missing rewards, moving from a single claim/distribution to multiple with more complex logic. Eventually, this accounted for:
- Eligible for
- Claimed
- Claimed but not yet distributed
- Eligible for but not claimed
Circling back, this cascade all started from the decision to increase eligibility from 30k to 204k wallets. Whilst we thought our approach would support this increase in volume, we were quickly disillusioned.
However, this baptism of fire did have some major upsides.
The good (again)
The result of some extremely late nights and colossal team efforts (special shout out to Ankur, Ross, Tasos and Javed here) was our infrastructure distribution and support processes had not only been volume tested but also drastically improved.
Whilst we would advocate that teams follow the typical snapshot processes, the distribution process we now have is much more feature complete, quicker and flexible than originally as well as having been thoroughly volume tested. We will be open-sourcing this part of the code-base for use in any future projects.
The ugly
The first inkling that something was off was when we downloaded the completed entries from Mission One, and spotted a suspicious series of emails, e.g. [email protected], [email protected], etc, when scanning through.
Since we had a lot of data points from Mission 1 (Telegram handle, Twitter handle, email addresses, who referred whom, etc.) we started investigating to understand the scale of the problem and the potential exposure from Mission Two since the rewards were much larger.
The most egregious of these were:
- Multiple cheqd accounts per Twitter account:
- 1 twitter account which had submitted 49 different addresses
- Overall we had 915 accounts where 1 Twitter account owned multiple addresses
- The same cheqd address submitted by multiple email addresses or Twitter accounts:
- 1 account submitted by 160 different email addresses
- Overall we had 781 addresses that were submitted by different email or Twitter accounts
We also had 116 instances in support tickets where the airdrop was transferred out of the account and then a ticket raised claiming it was never received.
Whilst investigating, we identified that the bulk of these were coming from the same country. This behaviour was confirmed as widespread when we saw it openly discussed on our Discord server with syndicates and dumping both openly talked about.
Since we identified 12k wallets from this country with a high proportion of gaming, we took the difficult decision to add these to a denylist and geoblock then work through valid claims via our support processes.
And, believe us… it wasn’t easy.
Currently, out of all the addresses we blocked and the tickets raised, we had less than 90 valid claims for this country. Further, the majority of these had already been dumped. In the spirit of fair play, since they followed the airdrop rules, we have rewarded them.
As we covered in The Bad, we attempted to highlight this to other projects about to execute airdrops but we didn’t get this right in a number of areas. However, now that we are through the majority of the tickets, we are comfortable that we took the right approach, although we should definitely have made sure all our processes were working before we highlighted the issues we were seeing.
If there are any projects about to do an airdrop, we would be more than happy to share in detail what we learned, the investigations we did, and what we would change for the future, including addresses which would be worth checking for exposure when shaping an airdrop to check.
The really ugly
Now, to the really ugly stuff…
After we communicated our decision about the geoblock, a group of people spread disturbing lies (which we won’t repeat here) and tried to ban us from social media by reporting our cheqd profiles. This same group also incentivised people to downvote us on CoinMarketCap and CoinGecko. Although we can’t prove it, we believe this group has done these activities before, as it was well coordinated and done at an impressive speed.
And…
You can tell a lot about someone by the way they react when challenged and regardless of whether we were right or wrong in our actions, the abuse that followed wasn’t and isn’t acceptable anywhere for any reason.
Rather than engaging constructively, our Community Managers (CMs), Ambassadors and team were subjected to disgusting racism, sexism, sharing of pornography, and threats of violence, including death threats. The correlation between polite genuine claims and vitriol from those trying to game the airdrop was pretty much perfect.
We have to say a massive thank you to our CMs, Ambassadors and the wider community for their patience and grit in responding to this!
The plot timeline
The plot points above are all intertwined, so we’ve created a timeline view below to show them in chronological order.
Lessons Learned
It’s not as easy as it seems to run an airdrop campaign. As you have seen with our airdrop, and many others, people will try to make the most out of it. It’s normal, a natural human behaviour. We knew this would happen and we tried our best to reward our real community (old and new).
We’re sharing this blog post to help other projects out there that are facing similar issues and to answer some questions that people outside our community might have.
If you have any questions feel free to drop a message on Telegram or Discord.
And, to close this chapter and move on to the next, we ask people that are participating in other airdrops, to be patient, have empathy, and try to understand that behind a project there is a team of people.
People need to eat, sleep, and are working their a** off to get things done right. Like anything in life, sometimes things go as planned, sometimes there are bumps in the way.
We would like to end this blog post by thanking our community members, Community Managers, and Ambassadors for backing us and giving us the strength to keep fighting the Data Wars.
