Profile of elizaoak in Optimism
Posts by elizaoak
-
When Is Deliberation Useful for Optimism Governance?
by elizaoak - This user is a moderator
Posted on: Oct. 25, 2024, 2:12 p.m.
Content: When Is Deliberation Useful for Optimism Governance? Experimental Evidence from Retro Funding 4 ’s Deliberative Process
→ Full draft of the academic paper (linked here soon)
Summary
One of the most important questions for Optimism governance is: How do we increase the quality of discussion and information exchange so that participants can make informed decisions? One idea is to facilitate spaces dedicated to deliberation, and deliberative processes are gaining traction as a democratic tool for both offline (eg, Ireland’s citizens assembly) and online (eg, Meta’s community forums 1 ) governance. Despite the rise in adoption, there are open questions about when and how deliberation is useful. Moreover, web 3 introduces new opportunities to study the effects of deliberation in a governance structure that progressively decentralizes over time, and in the case of Optimism participants hold immediate decision-making power and the results of the deliberative process were binding. To understand how deliberation might be useful in our governance system, we designed a deliberative process experiment prior to Retro Funding 4 in Optimism’s Citizens House. (See these links for more context on the topic and logistical details of Optimism’s deliberative process.)
In our experiment, we find that:
( 1 ) Participants in the deliberative process significantly increased their knowledge on arguments for and against the contested topic
( 2 ) Participants in the deliberative process significantly increased their trust in the opinions of other badgeholders
( 3 ) Deliberation seemed to increase support for deliberation as an effective decision-making tool
( 4 ) Deliberation moved participants toward consensus on only one of four categories
( 1 ) Deliberation increases knowledge on arguments for and against contested topic
Theory: Potential benefits of deliberation include more informed and reflective judgements, a greater sense of internal political efficacy (that is, confidence people have in their personal knowledge and competence to engage in political discussions and make informed decisions).
Outcome to test Hypothesis 1 : Deliberation Increases Knowledge
Outcome:
How would you rate your knowledge on the arguments for/against deducting external funding (e.g., VC funding, Optimism grants, or other grants) from public goods rewards?
Scale:
0 (very low knowledge) ... 4 ... 7 (very high)
i. Participants had higher post-deliberation knowledge than pre-deliberation
Paired T-test Results
MeanPre-deliberation
4 . 96
MeanPost-deliberation
5 . 83
Mean Difference
+ 0 . 88 ***
Standard Error
( 0 . 11 )
t-statistic
3 . 6
Degrees of Freedom
23
Comparing pre/ post-survey responses, participants increased about 0 . 88 on a 7 -point scale measuring their knowledge on the deliberated topic. With a p-value of 0 . 003 this is a statistically significant shift in opinions among participants, and substantively moving nearly 1 point on a 7 -point scale is quite meaningful. Thus, we find strong evidence that those who participated in the deliberative process significantly increased their knowledge on the arguments for and against the contested topic. However, it’s worth keeping in mind that there might be other factors during that time period that could’ve increased knowledge on these topics other than the deliberative process itself, so comparing the participants (treatment) to non-participants (control) is helpful to rule out those factors, which we discuss below.
ii. Participants reported more knowledge on the topic than non-participants
In addition to comparing changes among participants before and after deliberation, we also compared post-deliberation knowledge of deliberative process participants to non-participants. We were able to do this because all 105 badgeholders who voted in Retro Funding Round 4 answered these same survey questions as part of the post-voting survey (which took place shortly after the conclusion of the deliberative process).
If we simply compare the means of deliberative process participants (treatment group) and non-participants (control group), however, we’re not accounting for possible selection bias: Only half of the 50 randomly selected citizens ultimately chose to participate, and there are probably some systematic, non-random differences between those that accepted the invite and those that didn’t (for instance, maybe those who accepted the invite are more engaged, or have more time, etc). Those systematic differences might also influence the outcome of knowledge on the contested topic, as opposed to differences in knowledge being driven by participation in the deliberative process.
Effect of Deliberation on Knowledge (IV Estimation)
Variable
Estimate
Std. Error
t statistic
(Intercept)
5 . 113
0 . 148
34 . 54
Deliberation participation
0 . 381
0 . 414
0 . 92
Adjusted R-Squared
0 . 051
Wald Test
0 . 8455 on 1 and 103 DF
To get around this, we ran an instrumental variable (IV) analysis, which essentially takes into account this selection bias and adjusts the estimates accordingly. With this model, we see there is about a 0 . 4 increase in knowledge (fairly substantive on a 7 -point scale) but the relatively large standard errors suggest there’s some noise in the data (probably also due to the fact that n = 24 is a relatively small sample size). This noise, which results in a lack of statistical significance, might also mean that knowledge levels were not as distinctly different between both treatment and control groups because there were other factors in the broader environment that both groups were exposed to and that also influenced knowledge — such as spillover effects from the deliberative process “treatment,” e.g. public gov forum updates summarizing the arguments for or against deducting funding.
The table below shows the raw means of the control group (the 62 badgeholders who were not invited to participate in the deliberative process but who did vote in Retro Funding Round 4 ) compared to the “compliers” (the 24 badgeholders who participated in both deliberative process sessions) and the “non-compliers” (the 19 badgeholders who were randomly selected to participate in the deliberative process but did not choose to, but who did ultimately vote in Retro Funding Round 4 ). We see that there do seem to be some baseline differences in post-deliberation knowledge between the three groups.
Mean Knowledge by Compliance Group
Subgroup
n
Mean Knowledge
Std. Dev.
Median Knowledge
Complier
24
5 . 83
0 . 92
6
Non-Complier
19
4 . 67
1 . 24
4
Control
62
5 . 11
1 . 06
5
iii. Qualitative evidence suggests that the acquired knowledge came primarily from small group discussions, followed by the info kit
When asked “what arguments or information most influenced views on the topic during the deliberative process,” 15 of the 24 participants ( 62 . 5 %) mentioned small group discussions in their free response entry, for instance:
“A number of things specific individuals said in the small group breakouts I found most influential because they often filled in gaps in my knowledge”
“Generally, the opportunity to hear other people’s perspectives during the breakout sessions”
“Some of the discussion yesterday was helpful to broaden my view, but I didn’t really change my mind very much.”
This was followed by 6 out of 24 ( 25 %) mentioning the information kits which were shared as preparatory pre-reading, for instance:
“Information kit & small group breakouts were both incredibly helpful in advancing my thinking on the topic”
"The information kit was very helpful for setting context. I wouldn’t say they influenced my views much though, but they helped clarify things for me. "
It’s worth noting that multiple responses explicitly said that while the deliberative process enhanced knowledge on the topic, it didn’t necessarily change their opinion. This aligns with the findings we discuss in a later section on whether deliberation seemed to increase consensus on the contested topic.
( 2 ) Deliberation increases trust in opinions of other badgeholders
Theory: Deliberative processes like citizens’ assemblies provide a space where participants can develop trust through dialogue and problem-solving, and past empirical work has found that deliberation can increase empathy and trust between participants as individuals are exposed to the lived experiences and perspectives of others.
Outcome to test Hypothesis 2 : Deliberation Increases Social Cohesion
Outcome:
To what extent do you trust the opinions of other badgeholders?
Scale:
0 (very low trust) ... 4 ... 7 (very high trust)
i. Participants had higher trust in the opinions of other badgeholders post-deliberation compared to pre-deliberation
Paired T-test Results
MeanPre-deliberation
4 . 43
MeanPost-deliberation
5 . 13
Mean Difference
+ 0 . 7 ***
Standard Error
( 0 . 17 )
t-statistic
4 . 04
Degrees of Freedom
23
Comparing pre/ post-survey responses, participants increased about 0 . 7 on a 7 -point scale measuring their trust in other badgeholders. With a p-value of 0 . 001 this is a statistically significant shift in opinions among participants, and substantively moving nearly 1 point on a 7 -point scale is quite meaningful. Thus, we find strong evidence that those who participated in the deliberative process significantly increased levels of trust in the opinions of other badgeholders.
ii. Participants had higher post-deliberation trust than non-participants
Effect of Deliberation on Trust in Other Badgeholders (IV Estimation)
Variable
Estimate
Std. Error
t statistic
(Intercept)
4 . 839
0 . 151
32 . 054
Deliberation participation
0 . 456
0 . 423
1 . 078
Adjusted R-Squared
- 0 . 009
Wald Test
1 . 162 on 1 and 103 DF
Similar to above, we used IV estimation to account for potential selection bias given the opt-in nature of the treatment. We see there is about a 0 . 5 increase in trust in other badgeholders (fairly substantive on a 7 -point scale) but again, the relatively large standard errors suggest there’s some noise in the data (probably also due to the fact that n = 24 is a relatively small sample size).
( 3 ) Deliberation increases support for deliberative process itself
Theory: Participants in deliberative processes will come to view deliberation as a legitimate and effective way to make decisions, meaning participants gain trust and appreciation for the deliberative process itself.
i. Among participants, 22 out of 24 ( 92 %) were supportive of a deliberative process being useful for additional topics and even bigger settings
The table below summarizes responses to the question: “Do you think a deliberative process could be useful for additional contested topics and even bigger settings?” We see that the overwhelming majority of participants were supportive of deliberation as an effective governance tool.
delib_supportive 1584 × 1168 96 . 9 KB
Qualitative data source: This data comes from a free response question on the post-deliberation survey administered to participants.
ii. For both participants and non-participants alike, and for those for and against ratifying the definition of profit, there were consistent themes of positive support toward deliberation as an effective tool in the decision-making process
delib_ratify_discussion 1264 × 938 105 KB
Qualitative data source: This data comes from governance forum discussion comments, and ratification voting data on Snapshot.
( 4 ) Deliberation moved participants toward consensus on only one of four categories
Theory: Structured discussions can lead to significant shifts in opinion and create more nuanced, consensus-based decisions because deliberation creates an environment which participants can engage in with diverse viewpoints and can help bridge ideological divides.
Outcome to test Hypothesis 4 : Deliberation Increases Consensus
Outcome:
Which of the following types of external funding do you think should be deducted from public goods funding rewards?
Select all that apply:
Grants from Optimism
VC funding
Grants from other ecosystems
Revenue
NONE of these types of funding should be subtracted from the total amount of Retro Rewards
Other (enter here)
The tables below show the aggregate pre/ post- survey results for the question about which types of external funding should be deducted for the 24 badgeholders who participated in both deliberative processes. For the categories of VC Funding, OP Grants, and Other Grants, there is no strong evidence of consensus post-deliberation. In the case of VC Funding, the majority vote switched in favor of being “against deducting” but remained at a close margin. For the category of Revenue, it does appear that opinions became more aligned, with only about 20 % of badgeholders in favor of deducting post-deliberation, compared to 46 % being in favor of deducting pre-deliberation. The pairwise t-test results for changes in opinion on deducting Revenue also confirm that this is a statistically significant shift. There is no statistically significant shift in the other three categories.
Support for Deducting External Funding (Pre-deliberation):
VC Funding
OP Grants
Grants
Revenue
In Favor of Deducting
14
14
8
11
Against Deducting
10
10
16
13
Total
24
24
24
24
Support for Deducting External Funding (Post-deliberation):
VC Funding
OP Grants
Grants
Revenue
In Favor of Deducting
11
15
6
5
Against Deducting
13
9
18
19
Total
24
24
24
24
What’s Unique about the Optimism Case?
There are some things unique to Optimism’s deliberative process that should be taken into account when interpreting these findings:
i. Highly engaged population
This deliberative process was conducted among Optimism’s badgeholders, a group of highly engaged governance stakeholders who skew technical in their professions and are based in approximately 20 different countries. Of the 50 badgeholders randomly selected to participate, the 25 who showed up were likely especially engaged as they made time to participate in two 90 -minute deliberative sessions (and in most cases, attended a third optional 90 -minute deliberative session) in addition to completing the pre-reading before each session. In some cases, participants joined despite being based in inconvenient time zones (e.g., joining from Asia where the local time was 2 am). Throughout the process, participants repeatedly requested additional pre-reading homework and asked for a third deliberative session.
ii. Direct policy implications
Another unique feature of this deliberative process is the fact that, pending ratification from the full set of badgeholders, the proposed definition of profit decided upon during the deliberative process would be directly implemented into the upcoming Retro Funding Round 4 . This is distinct from deliberative processes in other contexts that have been criticized for being performative, or at best advisory. Optimism, on the other hand, is a governance structure that progressively decentralizes over time, meaning badgeholders held immediate decision-making power in Retro Funding Round 4 .
iii. Lack of clarity on the topic
Throughout the process, it became clear that the topic of whether external funding should be deducted from public goods funding rewards was a question that also contains value judgements, whereas the resulting policy was meant to focus on the practical implementation details of this question. For example, a discussion about whether VC funding should be deducted from public goods funding is different than a discussion about whether VC funding could be deducted given limitations in what funding data is available for projects and whether it’s possible to deduct accurately. There were also time constraints given that the new definition of profit needed to be ratified before the upcoming Retro Funding Round 4 .
Recapping Key Takeaways
In sum, the key takeaways from this analysis are:
( 1 ) Running another deliberative process could help to increase participants’ knowledge on the arguments for/ against the deliberated topic:
There’s strong evidence that those who participated in the deliberative process became more knowledgeable about the topic, with badgeholders reporting a significant increase in their understanding of the arguments for and against the topic after engaging in the deliberative process.
Comparing participants to non-participants, deliberative process participants tended to report higher knowledge than non-participants after the deliberative process, though whether deliberation alone is driving this difference is less certain (likely due to the small sample size of 24 participants, or perhaps spillover effects from the deliberative process also increased non-participant knowledge which makes the post-deliberation difference less meaningful)
( 2 ) Running another deliberative process could help to increase participants’ trust in the opinions of fellow governance participants:
There’s strong evidence that participating in the deliberative process led badgeholders to trust each others’ opinions more, with badgeholders reporting a significant increase in trust in the opinions of other badgeholders after engaging in the deliberative process.
Comparing participants to non-participants, deliberative process participants tended to report higher trust than non-participants after the deliberative process, though again, whether deliberation alone is driving this difference is less certain (likely due to the small sample size of 24 participants)
( 3 ) Running the deliberative process seemed to encourage support for the deliberative process itself as an effective decision-making tool
This support for a deliberative process as a useful governance tool came from participants and the broader population of non-participants
This support came from those who voted in favor and those who voted against the new definition of profit that was decided during the deliberative process
( 4 ) Participating in the deliberative process only seemed to increase consensus on one of four categories under the deliberated topic
This suggests that deliberation may foster agreement on specific aspects of a topic (perhaps through increased trust and information exchange, as documented above) though not universally across all categories, so this question probably merits further study
This finding is likely also influenced by the practical constraints and lack of clarity surrounding the topic of the deliberative process, and the fact that the category (Revenue) which did see movement toward consensus is less polarizing and more clearly scoped than other categories (e.g., moral aversion to VCs, or lack of clarity on what is considered an OP Grant, etc.)
What This Means for Optimism Governance
We found evidence that deliberation can be a tool for knowledge exchange and increasing social cohesion, so this might make sense in contexts where comprehension of complex topics, or where bridging a divide between parties with low levels of trust, is important. Specifically, our findings suggest that a deliberative process might be useful for improving comprehension of complex documents like the Law of Chains or Blockspace Charters. A deliberative process could also be useful as a tool to facilitate conversation between the Token House and Citizens House in a way that improves trust in the other House and thereby reduces gridlock.
We found no strong evidence that deliberation shifted views toward consensus on the overall topic deliberated, suggesting that this type of deliberation tied to a binding policy recommendation may be less useful for the most polarizing topics, or for philosophical values questions which are difficult to coerce into a measurement question. With this in mind, we believe further exploration and study is warranted as there were unique considerations related to the specific topic that was discussed and which may have impacted outcomes. There are no immediate plans to implement another deliberative process experiment but we may explore this as appropriate.
Finally, we are very grateful to our facilitators who made this deliberative process happen: @Antoine Vergne (Missions Publiques), Inês NW (Inês NW), and Andrea Gallagher (RnDAO).
Thanks for reading and would love to hear any reactions in the comments below!
Likes: 9
Replies: 1
Replies:
- ccerv1: Fascinating read, thank you for sharing!
elizaoak:
suggesting that this type of deliberation tied to a binding policy recommendation may be less useful for the most polarizing topics, or for philosophical values questions which are difficult to coerce into a measurement question
I often wonder how to determine what the most polarizing issues are among citizens, and then to get a sense of how/if people’s views shift over time. This seems to be at the heart of figuring out the impact equation.
-
When Is Deliberation Useful for Optimism Governance?
by elizaoak - This user is a moderator
Posted on: Oct. 25, 2024, 2:12 p.m.
Content: When Is Deliberation Useful for Optimism Governance? Experimental Evidence from Retro Funding 4 ’s Deliberative Process
→ Full draft of the academic paper (linked here soon)
Summary
One of the most important questions for Optimism governance is: How do we increase the quality of discussion and information exchange so that participants can make informed decisions? One idea is to facilitate spaces dedicated to deliberation, and deliberative processes are gaining traction as a democratic tool for both offline (eg, Ireland’s citizens assembly) and online (eg, Meta’s community forums) governance. Despite the rise in adoption, there are open questions about when and how deliberation is useful. Moreover, web 3 introduces new opportunities to study the effects of deliberation in a governance structure that progressively decentralizes over time, and in the case of Optimism participants hold immediate decision-making power and the results of the deliberative process were binding. To understand how deliberation might be useful in our governance system, we designed a deliberative process experiment prior to Retro Funding 4 in Optimism’s Citizens House. (See these links for more context on the topic and logistical details of Optimism’s deliberative process.)
In our experiment, we find that:
( 1 ) Participants in the deliberative process significantly increased their knowledge on arguments for and against the contested topic
( 2 ) Participants in the deliberative process significantly increased their trust in the opinions of other badgeholders
( 3 ) Deliberation seemed to increase support for deliberation as an effective decision-making tool
( 4 ) Deliberation moved participants toward consensus on only one of four categories
( 1 ) Deliberation increases knowledge on arguments for and against contested topic
Theory: Potential benefits of deliberation include more informed and reflective judgements, a greater sense of internal political efficacy (that is, confidence people have in their personal knowledge and competence to engage in political discussions and make informed decisions).
Outcome to test Hypothesis 1 : Deliberation Increases Knowledge
Outcome:
How would you rate your knowledge on the arguments for/against deducting external funding (e.g., VC funding, Optimism grants, or other grants) from public goods rewards?
Scale:
0 (very low knowledge) ... 4 ... 7 (very high)
i. Participants had higher post-deliberation knowledge than pre-deliberation
Paired T-test Results
MeanPre-deliberation
4 . 96
MeanPost-deliberation
5 . 83
Mean Difference
+ 0 . 88 ***
Standard Error
( 0 . 11 )
t-statistic
3 . 6
Degrees of Freedom
23
Comparing pre/ post-survey responses, participants increased about 0 . 88 on a 7 -point scale measuring their knowledge on the deliberated topic. With a p-value of 0 . 003 this is a statistically significant shift in opinions among participants, and substantively moving nearly 1 point on a 7 -point scale is quite meaningful. Thus, we find strong evidence that those who participated in the deliberative process significantly increased their knowledge on the arguments for and against the contested topic. However, it’s worth keeping in mind that there might be other factors during that time period that could’ve increased knowledge on these topics other than the deliberative process itself, so comparing the participants (treatment) to non-participants (control) is helpful to rule out those factors, which we discuss below.
ii. Participants reported more knowledge on the topic than non-participants
In addition to comparing changes among participants before and after deliberation, we also compared post-deliberation knowledge of deliberative process participants to non-participants. We were able to do this because all 105 badgeholders who voted in Retro Funding Round 4 answered these same survey questions as part of the post-voting survey (which took place shortly after the conclusion of the deliberative process).
If we simply compare the means of deliberative process participants (treatment group) and non-participants (control group), however, we’re not accounting for possible selection bias: Only half of the 50 randomly selected citizens ultimately chose to participate, and there are probably some systematic, non-random differences between those that accepted the invite and those that didn’t (for instance, maybe those who accepted the invite are more engaged, or have more time, etc). Those systematic differences might also influence the outcome of knowledge on the contested topic, as opposed to differences in knowledge being driven by participation in the deliberative process.
Effect of Deliberation on Knowledge (IV Estimation)
Variable
Estimate
Std. Error
t statistic
(Intercept)
5 . 113
0 . 148
34 . 54
Deliberation participation
0 . 381
0 . 414
0 . 92
Adjusted R-Squared
0 . 051
Wald Test
0 . 8455 on 1 and 103 DF
To get around this, we ran an instrumental variable (IV) analysis, which essentially takes into account this selection bias and adjusts the estimates accordingly. With this model, we see there is about a 0 . 4 increase in knowledge (fairly substantive on a 7 -point scale) but the relatively large standard errors suggest there’s some noise in the data (probably also due to the fact that n = 24 is a relatively small sample size). This noise, which results in a lack of statistical significance, might also mean that knowledge levels were not as distinctly different between both treatment and control groups because there were other factors in the broader environment that both groups were exposed to and that also influenced knowledge — such as spillover effects from the deliberative process “treatment,” e.g. public gov forum updates summarizing the arguments for or against deducting funding.
The table below shows the raw means of the control group (the 62 badgeholders who were not invited to participate in the deliberative process but who did vote in Retro Funding Round 4 ) compared to the “compliers” (the 24 badgeholders who participated in both deliberative process sessions) and the “non-compliers” (the 19 badgeholders who were randomly selected to participate in the deliberative process but did not choose to, but who did ultimately vote in Retro Funding Round 4 ). We see that there do seem to be some baseline differences in post-deliberation knowledge between the three groups.
Mean Knowledge by Compliance Group
Subgroup
n
Mean Knowledge
Std. Dev.
Median Knowledge
Complier
24
5 . 83
0 . 92
6
Non-Complier
19
4 . 67
1 . 24
4
Control
62
5 . 11
1 . 06
5
iii. Qualitative evidence suggests that the acquired knowledge came primarily from small group discussions, followed by the info kit
When asked “what arguments or information most influenced views on the topic during the deliberative process,” 15 of the 24 participants ( 62 . 5 %) mentioned small group discussions in their free response entry, for instance:
“A number of things specific individuals said in the small group breakouts I found most influential because they often filled in gaps in my knowledge”
“Generally, the opportunity to hear other people’s perspectives during the breakout sessions”
“Some of the discussion yesterday was helpful to broaden my view, but I didn’t really change my mind very much.”
This was followed by 6 out of 24 ( 25 %) mentioning the information kits which were shared as preparatory pre-reading, for instance:
“Information kit & small group breakouts were both incredibly helpful in advancing my thinking on the topic”
"The information kit was very helpful for setting context. I wouldn’t say they influenced my views much though, but they helped clarify things for me. "
It’s worth noting that multiple responses explicitly said that while the deliberative process enhanced knowledge on the topic, it didn’t necessarily change their opinion. This aligns with the findings we discuss in a later section on whether deliberation seemed to increase consensus on the contested topic.
( 2 ) Deliberation increases trust in opinions of other badgeholders
Theory: Deliberative processes like citizens’ assemblies provide a space where participants can develop trust through dialogue and problem-solving, and past empirical work has found that deliberation can increase empathy and trust between participants as individuals are exposed to the lived experiences and perspectives of others.
Outcome to test Hypothesis 2 : Deliberation Increases Social Cohesion
Outcome:
To what extent do you trust the opinions of other badgeholders?
Scale:
0 (very low trust) ... 4 ... 7 (very high trust)
i. Participants had higher trust in the opinions of other badgeholders post-deliberation compared to pre-deliberation
Paired T-test Results
MeanPre-deliberation
4 . 43
MeanPost-deliberation
5 . 13
Mean Difference
+ 0 . 7 ***
Standard Error
( 0 . 17 )
t-statistic
4 . 04
Degrees of Freedom
23
Comparing pre/ post-survey responses, participants increased about 0 . 7 on a 7 -point scale measuring their trust in other badgeholders. With a p-value of 0 . 001 this is a statistically significant shift in opinions among participants, and substantively moving nearly 1 point on a 7 -point scale is quite meaningful. Thus, we find strong evidence that those who participated in the deliberative process significantly increased levels of trust in the opinions of other badgeholders.
ii. Participants had higher post-deliberation trust than non-participants
Effect of Deliberation on Trust in Other Badgeholders (IV Estimation)
Variable
Estimate
Std. Error
t statistic
(Intercept)
4 . 839
0 . 151
32 . 054
Deliberation participation
0 . 456
0 . 423
1 . 078
Adjusted R-Squared
- 0 . 009
Wald Test
1 . 162 on 1 and 103 DF
Similar to above, we used IV estimation to account for potential selection bias given the opt-in nature of the treatment. We see there is about a 0 . 5 increase in trust in other badgeholders (fairly substantive on a 7 -point scale) but again, the relatively large standard errors suggest there’s some noise in the data (probably also due to the fact that n = 24 is a relatively small sample size).
( 3 ) Deliberation increases support for deliberative process itself
Theory: Participants in deliberative processes will come to view deliberation as a legitimate and effective way to make decisions, meaning participants gain trust and appreciation for the deliberative process itself.
i. Among participants, 22 out of 24 ( 92 %) were supportive of a deliberative process being useful for additional topics and even bigger settings
The table below summarizes responses to the question: “Do you think a deliberative process could be useful for additional contested topics and even bigger settings?” We see that the overwhelming majority of participants were supportive of deliberation as an effective governance tool.
delib_supportive 1584 × 1168 96 . 9 KB
Qualitative data source: This data comes from a free response question on the post-deliberation survey administered to participants.
ii. For both participants and non-participants alike, and for those for and against ratifying the definition of profit, there were consistent themes of positive support toward deliberation as an effective tool in the decision-making process
delib_ratify_discussion 1264 × 938 105 KB
Qualitative data source: This data comes from governance forum discussion comments, and ratification voting data on Snapshot.
( 4 ) Deliberation moved participants toward consensus on only one of four categories
Theory: Structured discussions can lead to significant shifts in opinion and create more nuanced, consensus-based decisions because deliberation creates an environment which participants can engage in with diverse viewpoints and can help bridge ideological divides.
Outcome to test Hypothesis 4 : Deliberation Increases Consensus
Outcome:
Which of the following types of external funding do you think should be deducted from public goods funding rewards?
Select all that apply:
Grants from Optimism
VC funding
Grants from other ecosystems
Revenue
NONE of these types of funding should be subtracted from the total amount of Retro Rewards
Other (enter here)
The tables below show the aggregate pre/ post- survey results for the question about which types of external funding should be deducted for the 24 badgeholders who participated in both deliberative processes. For the categories of VC Funding, OP Grants, and Other Grants, there is no strong evidence of consensus post-deliberation. In the case of VC Funding, the majority vote switched in favor of being “against deducting” but remained at a close margin. For the category of Revenue, it does appear that opinions became more aligned, with only about 20 % of badgeholders in favor of deducting post-deliberation, compared to 46 % being in favor of deducting pre-deliberation. The pairwise t-test results for changes in opinion on deducting Revenue also confirm that this is a statistically significant shift. There is no statistically significant shift in the other three categories.
Support for Deducting External Funding (Pre-deliberation):
VC Funding
OP Grants
Grants
Revenue
In Favor of Deducting
14
14
8
11
Against Deducting
10
10
16
13
Total
24
24
24
24
Support for Deducting External Funding (Post-deliberation):
VC Funding
OP Grants
Grants
Revenue
In Favor of Deducting
11
15
6
5
Against Deducting
13
9
18
19
Total
24
24
24
24
What’s Unique about the Optimism Case?
There are some things unique to Optimism’s deliberative process that should be taken into account when interpreting these findings:
i. Highly engaged population
This deliberative process was conducted among Optimism’s badgeholders, a group of highly engaged governance stakeholders who skew technical in their professions and are based in approximately 20 different countries. Of the 50 badgeholders randomly selected to participate, the 25 who showed up were likely especially engaged as they made time to participate in two 90 -minute deliberative sessions (and in most cases, attended a third optional 90 -minute deliberative session) in addition to completing the pre-reading before each session. In some cases, participants joined despite being based in inconvenient time zones (e.g., joining from Asia where the local time was 2 am). Throughout the process, participants repeatedly requested additional pre-reading homework and asked for a third deliberative session.
ii. Direct policy implications
Another unique feature of this deliberative process is the fact that, pending ratification from the full set of badgeholders, the proposed definition of profit decided upon during the deliberative process would be directly implemented into the upcoming Retro Funding Round 4 . This is distinct from deliberative processes in other contexts that have been criticized for being performative, or at best advisory. Optimism, on the other hand, is a governance structure that progressively decentralizes over time, meaning badgeholders held immediate decision-making power in Retro Funding Round 4 .
iii. Lack of clarity on the topic
Throughout the process, it became clear that the topic of whether external funding should be deducted from public goods funding rewards was a question that also contains value judgements, whereas the resulting policy was meant to focus on the practical implementation details of this question. For example, a discussion about whether VC funding should be deducted from public goods funding is different than a discussion about whether VC funding could be deducted given limitations in what funding data is available for projects and whether it’s possible to deduct accurately. There were also time constraints given that the new definition of profit needed to be ratified before the upcoming Retro Funding Round 4 .
Recapping Key Takeaways
In sum, the key takeaways from this analysis are:
( 1 ) Running another deliberative process could help to increase participants’ knowledge on the arguments for/ against the deliberated topic:
There’s strong evidence that those who participated in the deliberative process became more knowledgeable about the topic, with badgeholders reporting a significant increase in their understanding of the arguments for and against the topic after engaging in the deliberative process.
Comparing participants to non-participants, deliberative process participants tended to report higher knowledge than non-participants after the deliberative process, though whether deliberation alone is driving this difference is less certain (likely due to the small sample size of 24 participants, or perhaps spillover effects from the deliberative process also increased non-participant knowledge which makes the post-deliberation difference less meaningful)
( 2 ) Running another deliberative process could help to increase participants’ trust in the opinions of fellow governance participants:
There’s strong evidence that participating in the deliberative process led badgeholders to trust each others’ opinions more, with badgeholders reporting a significant increase in trust in the opinions of other badgeholders after engaging in the deliberative process.
Comparing participants to non-participants, deliberative process participants tended to report higher trust than non-participants after the deliberative process, though again, whether deliberation alone is driving this difference is less certain (likely due to the small sample size of 24 participants)
( 3 ) Running the deliberative process seemed to encourage support for the deliberative process itself as an effective decision-making tool
This support for a deliberative process as a useful governance tool came from participants and the broader population of non-participants
This support came from those who voted in favor and those who voted against the new definition of profit that was decided during the deliberative process
( 4 ) Participating in the deliberative process only seemed to increase consensus on one of four categories under the deliberated topic
This suggests that deliberation may foster agreement on specific aspects of a topic (perhaps through increased trust and information exchange, as documented above) though not universally across all categories, so this question probably merits further study
This finding is likely also influenced by the practical constraints and lack of clarity surrounding the topic of the deliberative process, and the fact that the category (Revenue) which did see movement toward consensus is less polarizing and more clearly scoped than other categories (e.g., moral aversion to VCs, or lack of clarity on what is considered an OP Grant, etc.)
What This Means for Optimism Governance
We found evidence that deliberation can be a tool for knowledge exchange and increasing social cohesion, so this might make sense in contexts where comprehension of complex topics, or where bridging a divide between parties with low levels of trust, is important. Specifically, our findings suggest that a deliberative process might be useful for improving comprehension of complex documents like the Law of Chains or Blockspace Charters. A deliberative process could also be useful as a tool to facilitate conversation between the Token House and Citizens House in a way that improves trust in the other House and thereby reduces gridlock.
We found no strong evidence that deliberation shifted views toward consensus on the overall topic deliberated, suggesting that this type of deliberation tied to a binding policy recommendation may be less useful for the most polarizing topics, or for philosophical values questions which are difficult to coerce into a measurement question. With this in mind, we believe further exploration and study is warranted as there were unique considerations related to the specific topic that was discussed and which may have impacted outcomes. There are no immediate plans to implement another deliberative process experiment but we may explore this as appropriate.
Finally, we are very grateful to our facilitators who made this deliberative process happen: @Antoine Vergne (Missions Publiques), Inês NW (Inês NW), and Andrea Gallagher (RnDAO).
Thanks for reading and would love to hear any reactions in the comments below!
Likes: 10
Replies: 1
Replies:
- ccerv1: Fascinating read, thank you for sharing!
elizaoak:
suggesting that this type of deliberation tied to a binding policy recommendation may be less useful for the most polarizing topics, or for philosophical values questions which are difficult to coerce into a measurement question
I often wonder how to determine what the most polarizing issues are among citizens, and then to get a sense of how/if people’s views shift over time. This seems to be at the heart of figuring out the impact equation.
-
Optimism Governance Survey - We Want to Hear from You!
by elizaoak - This user is a moderator
Posted on: Oct. 25, 2024, 12:32 p.m.
Content: To clarify, this survey is not being conducted by the Foundation.
Likes: 2
Replies: 0
No replies yet.
-
Retro Funding 4: Deliberative process on the definition of profit
by elizaoak - This user is a moderator
Posted on: May 30, 2024, 10:14 a.m.
Content: Gm! Reminder that Session 1 (of 2 ) of the Deliberative Process is happening today. If you have been selected (via lottery) to participate, you should have received preparatory information via email.
If you were not selected to participate, please stay tuned for a summary of Session 1 on the forum in the coming days. We’ll invite you to share your reactions to the decision from Session 1 , including completing a brief survey that the Deliberative Process participants will take into consideration during Session 2 next week.
After Session 2 , you should plan to vote in Special Voting Cycle # 23 b (June 13 - 19 ) to ratify the definition of profit decided by the Deliberative Process.
Likes: 0
Replies: 0
No likes yet.
No replies yet.
-
Retro Funding 4: Deliberative process on the definition of profit
by elizaoak - This user is a moderator
Posted on: May 13, 2024, 2:51 p.m.
Content: @joanbp great questions –
Badgeholders who were randomly selected to participate were contacted via email on May 9 . They do not have to participate and can opt out if preferred (though we hope they will join!). If we need to fill more spots we will randomly sample another batch at a later date.
Likes: 2
Replies: 0
No replies yet.
-
Did OP Airdrop 2 Increase Governance Engagement?
by elizaoak - This user is a moderator
Posted on: Dec. 20, 2023, 1:18 a.m.
Content: & Big thanks to Daren Matsuoka for data help, and @bobby and @MSilb 7 for helpful discussion! This work builds on earlier Dune analyses from arabianhorses, oplabspbc, and springzhang.
Likes: 3
Replies: 0
No replies yet.
-
Did OP Airdrop 2 Increase Governance Engagement?
by elizaoak - This user is a moderator
Posted on: Dec. 20, 2023, 1:16 a.m.
Content: Did OP Airdrop 2 Increase Governance Engagement? An Academic Analysis by @andyhall & @elizaoak → Link to full draft of the academic paper here 35 Summary One of the most important questions for projects exploring decentralized governance is: How do we broaden participation to include more community members? One idea is to offer token rewards to encourage people to engage in governance. To study whether this mechanism works in-practice, we took advantage of airdrop # 2 ’s clever design and looked at the data for ~ 1 . 2 million OP users. We find that: ( 1 ) revealing airdrop 2 ’s reward scheme caused a large increase in token delegation, including first-time delegations ( 2 ) receiving a larger reward for past participation increases the rate at which addresses subsequently delegate their tokens and vote. This raises some interesting questions for the future: Is this cost effective? Do you need to change the activities rewarded in future airdrops? How would you decentralize this process in the future while maintaining its efficacy? Open Debates about the Effectiveness of Airdrop Incentives for Community Engagement Airdrops are one way to distribute token rewards to potentially incentivize participation, though there are ongoing debates about the success of airdrops. Some key challenges include the fact that addresses may cash out the rewards and exit; airdrops may incentivize mindless harvesting of rewards which is not helpful for long-term civic engagement; and people may not value token rewards to begin with. Dune Analytics co-founder, for instance, says that airdrops create rich quick “communities,” are an absurd spending of capital to farmers, and create no long term value for anyone. An analysis of Uniswap’s airdrop 2 suggests that most awardees immediately dumped and exited, leading the author to conclude that “the airdrop model seems to be broken.” But Optimism’s airdrop 2 (and 3 , for that matter) is unique in that it specifically rewarded active governance participation. More generally, Optimism has been pioneering novel governance and public goods initiatives over the past few years – so there’s reason to think this setting might be a particularly interesting case to study incentives for prosocial behavior. We Exploit the Natural Experiment in Optimism’s Airdrop 2 to Study Causal Effects of Rewards on Participation On Feb 9 , 2023 Optimism distributed 11 . 7 million OP tokens to over 300 , 000 unique addresses. Unlike airdrop 1 , rewards were sent directly to addresses so there was no need to claim. Airdrop 2 ’s reward function consisted of A governance delegation reward ( 6 . 8 million OP sent to 57 , 204 qualifying addresses) A gas usage reward ( 2 . 5 million OP sent to 280 . 057 eligible addresses) Four bonus attributes ( 161 , 759 eligible addresses). Importantly, OP users knew an airdrop was eventually coming but they didn’t know when and they didn’t know how rewards would be allocated and what behaviors would be rewarded. This is an important detail as it means we could exploit the quasi-randomness in our statistical analysis to examine causal effects. The quasi-randomness refers to the previously unknown, and somewhat arbitrary nature of the exact reward criteria cutoff (e.g. spent at least $ 6 . 10 in gas, or had delegated at least 2 , 000 delegate-days). One concern with teasing out the causal effects from mere correlation is that there might be some potential confounding, where higher rewards simply go to addresses already more inclined to participate. To address this we exploit another important detail in how the airdrop was designed, namely the fact that two of the three reward function components are non-governance related. This means that we can control for a wallet’s prior delegation behavior and examine the residual variation in rewards that mainly comes from gas usage and bonus categories. We Find Positive Effects of Airdrop 2 Rewards on Subsequent Voting and Delegation The tl;dr is that airdrop 2 did seem to increase subsequent governance engagement (both delegation and voting). Finding 1 : Noticeable Increase in Delegation After Announcement of Rewards Scheme on Feb 9 , 2023 op_img 1936 × 360 74 . 1 KB First, we see an increase in new wallets that got rewarded but didn’t previously delegate – but who began to delegate post-airdrop. In the figures above, the left plot shows the total amount of new OP tokens (logged) that were delegated each day. The vertical red dashed line represents the day of the Airdrop 2 announcement (Feb 9 , 2023 ), and indeed we see a noticeable spike in delegation activity in the immediate aftermath of the announcement. Because the daily data is inherently bouncy, we also present binned averages that pool across days in the right-hand plot. We estimate lines of best fit to either side of the announcement date. Finding 2 : Receiving Larger Rewards Leads to More Delegation, On Average – Especially Delegation to Other Addresses In addition to documenting the overall informational effects of the announcement on delegations, we also looked into whether addresses who received airdropped tokens due to past behavior subsequently committed these new tokens to governance, as opposed to “dumping” and exiting. We find a strong positive conditional relationship between the size of the rewards and the amount of delegation (in terms of delegating a higher amount for more days) after receiving the reward. Our analysis accounts for pre-treatment delegation, as discussed above in the section on overcoming potential confounding. Of course in Optimism, users have the choice of delegating their tokens to themselves and directly vote on proposals or to delegate to a representative delegate on their behalf. Delegation to others is arguably an important part of scalable online democracy as it allows users to participate without requiring the time and skills necessary to study upcoming proposals. We find that delegations to others increase more than delegations to self – and specifically, as reward size increases, addresses delegate to others much more often than to themselves. We also find that the effect of rewards on delegation is larger for addresses with smaller OP token balances prior to the airdrop, which suggests that the rewards scheme is broadening democracy to an extent, bringing in smaller token holders into the governance process. Finding 3 : Larger Rewards Lead to More Voting Finally, we explored the effects of airdrop 2 rewards on the act of voting. Airdrop 2 only explicitly rewarded delegation, not voting, so one concern at this point might be that we’ve merely documented delegation farming so far. However, we also see an increase in voting as well as delegations, suggesting there is a strong relationship between reward size and governance participation more broadly. What’s Unique About The OP Case? There are some unique features of OP’s airdrop 2 that might help explain these strong, positive effects. First, there’s a promise of future rewards that is likely driving these anticipatory effects – people believe Optimism will do similar airdrops in the future (and indeed, airdrop 3 rewarded delegation as well). This is a clever approach to offer rewards retroactively and promise future rewards, though this raises the question of whether discontinuing airdrops at some point in the future will reduce the effectiveness of the reward scheme, and in turn how to sustain sequential airdrop rewards over time. Second, it’s worth noting the low value of the OP token at the time of airdrop 2 , meaning the incentive to sell and exit was lower. The first point raises an interesting question for Optimism: How long can the sequence of rewards go on, and how many different behaviors can be rewarded productively over time? If, for example, delegation continues to be rewarded, this may incentivize adversarial rewards harvesters to delegate mindlessly and crowd out the prosocial behavior across the ecosystem. Optimism can seek to change what kinds of prosocial behaviors are rewarded, but there are only so many observable components to governance participation. We look forward to further discussion on these considerations in the comments section! Fortunately, there’s an open design space for addressing some of these potential concerns and continually iterating and experimenting with new ideas. In fact, in the time since we began analyzing airdrop 2 , airdrop 3 has already been implemented and improves on some of the critiques of airdrop 2 (eg., providing on average higher reward amounts). In future airdrops, it might be interesting to experiment with Incorporating non-transferable reputation, or social rewards (e.g., an “I Voted” ERC- 5114 token) to overcome concerns about mindless reward harvesting, or dumping and exiting. Conclusion To summarize, there are two big takeaways from our analysis: Announcing an airdrop that explicitly rewards a certain type of governance participation (i.e., delegation) causes many addresses to delegate further in anticipation of future rewards. The particular addresses that received bigger rewards for past delegation reacted by delegating more – this could be some combination of anticipating future rewards as well as some deeper sense of ownership of the project. We think a large part of what is driving these positive findings is the fact that there is 1 ) a promise of future rewards (this is not a one-off instance), 2 ) the tokens give people a larger stake in the project (via governance power), 3 ) there’s a broader ecosystem emphasis and culture of promoting public goods and prosocial behavior, 4 ) the quasi-random, retroactive reward criteria which made this harder to game, and 5 ) focusing specifically on rewarding past governance behavior in hopes of incentivizing future governance behavior. These characteristics might serve as a model for other projects seeking to successfully design airdrop incentives for community engagement. Finally, this raises some interesting questions for the future: Is this cost effective? We know that giving out token rewards in airdrop 2 led to more delegation, but only time will tell if the amount of rewards distributed maps to meaningful, sustained governance participation. Do you need to change the activities rewarded in future airdrops? We know that airdrop 2 successfully created incentives for people to delegate more tokens looking ahead to the future, but if all future airdrops continue to reward the same action, will it encourage mindless delegating that doesn’t translate into meaningful governance participation? How would you decentralize this process in the future while maintaining its efficacy? We know that the retrospective, surprising aspect of the airdrop 2 announcement was able to change behavior and encourage delegation, but in the future, we will want to be able to create similar incentives without having to rely on a central actor to design the airdrop and announce it.
Likes: 20
Replies: 0
Likers:
joaokury,
msilb7,
zcf,
chain_l,
fractalvisions,
raho,
danftz,
griptoe666,
jeremylau,
gitcoin,
milo,
boardroom,
akura,
nkosearas,
cryptoreumd,
gonna.eth,
guiriba,
pumbi,
sude,
sixty
No replies yet.
-
Did OP Airdrop 2 Increase Governance Engagement?
by elizaoak - This user is a moderator
Posted on: Dec. 19, 2023, 8:18 p.m.
Content: & Big thanks to Daren Matsuoka for data help, and @bobby and @MSilb 7 for helpful discussion! This work builds on earlier Dune analyses from arabianhorses, oplabspbc, and springzhang.
Likes: 3
Replies: 0
No replies yet.
-
Did OP Airdrop 2 Increase Governance Engagement?
by elizaoak - This user is a moderator
Posted on: Dec. 19, 2023, 8:16 p.m.
Content: Did OP Airdrop 2 Increase Governance Engagement? An Academic Analysis by @andyhall & @elizaoak
→ Link to full draft of the academic paper here 38
Summary
One of the most important questions for projects exploring decentralized governance is: How do we broaden participation to include more community members? One idea is to offer token rewards to encourage people to engage in governance. To study whether this mechanism works in-practice, we took advantage of airdrop # 2 ’s clever design and looked at the data for ~ 1 . 2 million OP users.
We find that:
( 1 ) revealing airdrop 2 ’s reward scheme caused a large increase in token delegation, including first-time delegations
( 2 ) receiving a larger reward for past participation increases the rate at which addresses subsequently delegate their tokens and vote.
This raises some interesting questions for the future:
Is this cost effective?
Do you need to change the activities rewarded in future airdrops?
How would you decentralize this process in the future while maintaining its efficacy?
Open Debates about the Effectiveness of Airdrop Incentives for Community Engagement
Airdrops are one way to distribute token rewards to potentially incentivize participation, though there are ongoing debates about the success of airdrops. Some key challenges include the fact that addresses may cash out the rewards and exit; airdrops may incentivize mindless harvesting of rewards which is not helpful for long-term civic engagement; and people may not value token rewards to begin with. Dune Analytics co-founder, for instance, says that airdrops create rich quick “communities,” are an absurd spending of capital to farmers, and create no long term value for anyone. An analysis of Uniswap’s airdrop 4 suggests that most awardees immediately dumped and exited, leading the author to conclude that “the airdrop model seems to be broken.”
But Optimism’s airdrop 2 (and 3 , for that matter) is unique in that it specifically rewarded active governance participation. More generally, Optimism has been pioneering novel governance and public goods initiatives over the past few years – so there’s reason to think this setting might be a particularly interesting case to study incentives for prosocial behavior.
We Exploit the Natural Experiment in Optimism’s Airdrop 2 to Study Causal Effects of Rewards on Participation
On Feb 9 , 2023 Optimism distributed 11 . 7 million OP tokens to over 300 , 000 unique addresses. Unlike airdrop 1 , rewards were sent directly to addresses so there was no need to claim. Airdrop 2 ’s reward function consisted of
A governance delegation reward ( 6 . 8 million OP sent to 57 , 204 qualifying addresses)
A gas usage reward ( 2 . 5 million OP sent to 280 . 057 eligible addresses)
Four bonus attributes ( 161 , 759 eligible addresses).
Importantly, OP users knew an airdrop was eventually coming but they didn’t know when and they didn’t know how rewards would be allocated and what behaviors would be rewarded. This is an important detail as it means we could exploit the quasi-randomness in our statistical analysis to examine causal effects. The quasi-randomness refers to the previously unknown, and somewhat arbitrary nature of the exact reward criteria cutoff (e.g. spent at least $ 6 . 10 in gas, or had delegated at least 2 , 000 delegate-days).
One concern with teasing out the causal effects from mere correlation is that there might be some potential confounding, where higher rewards simply go to addresses already more inclined to participate. To address this we exploit another important detail in how the airdrop was designed, namely the fact that two of the three reward function components are non-governance related. This means that we can control for a wallet’s prior delegation behavior and examine the residual variation in rewards that mainly comes from gas usage and bonus categories.
We Find Positive Effects of Airdrop 2 Rewards on Subsequent Voting and Delegation
The tl;dr is that airdrop 2 did seem to increase subsequent governance engagement (both delegation and voting).
Finding 1 : Noticeable Increase in Delegation After Announcement of Rewards Scheme on Feb 9 , 2023
op_img 1936 × 360 74 . 1 KB
First, we see an increase in new wallets that got rewarded but didn’t previously delegate – but who began to delegate post-airdrop. In the figures above, the left plot shows the total amount of new OP tokens (logged) that were delegated each day. The vertical red dashed line represents the day of the Airdrop 2 announcement (Feb 9 , 2023 ), and indeed we see a noticeable spike in delegation activity in the immediate aftermath of the announcement. Because the daily data is inherently bouncy, we also present binned averages that pool across days in the right-hand plot. We estimate lines of best fit to either side of the announcement date.
Finding 2 : Receiving Larger Rewards Leads to More Delegation, On Average – Especially Delegation to Other Addresses
In addition to documenting the overall informational effects of the announcement on delegations, we also looked into whether addresses who received airdropped tokens due to past behavior subsequently committed these new tokens to governance, as opposed to “dumping” and exiting. We find a strong positive conditional relationship between the size of the rewards and the amount of delegation (in terms of delegating a higher amount for more days) after receiving the reward. Our analysis accounts for pre-treatment delegation, as discussed above in the section on overcoming potential confounding.
Of course in Optimism, users have the choice of delegating their tokens to themselves and directly vote on proposals or to delegate to a representative delegate on their behalf. Delegation to others is arguably an important part of scalable online democracy as it allows users to participate without requiring the time and skills necessary to study upcoming proposals. We find that delegations to others increase more than delegations to self – and specifically, as reward size increases, addresses delegate to others much more often than to themselves. We also find that the effect of rewards on delegation is larger for addresses with smaller OP token balances prior to the airdrop, which suggests that the rewards scheme is broadening democracy to an extent, bringing in smaller token holders into the governance process.
Finding 3 : Larger Rewards Lead to More Voting
Finally, we explored the effects of airdrop 2 rewards on the act of voting. Airdrop 2 only explicitly rewarded delegation, not voting, so one concern at this point might be that we’ve merely documented delegation farming so far. However, we also see an increase in voting as well as delegations, suggesting there is a strong relationship between reward size and governance participation more broadly.
What’s Unique About The OP Case?
There are some unique features of OP’s airdrop 2 that might help explain these strong, positive effects. First, there’s a promise of future rewards that is likely driving these anticipatory effects – people believe Optimism will do similar airdrops in the future (and indeed, airdrop 3 rewarded delegation as well). This is a clever approach to offer rewards retroactively and promise future rewards, though this raises the question of whether discontinuing airdrops at some point in the future will reduce the effectiveness of the reward scheme, and in turn how to sustain sequential airdrop rewards over time. Second, it’s worth noting the low value of the OP token at the time of airdrop 2 , meaning the incentive to sell and exit was lower.
The first point raises an interesting question for Optimism: How long can the sequence of rewards go on, and how many different behaviors can be rewarded productively over time? If, for example, delegation continues to be rewarded, this may incentivize adversarial rewards harvesters to delegate mindlessly and crowd out the prosocial behavior across the ecosystem. Optimism can seek to change what kinds of prosocial behaviors are rewarded, but there are only so many observable components to governance participation. We look forward to further discussion on these considerations in the comments section!
Fortunately, there’s an open design space for addressing some of these potential concerns and continually iterating and experimenting with new ideas. In fact, in the time since we began analyzing airdrop 2 , airdrop 3 has already been implemented and improves on some of the critiques of airdrop 2 (eg., providing on average higher reward amounts). In future airdrops, it might be interesting to experiment with Incorporating non-transferable reputation, or social rewards (e.g., an “I Voted” ERC- 5114 token) to overcome concerns about mindless reward harvesting, or dumping and exiting.
Conclusion
To summarize, there are two big takeaways from our analysis:
Announcing an airdrop that explicitly rewards a certain type of governance participation (i.e., delegation) causes many addresses to delegate further in anticipation of future rewards.
The particular addresses that received bigger rewards for past delegation reacted by delegating more – this could be some combination of anticipating future rewards as well as some deeper sense of ownership of the project.
We think a large part of what is driving these positive findings is the fact that there is 1 ) a promise of future rewards (this is not a one-off instance), 2 ) the tokens give people a larger stake in the project (via governance power), 3 ) there’s a broader ecosystem emphasis and culture of promoting public goods and prosocial behavior, 4 ) the quasi-random, retroactive reward criteria which made this harder to game, and 5 ) focusing specifically on rewarding past governance behavior in hopes of incentivizing future governance behavior. These characteristics might serve as a model for other projects seeking to successfully design airdrop incentives for community engagement.
Finally, this raises some interesting questions for the future:
Is this cost effective? We know that giving out token rewards in airdrop 2 led to more delegation, but only time will tell if the amount of rewards distributed maps to meaningful, sustained governance participation.
Do you need to change the activities rewarded in future airdrops? We know that airdrop 2 successfully created incentives for people to delegate more tokens looking ahead to the future, but if all future airdrops continue to reward the same action, will it encourage mindless delegating that doesn’t translate into meaningful governance participation?
How would you decentralize this process in the future while maintaining its efficacy? We know that the retrospective, surprising aspect of the airdrop 2 announcement was able to change behavior and encourage delegation, but in the future, we will want to be able to create similar incentives without having to rely on a central actor to design the airdrop and announce it.
Likes: 20
Replies: 0
Likers:
JoaoKury,
MSilb7,
zcf,
Chain_L,
FractalVisions,
raho,
danftz,
griptoe666,
Jeremylau,
Gitcoin,
Milo,
Boardroom,
akura,
nkosearas,
CryptoReuMD,
Gonna.eth,
Guiriba,
Pumbi,
sude,
Sixty
No replies yet.
-
Did OP Airdrop 2 Increase Governance Engagement?
by elizaoak - This user is a moderator
Posted on: Dec. 19, 2023, 8:16 p.m.
Content: Did OP Airdrop 2 Increase Governance Engagement? An Academic Analysis by @andyhall & @elizaoak
→ Link to full draft of the academic paper here
Summary
One of the most important questions for projects exploring decentralized governance is: How do we broaden participation to include more community members? One idea is to offer token rewards to encourage people to engage in governance. To study whether this mechanism works in-practice, we took advantage of airdrop # 2 ’s clever design and looked at the data for ~ 1 . 2 million OP users.
We find that:
( 1 ) revealing airdrop 2 ’s reward scheme caused a large increase in token delegation, including first-time delegations
( 2 ) receiving a larger reward for past participation increases the rate at which addresses subsequently delegate their tokens and vote.
This raises some interesting questions for the future:
Is this cost effective?
Do you need to change the activities rewarded in future airdrops?
How would you decentralize this process in the future while maintaining its efficacy?
Open Debates about the Effectiveness of Airdrop Incentives for Community Engagement
Airdrops are one way to distribute token rewards to potentially incentivize participation, though there are ongoing debates about the success of airdrops. Some key challenges include the fact that addresses may cash out the rewards and exit; airdrops may incentivize mindless harvesting of rewards which is not helpful for long-term civic engagement; and people may not value token rewards to begin with. Dune Analytics co-founder, for instance, says that airdrops create rich quick “communities,” are an absurd spending of capital to farmers, and create no long term value for anyone. An analysis of Uniswap’s airdrop suggests that most awardees immediately dumped and exited, leading the author to conclude that “the airdrop model seems to be broken.”
But Optimism’s airdrop 2 (and 3 , for that matter) is unique in that it specifically rewarded active governance participation. More generally, Optimism has been pioneering novel governance and public goods initiatives over the past few years – so there’s reason to think this setting might be a particularly interesting case to study incentives for prosocial behavior.
We Exploit the Natural Experiment in Optimism’s Airdrop 2 to Study Causal Effects of Rewards on Participation
On Feb 9 , 2023 Optimism distributed 11 . 7 million OP tokens to over 300 , 000 unique addresses. Unlike airdrop 1 , rewards were sent directly to addresses so there was no need to claim. Airdrop 2 ’s reward function consisted of
A governance delegation reward ( 6 . 8 million OP sent to 57 , 204 qualifying addresses)
A gas usage reward ( 2 . 5 million OP sent to 280 . 057 eligible addresses)
Four bonus attributes ( 161 , 759 eligible addresses).
Importantly, OP users knew an airdrop was eventually coming but they didn’t know when and they didn’t know how rewards would be allocated and what behaviors would be rewarded. This is an important detail as it means we could exploit the quasi-randomness in our statistical analysis to examine causal effects. The quasi-randomness refers to the previously unknown, and somewhat arbitrary nature of the exact reward criteria cutoff (e.g. spent at least $ 6 . 10 in gas, or had delegated at least 2 , 000 delegate-days).
One concern with teasing out the causal effects from mere correlation is that there might be some potential confounding, where higher rewards simply go to addresses already more inclined to participate. To address this we exploit another important detail in how the airdrop was designed, namely the fact that two of the three reward function components are non-governance related. This means that we can control for a wallet’s prior delegation behavior and examine the residual variation in rewards that mainly comes from gas usage and bonus categories.
We Find Positive Effects of Airdrop 2 Rewards on Subsequent Voting and Delegation
The tl;dr is that airdrop 2 did seem to increase subsequent governance engagement (both delegation and voting).
Finding 1 : Noticeable Increase in Delegation After Announcement of Rewards Scheme on Feb 9 , 2023
op_img 1936 × 360 74 . 1 KB
First, we see an increase in new wallets that got rewarded but didn’t previously delegate – but who began to delegate post-airdrop. In the figures above, the left plot shows the total amount of new OP tokens (logged) that were delegated each day. The vertical red dashed line represents the day of the Airdrop 2 announcement (Feb 9 , 2023 ), and indeed we see a noticeable spike in delegation activity in the immediate aftermath of the announcement. Because the daily data is inherently bouncy, we also present binned averages that pool across days in the right-hand plot. We estimate lines of best fit to either side of the announcement date.
Finding 2 : Receiving Larger Rewards Leads to More Delegation, On Average – Especially Delegation to Other Addresses
In addition to documenting the overall informational effects of the announcement on delegations, we also looked into whether addresses who received airdropped tokens due to past behavior subsequently committed these new tokens to governance, as opposed to “dumping” and exiting. We find a strong positive conditional relationship between the size of the rewards and the amount of delegation (in terms of delegating a higher amount for more days) after receiving the reward. Our analysis accounts for pre-treatment delegation, as discussed above in the section on overcoming potential confounding.
Of course in Optimism, users have the choice of delegating their tokens to themselves and directly vote on proposals or to delegate to a representative delegate on their behalf. Delegation to others is arguably an important part of scalable online democracy as it allows users to participate without requiring the time and skills necessary to study upcoming proposals. We find that delegations to others increase more than delegations to self – and specifically, as reward size increases, addresses delegate to others much more often than to themselves. We also find that the effect of rewards on delegation is larger for addresses with smaller OP token balances prior to the airdrop, which suggests that the rewards scheme is broadening democracy to an extent, bringing in smaller token holders into the governance process.
Finding 3 : Larger Rewards Lead to More Voting
Finally, we explored the effects of airdrop 2 rewards on the act of voting. Airdrop 2 only explicitly rewarded delegation, not voting, so one concern at this point might be that we’ve merely documented delegation farming so far. However, we also see an increase in voting as well as delegations, suggesting there is a strong relationship between reward size and governance participation more broadly.
What’s Unique About The OP Case?
There are some unique features of OP’s airdrop 2 that might help explain these strong, positive effects. First, there’s a promise of future rewards that is likely driving these anticipatory effects – people believe Optimism will do similar airdrops in the future (and indeed, airdrop 3 rewarded delegation as well). This is a clever approach to offer rewards retroactively and promise future rewards, though this raises the question of whether discontinuing airdrops at some point in the future will reduce the effectiveness of the reward scheme, and in turn how to sustain sequential airdrop rewards over time. Second, it’s worth noting the low value of the OP token at the time of airdrop 2 , meaning the incentive to sell and exit was lower.
The first point raises an interesting question for Optimism: How long can the sequence of rewards go on, and how many different behaviors can be rewarded productively over time? If, for example, delegation continues to be rewarded, this may incentivize adversarial rewards harvesters to delegate mindlessly and crowd out the prosocial behavior across the ecosystem. Optimism can seek to change what kinds of prosocial behaviors are rewarded, but there are only so many observable components to governance participation. We look forward to further discussion on these considerations in the comments section!
Fortunately, there’s an open design space for addressing some of these potential concerns and continually iterating and experimenting with new ideas. In fact, in the time since we began analyzing airdrop 2 , airdrop 3 has already been implemented and improves on some of the critiques of airdrop 2 (eg., providing on average higher reward amounts). In future airdrops, it might be interesting to experiment with Incorporating non-transferable reputation, or social rewards (e.g., an “I Voted” ERC- 5114 token) to overcome concerns about mindless reward harvesting, or dumping and exiting.
Conclusion
To summarize, there are two big takeaways from our analysis:
Announcing an airdrop that explicitly rewards a certain type of governance participation (i.e., delegation) causes many addresses to delegate further in anticipation of future rewards.
The particular addresses that received bigger rewards for past delegation reacted by delegating more – this could be some combination of anticipating future rewards as well as some deeper sense of ownership of the project.
We think a large part of what is driving these positive findings is the fact that there is 1 ) a promise of future rewards (this is not a one-off instance), 2 ) the tokens give people a larger stake in the project (via governance power), 3 ) there’s a broader ecosystem emphasis and culture of promoting public goods and prosocial behavior, 4 ) the quasi-random, retroactive reward criteria which made this harder to game, and 5 ) focusing specifically on rewarding past governance behavior in hopes of incentivizing future governance behavior. These characteristics might serve as a model for other projects seeking to successfully design airdrop incentives for community engagement.
Finally, this raises some interesting questions for the future:
Is this cost effective? We know that giving out token rewards in airdrop 2 led to more delegation, but only time will tell if the amount of rewards distributed maps to meaningful, sustained governance participation.
Do you need to change the activities rewarded in future airdrops? We know that airdrop 2 successfully created incentives for people to delegate more tokens looking ahead to the future, but if all future airdrops continue to reward the same action, will it encourage mindless delegating that doesn’t translate into meaningful governance participation?
How would you decentralize this process in the future while maintaining its efficacy? We know that the retrospective, surprising aspect of the airdrop 2 announcement was able to change behavior and encourage delegation, but in the future, we will want to be able to create similar incentives without having to rely on a central actor to design the airdrop and announce it.
Likes: 20
Replies: 0
Likers:
JoaoKury,
MSilb7,
zcf,
Chain_L,
FractalVisions,
raho,
danftz,
griptoe666,
Jeremylau,
Gitcoin,
Milo,
Boardroom,
akura,
nkosearas,
CryptoReuMD,
Gonna.eth,
Guiriba,
Pumbi,
sude,
Sixty
No replies yet.
