Profile of ccerv1 in Optimism
Posts by ccerv1
-
When Is Deliberation Useful for Optimism Governance?
by ccerv1 - No Role
Posted on: Oct. 25, 2024, 8:41 p.m.
Content: Fascinating read, thank you for sharing!
elizaoak:
suggesting that this type of deliberation tied to a binding policy recommendation may be less useful for the most polarizing topics, or for philosophical values questions which are difficult to coerce into a measurement question
I often wonder how to determine what the most polarizing issues are among citizens, and then to get a sense of how/if people’s views shift over time. This seems to be at the heart of figuring out the impact equation.
Likes: 1
Replies: 0
No replies yet.
-
Analysis of Voting Data for Retro Funding 5
by ccerv1 - No Role
Posted on: Oct. 24, 2024, 5:15 p.m.
Content: For the last three RF rounds, my team at OSO has published a report that analyzes the voting data. We just published our report on RF 5 here 7 . Here is our report on RF 4 and RF 3 .
Key points:
This was the flattest reward distribution yet. The top project (geth) received 235 K OP. The median project received 93 K. Even the lowest ranked project received 37 K, which was more than 70 % of projects received in Round 4 . Basically, it was a good round for “average” projects.
The reward distribution was flat because … voters voted on flat distributions. There’s a lot of dataviz illustrating this point in the report. Have a look. There’s an important idealogical question about how voters feel about power law distributions, ie, is it a “good” outcome for a handful of projects to receive most of the retro funding.
Experts voted differently than non-experts. In this round, there was an expertise dimension 1 (as well as expert guest voters from the developer community). There are meaningful differences in how these groups allocated rewards and differentiated among projects. This raises important questions about the composition of voters in a given round
The report has lots of data visualizations and statistics. You can also view my data science notebooks if you’re curious about the methods used. I hope it helps you draw your own conclusions.
@SPop is running a RF retro workshop on 30 October and I’ll be sharing the analysis in that forum too. Feel free to come with questions or drop them below so I can try to answer them.
Likes: 10
Replies: 1
Replies:
- MinimalGravitas: ccerv1:
The top project (geth) received 235K OP. […] the lowest ranked project received 37K,
Wow, that really is flat, only about 6x between top and bottom. That’s especially surprising considering it was across 3 separate categories without crossover over allocators. Am I reading it right that for the OP Stack R&D the spread was less than 3x?
Loads of interesting data, raising lots of questions… why do experts dislike Nimbus, why do non-experts not value Protocol Guild so highly…?
Definitely agree that we should continue to consider Pairwise in future rounds, it felt helpful and it’s nice to see that it correlated with a difference to outcomes.
Anyway, great to see the data and read your analysis, thanks.
-
Analysis of Voting Data for Retro Funding 5
by ccerv1 - No Role
Posted on: Oct. 24, 2024, 5:15 p.m.
Content: For the last three RF rounds, my team at OSO has published a report that analyzes the voting data. We just published our report on RF 5 here. Here is our report on RF 4 and RF 3 .
Key points:
This was the flattest reward distribution yet. The top project (geth) received 235 K OP. The median project received 93 K. Even the lowest ranked project received 37 K, which was more than 70 % of projects received in Round 4 . Basically, it was a good round for “average” projects.
The reward distribution was flat because … voters voted on flat distributions. There’s a lot of dataviz illustrating this point in the report. Have a look. There’s an important idealogical question about how voters feel about power law distributions, ie, is it a “good” outcome for a handful of projects to receive most of the retro funding.
Experts voted differently than non-experts. In this round, there was an expertise dimension (as well as expert guest voters from the developer community). There are meaningful differences in how these groups allocated rewards and differentiated among projects. This raises important questions about the composition of voters in a given round
The report has lots of data visualizations and statistics. You can also view my data science notebooks if you’re curious about the methods used. I hope it helps you draw your own conclusions.
@SPop is running a RF retro workshop on 30 October and I’ll be sharing the analysis in that forum too. Feel free to come with questions or drop them below so I can try to answer them.
Likes: 10
Replies: 1
Replies:
- MinimalGravitas: ccerv1:
The top project (geth) received 235K OP. […] the lowest ranked project received 37K,
Wow, that really is flat, only about 6x between top and bottom. That’s especially surprising considering it was across 3 separate categories without crossover over allocators. Am I reading it right that for the OP Stack R&D the spread was less than 3x?
Loads of interesting data, raising lots of questions… why do experts dislike Nimbus, why do non-experts not value Protocol Guild so highly…?
Definitely agree that we should continue to consider Pairwise in future rounds, it felt helpful and it’s nice to see that it correlated with a difference to outcomes.
Anyway, great to see the data and read your analysis, thanks.
-
Create your own Developer Reputation Scores - OpenRank Web Tool Intro
by ccerv1 - No Role
Posted on: Oct. 15, 2024, 8:33 a.m.
Content: I just want to chime in that this is OSO’s second collaboration with OpenRank (the first was during RF 4 for the trusted user model). They have been extraordinary partners to work with. Google built a trillion dollar business on top of its ranking algorithm. Now you have the power to point OpenRank at any p 2 p graph you care about!
Likes: 2
Replies: 0
No replies yet.
-
Joan’s RPGF5 Reflections
by ccerv1 - No Role
Posted on: Oct. 15, 2024, 8:24 a.m.
Content:
joanbp:
In this round, OSO kindly provided github stats, which definitely work as useful heuristics, but a project could have many more contributors than just the people who commit things to github.
Hey Joan, we have been experimenting with different variants of “contributor”. Of course, the hard part is that there has to be some public proof of contribution. When it comes to GitHub contributions, we have versions that look at code contributions as well as non-code contributions (eg, opening issues, commenting on issues, etc). We are also experimenting with different versions based on pattern of activity (eg, someone who comments on an issue once is not the same as someone who is responding to issues regularly). Personally, I loved the collaboration with OpenRank (see Create your own Developer Reputation Scores - OpenRank Web Tool Intro as a way of creating a pool of the most trusted developers. Curious for your thoughts on this and ideas for future iterations!
Likes: 2
Replies: 1
Replies:
- joanbp: ccerv1:
Of course, the hard part is that there has to be some public proof of contribution.
Yeah. That’s a tough one.
On the one hand - not all kinds of contributions inherently come with any kind of public proof. Some contributions don’t even take place in public. That’s a fact of life.
On the other hand - we do need to be able to justify our decisions, and basing them on objective data is the least contentious way to do that.
I will give it some thought.
It feels to me like we should strive to make systems that can handle the complex reality; not reduce reality to what our systems can handle.
Maybe we can come up with ways of making objective data that speak to the things we can’t directly count and measure. Some kind of phenomenological approach, maybe.
In a way that’s central to what Impact Garden and Devouch do. We can’t always directly measure the usefulness of a project, but we can collect subjective statements from a lot of people and this then becomes our objective data: “These people said this or that under these conditions, and here is what we know about each of them…”
The github contributor count and similar objective data points taken from the public internet are a great place to start! But there must be ways to
create objective data from the subjective (or just not inherently public) parts of reality
and
make systems that can acknowledge the fact that important parts of life take place in ways that are not visible to the algorithm - but relevant to it. Possibly this is where human citizens will truly get a chance to make themselves indispensable. Because we see things that the algorithm doesn’t, and they matter to us. And if we are smart, we make the algorithm care about what matters to us.
I know. That was more philosophy than data science. But it’s just a starting point. Like the github contributor count. Your words made me think. I’ll think some more.
-
Joan’s RPGF5 Reflections
by ccerv1 - No Role
Posted on: Oct. 15, 2024, 8:24 a.m.
Content:
joanbp:
In this round, OSO kindly provided github stats, which definitely work as useful heuristics, but a project could have many more contributors than just the people who commit things to github.
Hey Joan, we have been experimenting with different variants of “contributor”. Of course, the hard part is that there has to be some public proof of contribution. When it comes to GitHub contributions, we have versions that look at code contributions as well as non-code contributions (eg, opening issues, commenting on issues, etc). We are also experimenting with different versions based on pattern of activity (eg, someone who comments on an issue once is not the same as someone who is responding to issues regularly). Personally, I loved the collaboration with OpenRank (see Create your own Developer Reputation Scores - OpenRank Web Tool Intro 3 as a way of creating a pool of the most trusted developers. Curious for your thoughts on this and ideas for future iterations!
Likes: 2
Replies: 1
Replies:
- joanbp: ccerv1:
Of course, the hard part is that there has to be some public proof of contribution.
Yeah. That’s a tough one.
On the one hand - not all kinds of contributions inherently come with any kind of public proof. Some contributions don’t even take place in public. That’s a fact of life.
On the other hand - we do need to be able to justify our decisions, and basing them on objective data is the least contentious way to do that.
I will give it some thought.
It feels to me like we should strive to make systems that can handle the complex reality; not reduce reality to what our systems can handle.
Maybe we can come up with ways of making objective data that speak to the things we can’t directly count and measure. Some kind of phenomenological approach, maybe.
In a way that’s central to what Impact Garden and Devouch do. We can’t always directly measure the usefulness of a project, but we can collect subjective statements from a lot of people and this then becomes our objective data: “These people said this or that under these conditions, and here is what we know about each of them…”
The github contributor count and similar objective data points taken from the public internet are a great place to start! But there must be ways to
create objective data from the subjective (or just not inherently public) parts of reality
and
make systems that can acknowledge the fact that important parts of life take place in ways that are not visible to the algorithm - but relevant to it. Possibly this is where human citizens will truly get a chance to make themselves indispensable. Because we see things that the algorithm doesn’t, and they matter to us. And if we are smart, we make the algorithm care about what matters to us.
I know. That was more philosophy than data science. But it’s just a starting point. Like the github contributor count. Your words made me think. I’ll think some more.
-
Retro Funding 5: Voting Rationale Thread
by ccerv1 - No Role
Posted on: Oct. 15, 2024, 8:14 a.m.
Content: Hi all
As a voter, I really enjoyed participating in this round. I liked being able to go deep into a specific category but also have a say on some of the other important round parameters. It was (mostly) painless, apart from some last minute fiddling to my ballot. Kudos to the teams that worked on the interfaces! The more intimate groupchat experience was also a plus for me.
(I will save my thoughts on the effectiveness of the round from an allocation perspective for a separate post.)
How I voted
I used my lil spreadsheet 5 to determine my overall budget and allocations. You can see my inputs, although I ended up tweaking things a little bit in the actual UI.
Ethereum Core Contributions
Many strong projects, although many also were well represented in RF 3 . I allocated 45 % here.
image 1886 × 568 98 . 8 KB
OP Stack R&D
Probably the most diverse category, but I strongly believe that high risk - high upside R&D initiatives should receive outsized rewards. I also allocated 45 % for this category even though it has fewer projects than Ethereum Core Contributions.
image 1888 × 566 98 . 7 KB
OP Stack Tooling
This was the weakest category IMHO (and also the one that I was assigned to vote on). I ended up allocating 10 %.
image 1868 × 572 93 . 4 KB
Here was my comment in the groupchat:
I have found Pairwise to be pretty useful in getting a sense of the mix of projects in our category and getting a sense of how I think they stack up against each other.
One observation is that our category has a decent number of “side project” type contributions, with multiple submissions linked to the same person.
RaaS providers definitely have an important impact on OP. However, I personally have mixed feelings about giving a large amount of RF for these types of projects since they already have strong business models and are not always strongly aligned with OP (or even ETH DA).
In terms of metrics, I think “trusted stars” is probably the more useful indicator for this category, esp for projects that are more documentation-focused. When I star a repo, it means I want to remember it.
Likes: 4
Replies: 0
No replies yet.
-
Retro Funding 5: Voting Rationale Thread
by ccerv1 - No Role
Posted on: Oct. 15, 2024, 8:14 a.m.
Content: Hi all
As a voter, I really enjoyed participating in this round. I liked being able to go deep into a specific category but also have a say on some of the other important round parameters. It was (mostly) painless, apart from some last minute fiddling to my ballot. Kudos to the teams that worked on the interfaces! The more intimate groupchat experience was also a plus for me.
(I will save my thoughts on the effectiveness of the round from an allocation perspective for a separate post.)
How I voted
I used my lil spreadsheet to determine my overall budget and allocations. You can see my inputs, although I ended up tweaking things a little bit in the actual UI.
Ethereum Core Contributions
Many strong projects, although many also were well represented in RF 3 . I allocated 45 % here.
image 1886 × 568 98 . 8 KB
OP Stack R&D
Probably the most diverse category, but I strongly believe that high risk - high upside R&D initiatives should receive outsized rewards. I also allocated 45 % for this category even though it has fewer projects than Ethereum Core Contributions.
image 1888 × 566 98 . 7 KB
OP Stack Tooling
This was the weakest category IMHO (and also the one that I was assigned to vote on). I ended up allocating 10 %.
image 1868 × 572 93 . 4 KB
Here was my comment in the groupchat:
I have found Pairwise to be pretty useful in getting a sense of the mix of projects in our category and getting a sense of how I think they stack up against each other.
One observation is that our category has a decent number of “side project” type contributions, with multiple submissions linked to the same person.
RaaS providers definitely have an important impact on OP. However, I personally have mixed feelings about giving a large amount of RF for these types of projects since they already have strong business models and are not always strongly aligned with OP (or even ETH DA).
In terms of metrics, I think “trusted stars” is probably the more useful indicator for this category, esp for projects that are more documentation-focused. When I star a repo, it means I want to remember it.
Likes: 4
Replies: 0
No replies yet.
-
Retro Funding 5: OP Stack - round details
by ccerv1 - No Role
Posted on: Sept. 30, 2024, 2:01 p.m.
Content: tl;dr - here is a Google Sheet 28 you can copy to help set the amount of rewards per category.
Read on if you want to know why this might be helpful to you.
Dear voters,
As you know, you will be casting three types of votes in Round 5 :
First, you’ll propose the budget. You’ll propose the budget for the entire round, up to a max of 8 M.
Then you’ll decide how much of that budget should go to each category, eg, 40 % Ethereum Core Contributions, 25 % OP Stack R&D, 35 % OP Stack Tooling.
Finally, you’ll score and allocate rewards to projects in one category randomly assigned to you.
While the third decision will likely take the most time to perform, the first and second decisions could be even more consequential in terms of determining how much funding projects receive from the round.
You will be presented with this initial screen:
image 852 × 1077 87 . 2 KB
Resist the temptation to simply allocate the default ⅓ to each category!
There are 79 total projects in the round and the most popular category (Ethereum Core Contributions) has 33 / 79 projects ( 42 %), whereas the least popular (OP Stack Tooling) has 20 / 79 projects ( 25 %). If you believe each category is equally important, then allocating 42 / 33 / 25 would be better than 34 / 33 / 33 .
But perhaps you believe some categories are more impactful than others. Or you think the top projects in some categories should receive much more than an average project. If that’s you, then good news, I’ve created a little tool to help you vote your values!
:point_right: Here it is: RF 5 Voting Simulator 28 :point_left:
The tool is nothing more than a humble spreadsheet. Copy it into your personal Google Drive and play with the assumptions.
image 1156 × 557 102 KB
For each category, you decide:
How many projects you think should receive retro funding
How much the top project in that category should receive
How much the median project should receive
How much the bottom project (that receives any OP) should receive
If you like power law or linear distribution curves
After making your settings, the graph will update automatically and you’ll get a recommendation of what percentage to allocate to that category.
Repeat this process for all of the categories.
Behind the scenes, all that’s happening is some regression to try and fit a curve to your three data points. The fit probably won’t be perfect – the “modeled” reward sizes give you a sense of how close you are to your “target” rewards. You’ll probably need to play with the numbers a bit to get the right approximation. And if you put weird numbers in you might get a curve that violates the rules of the round.
Of course, the eventual results will be a function of how everyone else votes. But hopefully this little tool can help you vote more optimally and save you some time. Enjoy!
Likes: 6
Replies: 0
No replies yet.
-
Impact Metrics for Retro Funding 5
by ccerv1 - No Role
Posted on: Sept. 27, 2024, 10:16 a.m.
Content: Open Source Observer is happy to support the latest retro round with another suite of impact metrics.
As RF 5 features OP Stack contributions, the metrics we’ve developed for the round are focused on GitHub repo-level contributions. They are intended to give human voters some helpful “data in the loop” as a complement to projects’ self-reported impact statements.
Metrics comes in three flavors:
Basic vanilla GitHub stats. We took a snapshot of projects’ GitHub repo stats (stars, forks, etc) at the end of the review process ( 2024 - 09 - 23 ). Nothing fancy here. You can double-check any of these by clicking on the project’s repo link.
Contributor counts. We indexed the event history for each repo to identify unique, non-bot contributors over the lifetime and the past 6 months of the project. We also present a count of “trusted” contributors.
Trust weighted metrics. With our friends at OpenRank 12 , we re-ran the “developer rank” and “repo rank” algorithms used to identify guest voters 7 over a new time interval. The algorithm considers contributions to over 40 K repos, seeded by a set of “high impact” repos identified by core devs. These trust-weighted metrics offer a better approximation of the impact a project has had specifically on the Optimism ecosystem versus the broader Ethereum / open source universe.
As always, you can see the source code and underlying data behind metrics here 29 , and a copy of the results as a CSV file here 29 .
The Metrics
The following fields are included in the GitHub-based metrics analysis for Retro Funding 5 :
artifact_url: Repository URL associated with the project based on the application. A project may have multiple repos in its application.
project_name: Name of the project.
project_category_id: Category ID of the project ( 1 , 2 , 3 ).
num_contributors: Total number of contributors before August 1 , 2024 . A contributor is defined as any non-bot user that has contributed to the repository (since 2015 ) by committing code directly to a repository, opening an issue, or opening/reviewing a pull request.
num_trusted_contributors: Number of trusted contributors before August 1 , 2024 . A subset of the contributors defined above, this is the number of contributors that are also in the top 420 of the OpenRank 12 developer trust score.
num_contributors_last_ 6 _months: Number of contributors over the period February 1 , 2024 - August 1 , 2024 .
num_stars: Total number of stars as of September 23 , 2024 .
num_trusted_stars: Number of stars from trusted users before August 1 , 2024 . A subset of the stars defined above, this is the number of stars from users that are also in the top 420 of the OpenRank 12 developer trust score.
trust_weighted_stars: This metric is a percentage score between 0 % and 100 %, representing the sum of the reputation share of the developers who starred the repo. If all developers in OpenRank developer ranking have starred a particular repo, the metric’s value is going to be 100 % for this particular repo. The more and the higher ranked developer who starred the repo, the higher the percentage value, the higher impact and quality of this repo. We calculate this metric by first calculating every developer’s reputation share (%) based on their OpenRank score, then sum them up if they starred the target repo.
num_forks: Total number of forks as of September 23 , 2024 .
num_trusted_forks: Number of forks from trusted users before August 1 , 2024 . A subset of the forks defined above, this is the number of forks from users that are also in the top 420 of the OpenRank 12 developer trust score.
trust_weighted_forks: This metric is a percentage score between 0 % and 100 %, representing the sum of the reputation share of the developers who forked the repo. If all developers in OpenRank developer ranking have forked a particular repo, the metric’s value is going to be 100 % for this particular repo. The more and the higher ranked developer who forked the target repo, the higher the percentage value, the higher impact and quality of this repo. We calculate this metric by first calculating every developer’s reputation share (%) based on their OpenRank score, then sum them up if they forked the target repo.
trust_rank_for_repo_in_category: Ranking of the repository’s OpenRank trust score within its category. A score of 1 indicates the highest ranking repo in its category.
age_of_project_years: Age of the project in years, measured from the project’s first public commit to August 1 , 2024 .
application_id: Application ID in the sign-up and voting UIs.
Caveats
Metrics are continuously evolving and never a substitute for DYOR as a voter. A few caveats:
The metrics come exclusively from public GitHub activity.
The majority - but not all - projects in the round submitted GitHub repos as part of their impact statements.
Metrics are presented only at the repo level (we don’t roll them up to the project level). Some projects submitted more than one repo and therefore have multiple sets of metrics.
The underlying data behind the metrics comes from gharchive 6 . We only capture contributions to public repos; we don’t capture any private contributions that happened before the repo turned public. This may cause minor differences between the metrics on the project’s voting page and the ones you see on the project’s actual GitHub.
Feedback or questions on our GitHub forum 7 are always welcome!
Likes: 8
Replies: 0
No replies yet.
