Opinion

LessWrong Shows You Social Signals Before the Comment

​When reading a comment, the first thing you see is what other people think. That design choice reduces your ability to form your own opinion and makes the site’s karma rankings less related to the comment’s true value. I think the problem is fixable and propose some ideas for consideration.The LessWrong interface prioritizes social informationYou read a comment. What information is presented, and in what order?The order of information:Who wrote the comment (in bold);How much other people like this comment (as shown by the karma indicator);How much other people agree with this comment (as shown by the agreement score);The actual content.This is unwise design for a website that emphasizes truth-seeking. You don’t have a chance to read the comment and form your own opinion first. However, you can opt in to hiding usernames (until moused over) via your account settings page. A 2013 RCT supports the upvote-anchoring concernFrom Social Influence Bias: A Randomized Experiment (Muchnik et al., 2013):[1]We therefore designed and analyzed a large-scale randomized experiment on a social news aggregation Web site to investigate whether knowledge of such aggregates distorts decision-making. Prior ratings created significant bias in individual rating behavior, and positive and negative social influences created asymmetric herding effects. Whereas negative social influence inspired users to correct manipulated ratings, [an initial upvote] increased the likelihood of positive ratings by 32% and created accumulating positive herding that increased final ratings by 25% on average.Inline reaction indicators also seem anchoringInline reactions are shown as little icons to the right of the line of text. Here’s an image of sidelined reactions to a comment of mine:I find these “reactions” distracting. They discourage people from forming independent opinions and probably have produced too much agreement with my comment.When I’m reading LessWrong content and see an icon on the side, the icon grabs my attention and distracts me from the content.I wonder “ooh, who reacted?” and I mouse over it and start thinking about social implications instead of actually reading the content.I am now anchored to agree or disagree with the content in question.In order to avoid people’s first impressions being anchored by these reactions, I sometimes redirect users from LessWrong to my website.Concrete proposalsLikely the biggest win. Hide karma and agreement indicators in the hour after a comment is posted. This would reduce the initial “luck” of someone strong-upvoting a comment, leading to a cascade of other positive votes due to anchoring. This effect is evidenced by Social Influence Bias: A Randomized Experiment (Science, 2013).Move the username, karma, and agreement indicators to the bottom of the comment (or post) by default. For short comments, hide the username and numbers until the comment has been in the viewport for X seconds.Provide account-level toggles for both (2) and (2a).Don’t show reactions until the user has reached the bottom of the comment, at which point the user can:Mouse over the reactions to see who reacted in response to which content;Scroll back up and see the reactions off to the side.A mock-up of how (2) might be implemented. This assumes that Matthew’s comment was not collapsed (and just ends as shown). A more modest (but still good) change would be to just move the agreement score to the bottom. These ideas aren’t perfect. For example, karma is genuinely useful for selecting which comments you’d like to read. By making the karma less prominent, it’s harder to skim for comments above a karma threshold. Consider two cases: The comment is not collapsed. In this case, while skimming the webpage, you can scroll down and just learn to look at the bottom of comments instead of the top. If the comment passes a threshold, read it by scrolling up slightly. This is mildly inconvenient.The comment is collapsed. Then the karma count isn’t visible at the bottom (since otherwise it’d be visible early on). This is a problem.The fix might be to modify proposal (2) to keep “karma” at the top of the comment but keep “username” and “agreement” at the bottom. I’m open to other ideas which do an even better job of minimizing costs and maximizing gains!(And to anyone about to type “this can’t be fixed”, have you spent five minutes (by the clock) thinking about the issue first?)Prior discussion and resultsIn 2021, Max Harms talked about Improving on the Karma System. His proposal focused on augmenting the entire system, not just the way karma is displayed. The LessWrong team has made changes in related areas. Total karma is deliberately not displayed prominently. Side-comments default to “just an icon” until you mouse over them. Karma used to be much more prominent at the top of posts, but now (on desktop) it’s a smaller number in the top right. These seem like good choices. In 2013, gwern shared the results of a highly relevant experiment. gwern followed the posts made by eight participating authors. gwern used an alternate account to randomly upvote or downvote the article and post a comment with a boilerplate rationalized “explanation” for the vote. For example: “downvoted, not enough math.” A month later, gwern came back to measure the total karma of the post. After controlling for an outlier popular post by Scott Alexander, the data indicated that an early up/downvote produced a non-statistically significant effect (with a difference-in-mean karma of about 10). However, as gwern notes, the sample size was small, so it wasn’t highly powered to begin with.While gwern’s experiment measured a proxy of the bandwagon-y-ness of LessWrong at that point in time, it measured how an initial comment affected the final karma of the post. Not how the visual prominence of karma- and agree-counts on a comment affected the final karma- and agree-counts of that comment. Related, but not quite the same. This is evidence against super strong versions of the effect (e.g. “most voters bandwagon off of the existing score”), but compatible with meaningful anchoring due to current design choices. Also, being able to see agreement right away can be a stronger effect than a single comment saying “upvoted” or “downvoted” (and a +1 or -1 to post karma), since the initially displayed agreement might be quite strong (e.g. +25). Seeing a tally of multiple votes all at once likely has a stronger impact on decision-making.Please show social signals after the comment!To me, the most valuable part of LessWrong was how it encouraged interesting contrarian comments. Many of us value truth-seeking, so I hope users and moderators optimize the website to better reflect that value.Inspired to finally share this critique due to Ryan Greenblatt’s comment arguing that more people should post on LW rather than X. Appendix: Filter listHere’s what I use in my Brave browser to filter out AF / LW karma and agree-votes. www.lesswrong.com##.Typography-root.Typography-headline.LWPostsPageTopHeaderVote-voteScorewww.lesswrong.com##.CommentsTableOfContents-commentKarmawww.lesswrong.com###36 mokQtNacRh56foNv > div > .CommentsItem-root.recent-comments-node > .CommentsItem-body > .CommentsItemMeta-root > .NamesAttachedReactionsVoteOnComment-root > .AgreementVoteAxis-agreementSection > .AgreementVoteAxis-agreementScore > .LWTooltip-root > spanwww.lesswrong.com##.OverallVoteAxis-voteScorewww.lesswrong.com##.PingbacksList-list > div > .Pingback-root > .Typography-root.Typography-body2.PostsItem2MetaInfo-metaInfo.Pingback-karma > .LWTooltip-rootwww.lesswrong.com##.Typography-root.Typography-headline.PostsVoteDefault-voteScore.PostsVoteDefault-voteScoreFooterwww.lesswrong.com##.AgreementVoteAxis-agreementSection > .AgreementVoteAxis-agreementScore > .LWTooltip-root > spanwww.lesswrong.com##.Typography-root.Typography-body2.PostsItem2MetaInfo-metaInfo.LWPostsItem-karmawww.lesswrong.com##.Typography-root.Typography-body2.MetaInfo-rootwww.lesswrong.com##.OverallVoteAxis-secondaryScoreNumberwww.lesswrong.com##.SingleLineComment-karmawww.lesswrong.com##.OverallVoteAxisSmall-voteScorewww.lesswrong.com##.SingleLineComment-leadingInfowww.lesswrong.com##.ais-Hits-list > li.ais-Hits-item > .ExpandedPostsSearchHit-root > .ExpandedPostsSearchHit-metaInfoRow > span
www.lesswrong.com##.ais-Hits-list > li.ais-Hits-item > .ExpandedCommentsSearchHit-root > .ExpandedCommentsSearchHit-authorRow > spanwww.alignmentforum.org##.Typography-root.Typography-headline.LWPostsPageTopHeaderVote-voteScorewww.alignmentforum.org##.OverallVoteAxis-voteScorewww.alignmentforum.org##.AgreementVoteAxis-agreementScorewww.alignmentforum.org##.LWPostsItem-karmawww.alignmentforum.org##.SingleLineComment-leadingInfowww.alignmentforum.org##.CommentsTableOfContents-commentKarma^Unlike LessWrong’s design, this study didn’t increase the visibility of highly rated posts. That would likely have strengthened the effects, as an initial upvote increases view count, which can lead to a compounding “rich get richer” outcome.Discuss ​Read More

​When reading a comment, the first thing you see is what other people think. That design choice reduces your ability to form your own opinion and makes the site’s karma rankings less related to the comment’s true value. I think the problem is fixable and propose some ideas for consideration.The LessWrong interface prioritizes social informationYou read a comment. What information is presented, and in what order?The order of information:Who wrote the comment (in bold);How much other people like this comment (as shown by the karma indicator);How much other people agree with this comment (as shown by the agreement score);The actual content.This is unwise design for a website that emphasizes truth-seeking. You don’t have a chance to read the comment and form your own opinion first. However, you can opt in to hiding usernames (until moused over) via your account settings page. A 2013 RCT supports the upvote-anchoring concernFrom Social Influence Bias: A Randomized Experiment (Muchnik et al., 2013):[1]We therefore designed and analyzed a large-scale randomized experiment on a social news aggregation Web site to investigate whether knowledge of such aggregates distorts decision-making. Prior ratings created significant bias in individual rating behavior, and positive and negative social influences created asymmetric herding effects. Whereas negative social influence inspired users to correct manipulated ratings, [an initial upvote] increased the likelihood of positive ratings by 32% and created accumulating positive herding that increased final ratings by 25% on average.Inline reaction indicators also seem anchoringInline reactions are shown as little icons to the right of the line of text. Here’s an image of sidelined reactions to a comment of mine:I find these “reactions” distracting. They discourage people from forming independent opinions and probably have produced too much agreement with my comment.When I’m reading LessWrong content and see an icon on the side, the icon grabs my attention and distracts me from the content.I wonder “ooh, who reacted?” and I mouse over it and start thinking about social implications instead of actually reading the content.I am now anchored to agree or disagree with the content in question.In order to avoid people’s first impressions being anchored by these reactions, I sometimes redirect users from LessWrong to my website.Concrete proposalsLikely the biggest win. Hide karma and agreement indicators in the hour after a comment is posted. This would reduce the initial “luck” of someone strong-upvoting a comment, leading to a cascade of other positive votes due to anchoring. This effect is evidenced by Social Influence Bias: A Randomized Experiment (Science, 2013).Move the username, karma, and agreement indicators to the bottom of the comment (or post) by default. For short comments, hide the username and numbers until the comment has been in the viewport for X seconds.Provide account-level toggles for both (2) and (2a).Don’t show reactions until the user has reached the bottom of the comment, at which point the user can:Mouse over the reactions to see who reacted in response to which content;Scroll back up and see the reactions off to the side.A mock-up of how (2) might be implemented. This assumes that Matthew’s comment was not collapsed (and just ends as shown). A more modest (but still good) change would be to just move the agreement score to the bottom. These ideas aren’t perfect. For example, karma is genuinely useful for selecting which comments you’d like to read. By making the karma less prominent, it’s harder to skim for comments above a karma threshold. Consider two cases: The comment is not collapsed. In this case, while skimming the webpage, you can scroll down and just learn to look at the bottom of comments instead of the top. If the comment passes a threshold, read it by scrolling up slightly. This is mildly inconvenient.The comment is collapsed. Then the karma count isn’t visible at the bottom (since otherwise it’d be visible early on). This is a problem.The fix might be to modify proposal (2) to keep “karma” at the top of the comment but keep “username” and “agreement” at the bottom. I’m open to other ideas which do an even better job of minimizing costs and maximizing gains!(And to anyone about to type “this can’t be fixed”, have you spent five minutes (by the clock) thinking about the issue first?)Prior discussion and resultsIn 2021, Max Harms talked about Improving on the Karma System. His proposal focused on augmenting the entire system, not just the way karma is displayed. The LessWrong team has made changes in related areas. Total karma is deliberately not displayed prominently. Side-comments default to “just an icon” until you mouse over them. Karma used to be much more prominent at the top of posts, but now (on desktop) it’s a smaller number in the top right. These seem like good choices. In 2013, gwern shared the results of a highly relevant experiment. gwern followed the posts made by eight participating authors. gwern used an alternate account to randomly upvote or downvote the article and post a comment with a boilerplate rationalized “explanation” for the vote. For example: “downvoted, not enough math.” A month later, gwern came back to measure the total karma of the post. After controlling for an outlier popular post by Scott Alexander, the data indicated that an early up/downvote produced a non-statistically significant effect (with a difference-in-mean karma of about 10). However, as gwern notes, the sample size was small, so it wasn’t highly powered to begin with.While gwern’s experiment measured a proxy of the bandwagon-y-ness of LessWrong at that point in time, it measured how an initial comment affected the final karma of the post. Not how the visual prominence of karma- and agree-counts on a comment affected the final karma- and agree-counts of that comment. Related, but not quite the same. This is evidence against super strong versions of the effect (e.g. “most voters bandwagon off of the existing score”), but compatible with meaningful anchoring due to current design choices. Also, being able to see agreement right away can be a stronger effect than a single comment saying “upvoted” or “downvoted” (and a +1 or -1 to post karma), since the initially displayed agreement might be quite strong (e.g. +25). Seeing a tally of multiple votes all at once likely has a stronger impact on decision-making.Please show social signals after the comment!To me, the most valuable part of LessWrong was how it encouraged interesting contrarian comments. Many of us value truth-seeking, so I hope users and moderators optimize the website to better reflect that value.Inspired to finally share this critique due to Ryan Greenblatt’s comment arguing that more people should post on LW rather than X. Appendix: Filter listHere’s what I use in my Brave browser to filter out AF / LW karma and agree-votes. www.lesswrong.com##.Typography-root.Typography-headline.LWPostsPageTopHeaderVote-voteScorewww.lesswrong.com##.CommentsTableOfContents-commentKarmawww.lesswrong.com###36 mokQtNacRh56foNv > div > .CommentsItem-root.recent-comments-node > .CommentsItem-body > .CommentsItemMeta-root > .NamesAttachedReactionsVoteOnComment-root > .AgreementVoteAxis-agreementSection > .AgreementVoteAxis-agreementScore > .LWTooltip-root > spanwww.lesswrong.com##.OverallVoteAxis-voteScorewww.lesswrong.com##.PingbacksList-list > div > .Pingback-root > .Typography-root.Typography-body2.PostsItem2MetaInfo-metaInfo.Pingback-karma > .LWTooltip-rootwww.lesswrong.com##.Typography-root.Typography-headline.PostsVoteDefault-voteScore.PostsVoteDefault-voteScoreFooterwww.lesswrong.com##.AgreementVoteAxis-agreementSection > .AgreementVoteAxis-agreementScore > .LWTooltip-root > spanwww.lesswrong.com##.Typography-root.Typography-body2.PostsItem2MetaInfo-metaInfo.LWPostsItem-karmawww.lesswrong.com##.Typography-root.Typography-body2.MetaInfo-rootwww.lesswrong.com##.OverallVoteAxis-secondaryScoreNumberwww.lesswrong.com##.SingleLineComment-karmawww.lesswrong.com##.OverallVoteAxisSmall-voteScorewww.lesswrong.com##.SingleLineComment-leadingInfowww.lesswrong.com##.ais-Hits-list > li.ais-Hits-item > .ExpandedPostsSearchHit-root > .ExpandedPostsSearchHit-metaInfoRow > span
www.lesswrong.com##.ais-Hits-list > li.ais-Hits-item > .ExpandedCommentsSearchHit-root > .ExpandedCommentsSearchHit-authorRow > spanwww.alignmentforum.org##.Typography-root.Typography-headline.LWPostsPageTopHeaderVote-voteScorewww.alignmentforum.org##.OverallVoteAxis-voteScorewww.alignmentforum.org##.AgreementVoteAxis-agreementScorewww.alignmentforum.org##.LWPostsItem-karmawww.alignmentforum.org##.SingleLineComment-leadingInfowww.alignmentforum.org##.CommentsTableOfContents-commentKarma^Unlike LessWrong’s design, this study didn’t increase the visibility of highly rated posts. That would likely have strengthened the effects, as an initial upvote increases view count, which can lead to a compounding “rich get richer” outcome.Discuss ​Read More

Related Posts

Leave a Reply

Your email address will not be published. Required fields are marked *