IdentityTalk

View Github Repository Open presentation in a new window

seanjtaylor

See all presentation from seanjtaylor

IdentityTalk

0 0

IdentityTalk

On Github seanjtaylor / IdentityTalk

Identity and Opinion:

A Randomized Experiment

Sean J. Taylor (NYU), Lev Muchnik (HUJI), and Sinan Aral (MIT)

Identity and Opinion:

A Randomized Experiment

Sean J. Taylor (NYU), Lev Muchnik (HUJI), and Sinan Aral (MIT)

Identity and Opinion:

A Randomized Experiment

Identity and Content are Tied

What makes social media social

Twitter

(18% of online adults, Pew 2013)

Reddit

(6% of online adults, Pew 2013)

Facebook

(665m daily active users, Facebook 2013)

Pinterest

(70m users, Semiocast 2013)

Identity Cues

  • are pervasive
  • provide quality signal
  • have a learned meaning/association over time
  • affect our evaluation of content?

How Valuable is an Identity?

more favorable ratings? higher rate of engagement, response, or sharing? NOT the same as audience size

Identity and Content are Endogenous

Related Work

  • measuring user content quality (Agichtein et al. 2008)
  • measuring sharer reputation (Yang et al. 2013)
  • source effects in evaluation of social media (Messing 2013)
  • economic value of online reputation (Resnick et al. 2002)
  • causal effect of social cues (Bakshy et al. 2012, Muchnik et al. 2013)

Field Experiment

Social news website (similar to Reddit)

Comments on posted articles

  • ~ 2 years on a social news website
  • ~ 3,700 commenters
  • ~ 6,700 comment viewers
  • ~ 3.5 x 105 comments
  • ~ 1.7 x 107 comment exposures

Conditions

identified author

95% of viewer-comment exposures

anonymous author

5% of viewer-comment exposures

Users were informed this was a site-wide experiment.

Two year study allows us to rule out novelty effects.

Content Interactions

Comment rating is anonymous, ruling out reciprocity/retribution effects

Measures

  • Turnout rate: either up-vote or down-vote
  • Up-vote rate
  • Down-vote rate
  • Positivity (opinion change): up-vote given turnout
  • Reply rate

Experiment Design

Model

\( \Pr(Y_{ijk} = 1| D_{jk}) \sim \alpha_{ij} + \delta D_{jk} \)

  • $Y_{ijk}$ — 1 if the viewer $k$ interacts with $i$'s comment $k$
  • $D_{jk}$ — viewer $j$ sees commenter for comment $k$
  • $i$ — commenter
  • $j$ — viewer
  • $k$ — comment

ATE of Identity

ATEI = \( \frac{\Pr(Y_{ijk} = 1 | D_{jk} = 1)}{\Pr(Y_{ijk} = 1 | D_{jk} = 0)} \)

Conditional ATE of Identity

ATEI(x) = \( \frac{\Pr(Y_{ijk} = 1 | D_{jk} = 1, X_{ijk} = x)}{\Pr(Y_{ijk} = 1 | D_{jk} = 0, X_{ijk} = x)} \)

Average Treatment Effects

Bootstrap 95% Confidence Intervals for Relative Risk

Commenter-level Effects

Comments are highly skewed toward active users, allowing a within-subject analysis.

Active commenters seem to mostly be helped by their identities.

Dramatic positive opinion change for some active users.

Higher reply rates for some commenters when identity is shown.

Hypotheses

new vs. old users articulated relationship accumulating (dis)advantage

Uncertainty about identity

Articulated Relationship

  • friends
  • enemies
  • neutral
  • no relationship

You are how you're rated?

Usually seen next to high scores → helpful identity.

Usually seen next to low scores → harmful identity.

New users

New vs old viewers

New viewer subgroup: 4.6x10^5 exposures

New vs old commenters

New commenter subgroup: 4.9x10^5 exposures

Articulated Relationship

Influence of Prior Score Cues

Influence of Prior Score Cues

Influence of Prior Score Cues

Findings

rating and reply behavior are altered by identity for active commenters, identity seems to improve ratings new users creating accounts out of spite? tenured users strongly biased against new users strong identity effects for users with articulated relationships associated bad scores hurt ratings, good scores help replies

A mechanism for cumulative advantage in ratings?