Sue Gardner on editor retention and the “holy shit” death spiral slide

screenshot sue speaks to wikimedia uk1Sue Gardner speaks to Wikimedia UK’s meeting of 11-19-2011. Slides here.  The video, as well as the visuals used in the presentation, are now available on Commons in three parts:

Section 1 of 3
Section 2 of 3
Section 3 of 3

Note: The following is a transcript of the first video only.  I have made arbitrary headings to highlight themes I find to be still worthy of attention: “the ‘holy shit’ death spiral slide”, “endless September”, and “Wikipedia as a video game — ‘it’s all nuns and tourists'”.

Sue Gardner: [continuing a conversation]…towards him and then that part will feel maybe like remedial and old school for other people, or maybe it won’t, right? I don’t know.

So, have you folks read the March 11th update, that we published, the Wikimedia Foundation –March 2011? No? None of you? Oh, this is going to be important for everybody, then.

[crosstalk]

Okay, yeah, that’s possible, that’s for sure possible. It was around the time of the board’s resolution on openness, which was published, which was, I can’t remember, 6 weeks either way of the update. I think we wrote the update–I think the board met in April and published a resolution on openness afterwards.

So, essentially, going back to what Andrew was saying before to this thing. (points to notes on easel board)
So we built—in the strategic plan–we built the five pillars for what was strategically important, what the Wikimedia movement out to be focused on. And when we built them, we did not know, how bad the editor retention problem was, right?

Wikimedia_UK_Nov_2011_Discussion_sheet_1 ASo when I go back to my board every year, and I say, you know, here is the plan for the year, what I say is:
~Stabilize infrastructure? We’re okay, right, we’re doing okay. We gradually improve, everything gets better and better.
~Quality…. Continues to increase, it increases naturally through the projects, that’s what happens, right? And there are lots of people focused on increasing quality with things like Wiki Loves Monuments.
~Reach. The number has continued to climb, there’s no problem, right? We need to do a lot of work on mobile to support people who are only going to use the internet via mobile devices, especially in the Global South. But it’s fine, it’s in hand. The numbers are going in the right direction, 477 million unique visitors, by the way, reported by comScore like, yesterday, which is our highest ever number. So we’re all good, right?
~Innovation. We’re wonky. Innovation happens unpredictably; it’s not particularly focused, necessarily. But there’s lots of decentralized stuff going on, so that’s okay too.
~(pointing to..what? apparently “increase participation”) This is not okay. This is not, not, not, not, not, at all okay. This is a huge problem, right?
So, we didn’t know it when we built these five. And so what I’ve done is, in my planning for the Wikimedia Foundation, since doing this work, we’ve pulled this out and highlighted it and emphasized it. And it’s tough, because money doesn’t necessarily solve that problem, right. So it’s not easy, it’s not like we can just dedicate the largest proportion of the budget to it and that will solve it. But it’s the thing that needs to be solved; it’s the hard problem that needs to be fixed. So, I will take you through it, and I’ve got links all through this, so you can go read after,wards, or whatever.

The “holy shit” death spiral slide

[2:30] Have you seen this slide? Okay, at the Wikimedia Foundation we call this the “holy shit slide”, okay? It’s like really, really bad.

800px-WMUK_November_2011_Sue_presentation

The blue is current active editors at any given moment in time, you know, beginning in January 2001, and then going pretty much to today. You can see that it peaked—I think around 2005, 2006.

Endless September

We think we know why–there’s lots of reading you can do if you’re curious—it’s essentially Endless September, variations of Endless September, if you’re familiar with that phrase. If you’re not, you should read about that phrase, ’cause that’s the heart of the problem. [Note: see Wikipedia “Eternal September”]

[3:03] And then red, I believe is new editors, who survived one year after their start date. So we’ve done lots of research, the problem is, new editors aren’t making it to their first year anniversary. If they make it to their first year anniversary, they’re about as okay as they always were; they have the normal life cycle after that, but they’re not getting that far, that’s the core of the problem. So the problem is, people are coming in, in large numbers as they always have, right; they’re trying to enter the community–they want to edit–and they’re getting rebuffed. The situation is worst on the German and the English Wikipedias. Which is why I’m interested in you guys, right? Because you should be able to be a part of the solution.

So, why are they failing to penetrate the community? We think that the reason is because warnings have gone up, okay, criticism is way up, and praise and thanks have been in decline. So you can see that here, right, (points to board) and find that. The obvious question is, does that mean it’s just spammers and vandals–of course they’re getting warned and criticized, because they’re jerks and we don’t want them anyway? The answer is no, it is not because they’re vandals and spammers. So this chart, took out all vandalism, it took out all spamming, and it took out all known sock puppeting, and only what remained was good faith editors. We’ve got other charts that show, you know, that the percentage of good articles that they’re writing is the same as it ever was, the percentage of good faith edits is the same as it ever was, it’s not an increase in problematic new contributors.

What’s happening is that people who are just as good as they ever were, are coming in and getting smacked. You can see it, it’s actually incredibly heart-breaking. If you go onto the talk pages of new editors who have newly registered, it’s just template after template after template after template, right. And you know, it’s heart-breaking on both sides, because you can easily imagine yourself in the shoes of those new people, and how (laughs), like, why would they want to be part of this, right? Like they come and they make a change and they do something, and then they just get yelled at. And then you can also imagine it from the shoes of the Wikipedians who have made those templates, right? Because they’re trying, right, they’re giving them advice, they’re giving them forty links of places to go to get help, they’re not trying to hurt those people, right? They’re trying, actually ironically and kind of sadly, they’re trying to help them. But the effect of it, if you’ve got dozens and dozens and dozens of those templates, it doesn’t make you feel at all welcome. It makes you feel like Wikipedia hates you and wants to go away. And so, they do go away.

Are you guys with me? Like, yeah? You should tell me, if you disagree, at any point, or you’re not buying it, like tell me.

[Inaudible question]

In terms of the warnings? Yeah. [inaudible] That’s right. It’s interesting, I think I’ve got another slide that shows, um, that shows how it breaks down, let me see if I do. Yeah here, so, what’s interesting about this, right, is that you can see—what I found the most interesting about this, is pre-2006, there were no bots, there were no automated warnings, right? It was real people talking to real people. And if you go back and look at people’s talk pages, like, a person might make a crappy first edit, and then someone would go to their page and they would say, “oh, hey, you know, what you did is kind of wrong, do you want to talk about how to fix it?” And then the person, the new editor, would probably feel sort of okay about that, and they’d be like “okay, what did I do wrong”, and it was individual people talking to individual people. And then with the rise of the bots and so forth, suddenly they were depositing 500 words on your page of instruction, right, which just didn’t, you know, feel helpful. So you can see there, there was nothing before 2006, but today 4 out of 5 of the messages are bot-delivered, and 65% of them are straight-up warnings.

[6:56] [off-mike question]

[7:39] You’re completely wrong. I can see why you think that, not having seen, no but really, like I just, you know, we did, we gathered together,… we did kind of a meta-analysis where we pulled together everything that we knew, all the research that we had done, and that other entities had done, and so forth–I’ll show it to you later–we did a kind of testing of hypotheses. So, we brain-stormed maybe 20 hypotheses. One of them was that which we at the Wikimedia Foundation called the Gold Rush Theory, right? So the notion that the early prospecting days, where there was lots of gold to be made, are over, and so naturally today, you know, you’d have fewer people because there’s less to write about and so forth. There’s very little evidence to support that, there’s really no evidence to support that, and in fact, quite the contrary, you see the same pattern that we’re seeing on the English Wikipedia, and the German Wikipedia. You see it on the Hindi Wikipedia. You see it on Wikipedias that have five thousand articles, right. So there’s nothing to support that that is in fact what’s happened and there’s lots of evidence to suggest that it’s the Endless September and the other hypotheses that are actually true. You know, we see the same pattern, it’s essentially, if you go…let me go back…
.
[off-mike question about graph]

The blue is the same, right, it’s the same for all the language versions. It seems to be a timing issue. It’s that after—what it looks like to us, is that after a couple of years, a wiki reaches some kind of point at which it starts getting crufted over with rules and policies and warnings and templates and so forth, right, and it just becomes impenetrable. It grows a hide around itself where it’s no longer as porous, right, and no longer as open.
[9:20] And we know that—someone did a somebody did a thing on the English Wikipedia–I can’t remember who it is, maybe it is WereSpielChequers. Somebody did a thing where they got people to talk about their very first edits. I’m not talking about that other thing he did, that was great, I love that thing. But somebody did a thing where they got folks to talk about their first edits, right, and almost everybody’s edit was like, a terrible edit, right, like, you know, it was a COI edit, or it was a vandalism, or something awful. And it used to be the case that you could recover from that and become a good editor, and that now is not true. So sometimes when I talk to people who think that this is maybe okay or it’s not as bad as it seems or whatever, I’m always curious to know when they started editing. Because, I think, often we wrongly think it’s still the same for new people as it was for us. And it’s not even the same today as it was for me in 2007, right. Like, they’re having – look at their user pages—they’re just having a terrible experience. They’re just flooded with criticism, right. So that’s what’s happening.
.
[off-mike question]

[10:37] So, this is a piece of research that was done that shows us that deletions and reversions make new people less likely to stick around. And the effect, is actually, ironically and sadly, the effect is strongest for new editors who are highly prolific.

[10:50] So we did some analysis of what kinds of people turn out to be good Wikipedians, right. One of the indicators that someone is going to be a good Wikipedian is that they make a lot of edits in their first few days, like they basically get really excited and they really enjoy it and they’re suited to it, and they like it and so they do a lot of it. So I think it’s the duration of your first editing session and the number of edits you save in your first editing session is some kind of determining factor of how you’re going to take to the community. And those apparently, our research suggests, are the people who are most disincentivized, or most pushed away by being deleted and being reverted.

[11:30] So we know what the problem is, right, the problem is that you make a crappy edit just like everyone else has always made a crappy edit. You get deleted, nobody tells you why, you don’t get an explanation, if you do get an explanation it’s not from a human being, it’s machine language saying you violated some acronym you don’t know what it means, you’ve never been here before, you don’t know what’s going on.

Our qualitative research, and some of the quant stuff we’re done with free text fields tells us that people don’t distinguish between other Wikipedians and the website, right? So they think that the website is yelling at them, they don’t understand. And so it’s the same as if I’m on Facebook, and Facebook deletes something I put up. I feel reprimanded by Facebook. I feel like Facebook doesn’t like me, I’m using it wrong, something’s wrong. That’s the experience that people are having on Wikipedia. They don’t understand that maybe they’re having a dispute, an argument with another editor on a talk page. They don’t understand that that’s another person with their own opinion, they think that’s the website yelling at them. And so they just feel terrible, right? Like, we know, because we ask them. They just feel terrible, they feel like the website hates them, nobody wants their input, they’re doing it wrong, they all say they’re doing it wrong, they have no idea what’s going on around them they don’t know what a talk page is, etcetera etcetera, right? So super, super challenging.

[12:48] So. Why is it a problem? You guys stop me at any point if you want to stop me. John, you should stop me too, all right?

[Question from audience: “You focus very much on new editors, the bar for them is becoming more and more hostile. Have you seen any patterns in terms of old editors, particularly non-admins, who have been doing a small number of edits for a long time. Who are also kind of getting the same thing]

[13:22] Yeah, we’ve only started doing editor—like proper editor surveys–where the margin of error is something reasonable–about two years ago. I think we might have two data points, or we might just have one and then a secondary data point coming. Now what we have is just a baseline. We don’t have change over time, yet. But what we think, we’ve observed anecdotally in talking to people is that it’s harder for everyone, in fact, there’s a kind of ripple effect, so I’ll get to that next, so there’s a ripple effect which is kind of troubling, which you can see in places like Arbcom.

[14:01] So why does it matter?–this is getting to your point. Really what it means, is there aren’t enough people to do the work. So the people who are here, what that means is, the existing editors, this is anecdotal, right, but we know, qualitatively, people are stressed, they’re overworked, and they’re burned out, right? We can see it, right. And it exacerbates, it makes the entire culture fightier than it needs to be. It means that experienced editors have a harder time taking on leadership roles, because they get bogged down in basic tasks.

This is something I observed pretty early at the Wikimedia Foundation because Wikipedia operates close enough to the way a news room operates, that I can understand it from when I was a journalist, right? And journalists in a newsroom—senior, experienced journalists, behave in a certain way, it’s predictable, it’s understandable. It’s like if you’ve been chasing ambulances and waking up at three in the morning to go to fires for fifteen years or so, just comes a point where you want a desk job, right? And you should get a desk job. Because you’re older and wiser, and you’re been around, and you can be judicious, and all of that.

[15:03] And what I’ve noticed at Wikipedia, and again this is just my own impression, right, but my impression based on having been a working journalist for fifteen years. What I noticed was those old guys who had good judgment and had been around, were still going after fires, right? And it was heartbreaking, because, there was, I mean, you do have folks like NewYorkBrad, who’s on arbcom, who has a more traditional, wise, elder person’s role, right? But you still have lots of people who are doing scut work, right? And where—so one of my first questions when I came in was, where are the new generations of young people relieving them of the need to do all the scut work, right? Um, so, we see bureaucrat, administrator, arbcom, positions getting harder to fill, steward type positions getting harder to fill, right. For a whole bunch of reasons that become self-reinforcing, and then the requirements to become a steward are so ridiculous that nobody can ever be a steward, right, because you’re going to be a steward for life and there are only ten of you so all must be perfect, right?

[16:02] A vicious circle comes into play, the newbies are increasingly being criticized, warned and driven off, which leads into a kind of death spiral, right, cause it self-reinforces, again, the whole system self-reinforces over time. From a content perspective, from a quality perspective, systemic bias will get worse, right? We don’t benefit from the quality improvements that we should get, when new people come in, right?. And the older editors–the supposition is–the older editors are going to naturally age out, and there’s nobody to replace them right?

[16:34] We know that Wikipedia editors have a life cycle, right? Not all of them, everybody is different, but they tend—there are some tendencies—that, you know people typically tend to have, and what it is, is, you know, you start editing Wikipedia when you’re in school, usually you know doing your undergraduate, or doing your graduate work, you get hooked on it, and you love it, and then frankly you edit until, and forgive me for saying this, your wife makes you stop, [laughs] right? A lot of editors have told me that that’s their biggest challenge in editing Wikipedia is that their partner–usually a wife, but not always—their partner doesn’t support it, right?

[off-mike question: “let me just get the other half to join in”]

That’s right. And this is why when I go to meetups and stuff, if there’s someone’s wife there, I love the wife, it’s like, “can we get you anything? Do you like us? Can we introduce you to people”, you know.

I think so too. And there’s a whole bunch of reasons why. I mean, sometimes it happens, and “somebody’s wife” is kind of a label that also applies to the family in general, and so forth, right, a lot of the editors who I’ve talked to it’s their, sometimes their parents, right, sometimes a girlfriend, whatever, but it’s essentially somebody saying “time to put away the childish things, right, which I think is…horrific…because I don’t think of this as childish, right, but it, you know, I had a guy in Israel tell me his partner—he’s a lawyer—and his partner was saying like “When are you going to make partner, like when are you going to start investing in your career”, right? And I think that you’re right. Even if it’s not that, even if it’s not like pressure from someone in your family or whatever, people ebb and flow and they go through different life stages.

[Question off-mike: “(unintelligible) this is what I want to do, (unintelligible) and they do it, and they stop, you’re talking about every three months to check out see if they still (unintelligible) and that’s it “]

That’s right

[Question: “I’d like to the new editors here, …an example is when I joined the project, after I’ve done my time getting myself a hundred thousand edits, I’ve become an admin, I then worked in the DYK project on the page. You could put up an article there, four years ago that was basically, everything, write about my village, make a hundred words, and make a reference, a picture, and a page, and that thing there and they want twelve references]

Precisely right.

[crosstalk]

[Off-mike: “and there are an army of people to pick on somebody, even as an experience editor, if I’m walking a newbie through, I get spears in my back for walking that person through.”]

I have also heard that FA now, they, you actually can’t, there are articles for which you will never get an FA, no matter how good the article is, because the nature of the topic doesn’t lend itself to the kinds of citations and so forth that are required.

[crosstalk. “…sorry, it’s politically unstable, so it’s …(unintelligible) ]

[20:20] But I think it’s part and parcel of the same thing, you know what I mean, like, in 2005 2006, the walls started to go up around things, and it’s really interesting, right, because I remember talking to Florence Devouard, when she was chair of the board, when I first came in 2007, right? And she and I had a lot of difficulty working well together, it was tough, right? And we talked about it really explicitly, which was the only thing that was good about it, that we were able to talk about it really explicitly. But I remember her saying to me, you know, the kind of way that she liked things to be, that she liked things to be sloppy and loose and messy and noisy and difficult and all sorts of stuff, right? And was kind of like, you know, I am representing a different world, right, and like, we’re evolving and we’re changing and we’re going to become much more professional, blah blah blah. All of that was about the board, and the leadership of the Wikimedia Foundation, but it was also in a way a metaphor or a comparative to what was happening in the projects, right.

What I didn’t realize at the time, that I do realize now, is that the projects were changing and not all the changes were good, right. So you know, it’s a different kind of person, like Florence, who likes a green field Wikipedia right, 2001 where there is no article about Saturn. That’s a different person from person who wants to polish the shining gem of the Saturn article or protect it from other people who are going to hurt it, or whatever right. And I honestly think we’ve lost our way, I really do, right? I think that , you know, what we—I sometimes think that we’ve become Nupedia and we need a new Wikipedia, right? to feed into it. I think that we.. we..we’re I just think we’re really, really rigid, right? I’m not disagreeing with you on the FA thing, because I’m sure that there are some of our processes that make perfect sense, but in general, right, I think we’ve become really rigid and really closed, and I think we’ve lost sight of what makes the project special, which is the idea that everybody does have something to contribute, it’s not just a priesthood of, you know, a small number of people.

[audience question: “The compromise between quality and the quality you’re increasing traffic, so you get so you’re a perfect article, one sentence that you’ve cut and pasted our of another article, some other source, they’re treated as if they’re kind of…animals…a few more seconds and a few more words and they would have got through.]

[another speaker: “My criticism of the leadership so far is that everything you’ve said so far is that you’ve underestimated (unintelligible) I was contacted by someone I professionally who said, “can you tell me what I’ve done wrong” and they’d started an article in their user space, and they were drafting it, and the first thing on their talk page said ‘if you don’t stop your disruptive editing, you will be blocked’”]

Oh, my God. Oh.

[Off-mike comment: and they were writing about a book, and it was [unintelligible) and they were trying to write it and get help with it]

And they’re doing it in user space.

[Off-mike: “and no welcoming, ‘if you don’t stop your disruption you will be blocked’ and that’s (unintelligible) and no explanation of why is this disruption”]

[23:30] And you know what I think, a lot of that comes out of is the Seigenthaler thing, right? That I think that the Seigenthaler –so the Seigenthaler –does everybody know the Seigenthaler story? John do you know? [“no.”] Okay. So, the Seigenthaler thing happened and then after that Jimmy went to Wikimania and said quality, we need to do better, etcetera, and then I think it was around the same time that Essjay was outed as falsely claiming credentials he didn’t have, right, in the New Yorker, and so I think that there was this moral panic created around quality, right. And I think that it became—it’s funny, right, because like, Jimmy said something, even today, but much more so then, and the ripples that go out in the community and the language versions and , you know, like everybody kind of a photocopy of a photocopy of a photocopy of what Jimmy said, right?

Wikipedia as a video game — “it’s all nuns and tourists”

And I think that what Jimmy said gave a whole lot of people license, they felt like they had license to be jerks, right, and to be like, we talk about it at the Wikimedia Foundation office like a “first person shooter”, right? Like, folks are like, playing Wikipedia like it’s a video game, and their job is to kill vandals, right, and then we talk about how every now and then a nun, or a tourist, wanders in front of the AK47 [laughter] and they just get murdered, but in actual fact, what we think now is that it’s all nuns and tourists, right, and it’s a big massacre, right, and there’s one vandal running away in the background, you know, and meanwhile everybody else is dead. [laughter] [“Yes”]

[“Everything on that chart (unintelligible) is anecdotal, is key to strategy, so why is it still anecdotal, why isn’t it underpinned (crosstalk) about what happens to members, certainly goes through, things are getting harder to fill, these are still anecdotal, you know there should be charts for these things, measure…]

[25:35] I think there probably will be, there needs to be, like I say we’ve got one data point right now, right, we’ve got one editor survey . We’re gathering data on all this stuff, and there’s some more stuff to come, so [crosstalk] one would hope, I mean, I deliberately kept the phrase “death spiral“, I just don’t want to put I writing, you know [laughter] I mean, it’s too depressing. I know, I didn’t want [crosstalk] image of the slide, with the word “death spiral”.

[crosstalk: “I think we should be given good motivation”]

[26:07] Yeah exactly, that’s right, fear. FUD, just call it FUD.

[26:11] So what should we do. So this is the March 2011 resolution that the board –this is a piece of it—that the board released, right. It’s the Wikimedia Foundation board of trustees, um, rarely, uh, speaks directly to editors, right? in this kind of way. I think it’s only done it a few times before, partly because it used to be Jimmy, right? It used to be Jimmy would proclaim something, and that was sufficient, and then as a movement develops and it gets more mature, you know, it needs institutions, not individuals. Jimmy still plays an individual role but the board as an institution has more of a role today than it used to, and so the first time I think it did something like this was the BLP resolution asking for people to take that a little more seriously because it was real people’s lives at stake, real people’s reputations. Um, and this was the second time, and so what they said was, the important part about this is, this is our top priority, right, that was the purpose.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s