Dark social traffic in the mobile app era

About two years ago, I wrote a story about a strange phenomenon on the web: in a medium known for its ability to track people—following them around with Zappo’s ads and such—it turns out that websites don’t know where a substantial percentage of their visitors come from. That is to say, when a visitor arrives at Fusion.net, we often don’t know how they got there or what link they followed. In my story, I called this kind of traffic dark social, and the name stuck. Dark social became a rallying cry for people who wanted the old, pre-Facebook web to thrive! Now, there are hundreds of thousands of references to the phrase across the Internet.

I think I was mostly right in the original story: people do and did send many links privately, which were not being counted as “social” by the web beancounters. But over the last two years, the Internet landscape has been changing. People use their phones differently from their computers, and that has made Facebook more dominant.

In the original story, I relied on data from Chartbeat, the web app that media people use to obsessively track their traffic in real-time. Since it came out, I’ve done research that’s caused me to rethink dark social—and so has Chartbeat. And they released new research to me this week showing that a good chunk of what we might have called dark social visits are actually Facebook mobile app visitors in disguise.

The takeaway is this: if you’re a media company, you are almost certainly underestimating your Facebook traffic. The only question is how much Facebook traffic you’re not counting.

The good news is that, as of yesterday at 6 pm, Chartbeat began tracking Facebook mobile users much more effectively. With the new changes, they’ll be able to properly attribute between 10 and 50 percent of dark social traffic into the right buckets. (It varies on a story by story basis.)

The bad news is that, if you didn’t know before, it should be even more clear now: Facebook owns web media distribution.

An Internet of phones

Let’s review the changes to the basic Internet paradigm that are driving this change. The mobile web has exploded. This is due to the falling cost and rising quality of smartphones. Now, both Apple and Google have huge numbers of great apps, and people love them. As this slide from venture capital analyst Benedict Evans shows, people spend about as much time in apps as they do on the desktop and mobile webs combined.

Chief among the non-gaming apps is Facebook. They’ve done a remarkable job building a mobile app that keeps people using it. And beginning last October, Facebook made changes in its algorithm that started pushing massive amounts of traffic to media publishers. In some cases, as at The Atlantic, where I last worked, our Facebook traffic went up triple-digit percentages. Facebook simultaneously also pushed users to like pages from media companies, which drove up the fan-counts at all kinds of media sites. If you see a page with a million followers, there is a 99 percent chance that it got a push from Facebook.

At around this same time, The Atlantic began to notice a huge rise in our dark social traffic. We’re talking on the order of millions of visitors—and it was all coming from mobile devices. This was in addition to the rise in Facebook traffic.

How were all these people arriving on our website from their phones, but without referrers? Were people WhatsApping links to each other? Was it e-mail sharing?

I suspected that this was not the case. Based on the spidey-sense one gets watching Chartbeat, I began to think Facebook was both responsible for the huge boost in referrals that we could see and for most of the boost in dark social traffic, too.

Here’s why: when people are going through their news feeds on the Facebook app and they click on a link, it’s as if someone cut and pasted that link into the browser, meaning that the Facebook app and the target website don’t do the normal handshaking that they do on the web. In the desktop scenario, the incoming visitor has a tout that runs ahead to the website and says, “Hey, I’m coming from Facebook.com.” In the mobile app scenario that communication, known as the referrer, does not happen.

If the theory was right, and this was happening at scale, Facebook—which every media publisher already knows owns them—actually has a much tighter grip on web traffic than anyone had thought. Which would make their big-footing among publishers that much more interesting. Because they certainly know how much traffic they’re sending to all your favorite websites, even if those websites themselves do not.

The Early Experiments

So, earlier this year, when I was still at The Atlantic, I began to run some tests. I created a post deep in the sands of time, so that no one not in the experiment would find it. Then I posted a link to that story on my Facebook page and told my friends only to click on it if they were using the mobile app. Then I looked at the referrers for that page, a page where I knew 100 percent of the traffic was coming from Facebook. A few people did show up as coming from Facebook.com, but the rest showed up as “Typed/Bookmarked,” which is the analytics package Omniture’s misnamed category for dark social.

The chart also told me that while we were missing most Facebook people using phones, there were some Facebook mobile referrals. Through some more research, I found that when people clicked for a second time after arriving at The Atlantic from Facebook, then we’d see a Facebook referral.

I figured that these numbers would be tightly correlated with the overall number of Facebook mobile visitors we had. So I took them and compared them against the dark social traffic we were getting. As you can see below, the correlation is good. When mobile Facebook referrals went up or down, the dark social traffic generally moved in accordance, too. (Note in the example below, the two y-axes are different: there are way, way fewer mobile Facebook referrals than total dark social traffic. NB: I’ve scrubbed the actual numbers to protect The Atlantic‘s internal statistics.)

This same phenomenon occurs across the web, and I knew from friends at other media companies that The Atlantic‘s mobile experience was pretty standard for medium-to-large web publishers.

So, I wasn’t the only person starting to suspect that mobile dark social works in a completely different way from dark social on the desktop. Some sites began to use tools like Crowdtangle to peer inside the Facebook ecosystem. Others append tracking codes to the links they pushed out on social media. Facebook itself gives publishers access to decent data about their own social media efforts, too. But none of these tools captures all the traffic Facebook sends to publishers. The Guardian specifically noted that dark traffic was flowing to their stories on mobile, too. They only lacked referrers for 10-15 percent of readers, they told Business Insider, but I’ve seen and heard about much higher numbers at other sites. At The Atlantic, the single largest category of referrer was dark social traffic coming from mobile devices.

Tracking Dark Facebook

Now, back to the new evidence from Chartbeat that Facebook is really the big player in mobile dark social. “These days, dark social accounts for about a third of external traffic to sites across our network,” Chartbeat’s Josh Schwartz wrote to me in an e-mail. “That number is dramatically higher on mobile, with upwards of 50 percent of mobile external traffic lacking a referrer.” Think about that: half the traffic doesn’t have a point of origin.

So Schwartz and the Chartbeat team started looking at stories across their network and trying to pull apart where the dark social traffic was coming from. They identified three possible large sources: Facebook app traffic, Facebook desktop traffic when a new tab opens, and Reddit apps. They found that for most stories, if they looked closely enough, they could find a single social network driving the dark social on mobile.

“A careful analysis of a particular story is likely to be able to turn up the source of the majority of its dark social,” Schwartz said. “Of course, there are also person-person shares (IM, e-mail, etc.), shares on apps with no corresponding website, etc. that account for a chunk of dark social.”

During the course of their research, the Chartbeat team discovered that while Facebook visitors don’t shout to the analytics programs, “Hey I’m from Facebook.com,” they do wear a sort of name tag. Whenever you go to a website, you take along a little profile called a “user agent.” It says what my operating system is and what kind of browser I use, along with some other information. Here’s mine from a search I made:

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.71 Safari/537.36

A user coming from the Facebook app, at least on an iPhone, would have a little message in there saying they originated at Facebook. Most analytics programs don’t parse that, however. Or at least they didn’t. As of 6 pm ET yesterday, however, Chartbeat will now count many more mobile Facebook users.

Schwartz told me that The Atlantic‘s dark social fell 40 percent and its Facebook referrals went up by roughly the same amount under this new analysis. Sooner or later, all the analytics packages will do this, and the media world will see the true power of Facebook (even if some of referrals are still not being captured by this method for reasons too byzantine to go into here). Buzzfeed has been tracking mobile Facebook users this way since May of 2013, publisher Dao Nguyen told me after this story published. The Guardian, has employed similar means in recent months, too, said chief digital officer Tanya Cordrey.

In my original dark social article, I had two main takeaways. The first was that the only way to truly optimize for “social” is to make the content itself shareable, regardless of platform. Dark social is distributed, so there is no way to game the system. I’m not sure this is true anymore. A story’s shareability is now largely determined by its shareability on Facebook, with all its attendant quirks and feedback loops. We’re all optimizing for Facebook now, which is why you saw 50+ publications all posting the same John Oliver videos.

The second takeaway was historical: I said that social networks have only structured the experience of sharing on the web, not created it. Facebook was not only just a subset of sharing, but one that antedated many other forms. That is still true, but the social networks—by which I mostly mean Facebook—have begun to eat away at the roots of the old ways of sharing on non-commercial platforms. Mobile is becoming the dominant way people access the Internet. And true person-to-person dark social appears to be less prevalent on mobile devices. Because what people like to do with their phones, en masse, is open up the Facebook app and thumb through their news feeds.