There's a searchable cache of the web's darkest corners of the anonymous internet

Gwern Branwen, a deep web researcher and cryptocurrency advocate, is working to make the dark net a more accessible place to “normal” people one website at a time. After years of scraping and cataloguing the activity happening on dark net markets (DNMs) like the Silk Road, Branwen’s turned the trove of data he mined into a downloadable and searchable directory that can give anyone a peek into the sometimes unseemly world of the deep, anonymous internet.

The dark net, as it’s traditionally thought of, is a section of the larger internet only accessible through special browsers like TOR, the onion router, that obfuscates a user’s identity. By using TOR, people are able to access websites built largely on anonymity. As a result, a lot of the dark net functions as a safe haven for illegal activity.

While more popular DNMs like the Silk Road expressly forbid the sale and purchase of child pornography, for example, other dark web sites specialized in it. One, which Branwen cites, went so far as to create sophisticated crowdfunding market designed to make sure that you had to pay for the illegal content instead of downloading it for free from other seedy corners of the dark net.

Branwen first became interested in dark net markets (DNMs) back in 2011 when Gawker’s Adrian Chen first introduced the general public to the Silk Road, a marketplace where you could buy virtually any drug you can imagine using Bitcoins. Initially, Branwen, like many other people, believed Silk Road to be a scam designed to bilk people out of their money, but he quickly learned that the anonymous marketplace was very, very real.

“Fascinated, I signed up, made my first order, and began documenting how to use SR1 (Silk Road 1) and then a few months later, began documenting the first known SR1-linked arrests,” Branwen explains in a description of his years-long project. “Monitoring DNMs was easy because SR1 was overwhelmingly dominant and BlackMarket Reloaded was a distant second-place market, with a few irrelevancies like Deepbay or Sheep and then the flashy Atlantis.”

From 2013 to 2015, Branwen obsessively tracked a wide variety of metrics associated with nearly every English-language DNM and compiled it into an easily searchable database that includes hundreds of mirrored versions of different sites. The entire package, at about 1.6 terrabytes, is sizable, but you can easily search through replicated versions of sites like Silk Road in their entirety as simple (but exhaustive) HTML files.

Branwen’s gone a step further with his data dump, incorporating the same metrics from other researchers including academics and fixtures from within the DNM community. His goal, he explained in a post to r/DarkNetMarkets, a subreddit dedicated to DNMs, was to create a lasting record of the activity (good and bad) that took place during his time in the deep web before he scaled back his own personal activity in DNMs.

“It’s a good archive, I think,” Branwen wrote. “Sorting through it, I refound a lot of things I’d forgotten: the BMR mods chatting on Pastebin about one of them being arrested, unaware that Dutch police were watching them and tapping all their phones and they would be going to jail for years in weeks/months, taking down Utopia with them.”

In his description of the DNM data dump, Branwen describes a number of different things the dataset could potentially be used for, like an analysis of popular drugs being sold at a given point in time or compiling a list of all the people who’d been arrested for their DNM activity. More than anything else, though, Branwen wanted to give the data to the DNM community as a gift—as a means of tapping into a bit of deep web nostalgia.