A recent article in the Economist magazine “Shedding light on the dark web” brought to light the work of Gwern Branwen (reportedly a pseudonym). Branwen as a researcher decided to collect information on the dark net. According to the article, roughly once a week from December 2013 through July 205 he crawled 90 different Dark net market places (including Agora, Evolution and Silk Road 2) and archived a snapshot of each page. The Economist reports that the data collection is 1.5 terabytes of data. Included in the various 360,000 sites is information on the items sold, the Bitcoin price of the item, the date of the sale, shipping information, customer ratings and the vendor’s pseudonym. On his website Gwern says “I scraped/mirrored on a weekly or daily basis all existing English-language DNMs as part of my research into their usage, lifetimes/characteristics, & legal riskiness”.
Wow, that is a heck of a data set to crawl through. Anyone investigating crimes on the dark net finally has some historical data with which to do research. This can provide investigators with some valuable information as to targets and suspects. The article admits that the collection was not everything on those sites and excluded certain data.. Still this is a treasure trove of information not previously available to researchers and law enforcement. Gwern also states on his website that the data set contains various vendor PGP keys, username (even clearnet names), and email addresses.
You can find the complete archive at https://www.gwern.net/Black-market%20archives. Or by going to the ever popular Archive.org where they have a Torrent link to download the data https://archive.org/download/dnmarchives/dnmarchives_archive.torrent