Home » Education, Network Security » Deep Web, Dark Web, Darknet, and Dark Internet

Deep Web, Dark Web, Darknet, and Dark Internet

August 4th, 2012 Go to comments

The terms Deep Web, Dark Web, Darknet, and Dark Internet are ones I see confused and misused on a regular basis on the Internet and in the media. This is my attempt to rectify this confusion and misuse by explaining what each of these terms means and when you should use them.

Deep Web

The Deep Web is quite simply any content on the Web which is not accessible to or indexed by standard search engine spiders. A search engine spider will typically crawl a website by visiting it and then visiting all the pages it links to, which includes pages local to the site and pages on other sites. Whilst this gives the search engine a pretty good view of the web, it misses out on a lot of other resources for various reasons:

  • Standard search engine spiders do not try to log into any websites, so any resources protected by a login are not accessible to it.
  • Content which explicitly denies access to search engine spiders (e.g. using a robots.txt file) is also left off the search engine index.
  • A web server may host a file or directory of files that isn’t linked to anywhere on the web. These files and directories would be missed by search engines as they would (most likely) be by humans too.
  • Content that requires input by a user to be generated (i.e. search results) may also be effectively invisible to search engine spiders.
  • Some websites may require a special browser configuration to gain access.

You can think of the web as an ocean of content. Anything on the surface of this ocean is content that is being linked to openly. A search engine spider can only look at the content on the surface of the ocean, and any content in the deeper parts of the ocean (whether protected by a login, or just hidden from view) is inaccessible to it.

What it is important to remember is that the Deep Web has nothing necessarily to do with illegal activity, nor is it about being anonymous or hiding your identity. Most of us access the Deep Web on a regular basis, whenever we check web mail, or log in to a social networking site. If a search engine can’t see it, for whatever reason, it’s part of the Deep Web.

Dark Web

Conversely, the Dark Web does have numerous links to illegal activity and hiding one’s identity. It is a collection of websites that are only accessible over the Tor network, which hides your IP address and gives you complete anonymity. Not every website accessed over Tor is part of the Dark Web, since Tor allows you to browse anonymously on the regular web as well. However, the Tor network has a special pseudo-top-level domain suffix called “.onion” which is used to get to websites which host themselves over Tor, and are therefore only accessible via Tor.

Going to these websites without using a browser configured to use Tor is impossible, so the Dark Web is actually a subset of the Deep Web, and as such is not indexed by search engines. Whilst there are many websites on the Dark Web which do not promote illegal activity, there are plenty that do, including sites that sell drugs and weapons. A BBC report earlier this year highlighted the Dark Web quite well, and the hacktivist group Anonymous have attacked pedophilia-related websites on the Dark Web before.

Darknet

Wikipedia asserts that a darknet is a “private, distributed P2P filesharing network, where connections are either made only between trusted peers using non-standard protocols and ports or using onion routing.” Limiting the term to certain types of filesharing network is unhelpful in my opinion, and I see no reason a darknet cannot simply be any such network. This would make the onion routed part of the Tor network itself a darknet, and it is often called “The Darknet” (though there is more than one darknet, the onion routed part of the Tor network is still the most well known).

This too would make the Dark Web a part of the Darknet. However, it is important to point out that the Dark Web and the Darknet are not synonymous. Many other services can run on the Darknet, such as email, IRC, etc. The Dark Web is just one of these services, contributing a subset of traffic over the Darknet.

So a darknet (no capitalisation) is any network where connections are made only between trusted peers using non-standard protocols and ports or using onion routing. The Darknet (capitalised) is the onion routed part of the Tor network. This means that the Darknet is a darknet, in the same way as the Internet is an internet.

To make matters slightly more confusing, Project Meshnet used to be known as the “Darknet Plan”, though luckily the name was changed to more accurately reflect the nature of their project (and possibly to alleviate confusion).

Dark Internet

Finally, we end with a term which is completely unrelated to the three above, yet still manages to get confused with them. The Dark Internet refers to the unreachable network hosts on the Internet. They could be unreachable because a machine is turned off, or a network cable is damaged, or even because routing tables have become corrupted somewhere. Nobody, not even regular Internet users, can reach them. The Dark Internet is constantly changing; machines get taken offline, and some get put back online, but whilst they are offline, they are part of the Dark Internet.