Anatomy of a page load: DNS lookup
So, you’ve typed that URL in your browser and you press go, what happens next?
How does your device know where to go from just a series of letters, numbers and symbols?
How does all that JS and CSS appear on the page and make it what it is?
We spend so much time on the web, especially if you’re an engineer who works on the front end.
I’m a big believer that not only should we understand what we’re doing, we should also understand our environment and how the picture all fits together.
In this series of posts, I’m going to cover, from a high level, what exactly happens when you press “go” in the URL bar of your browser, all the way up until the page loads and you’re able to start interacting with it. As a caveat I won’t go into huge detail in everything, my aim is to give a nice high-level overview of this process, providing resources where I can.
To start off, we need to answer the question of how, after pressing “go”, your browser finds out where it needs to go to speak with the webserver of the site you’re trying to access and show something.
The URL
Lets take this URL for example:
https://www.example.com/products
It’s a straight forward URL that we likely type into our browsers 100’s of times a day. It’s in a nice human readable (and memorable) format, you know it’s going to take you to the products page at example.com — simple.
Everything in the world that is connected to the internet has it’s own unique identifier, known as an IP address (IPv4 or IPv6 — you can read more about the differences here)- this includes your phone, computer, server or even that WiFi enabled kettle you bought yourself on Black Friday.
The problem here is, when we type a URL into our browser, this human readable format doesn’t mean much to your machine or browser as it needs to know where it should go and open a connection with.
In order to understand how we achieve this connection, we must first understand exactly what a URL consists of.
You can see that this URL consists of 5 parts, each is going to play a part in getting a page loaded in our browser. (There are definitely more than 5 possible parts to a URL, for a more in-depth look you can read through the spec at https://url.spec.whatwg.org).
DNS
Ok, so we have our URL, and we know it’s made of a few distinct parts — how can we get the IP address for it?
Well, we can see our URL as an address, similar to a postal address we use for our homes and businesses.
Postal addresses are constructed of several parts, each part giving us a little more information about the next. i.e. A country helps narrow down our search for a city, a city leads to finding the specific street and the street leads to the house. We can see the house as being our web server.
In order to understand or read this URL/address, we can use the Domain Name System or DNS.
The DNS process has 3 main actors:
- Your browser
- Your OS
- A DNS Resolver
Initially, when you hit go in that URL bar the browser will ask your OS for a DNS resolution. Your OS will then submit a request to its configured DNS resolver, this can be set manually or automatically and is often just your ISP (popular DNS resolvers include the likes of 1.1.1.1 and openDNS).
The resolver now gets on with the hard part, working out an IP for the address it’s been given, it will do this by speaking with a variety of parties.
Resolution
There are 2 main parts of the URL that the resolver cares about, that’s the Top Level Domain or TLD (.com) & Domain Name (example). (There is a slightly extended version of this that includes working out any subdomains, but I’ll skip this for now.)
Thinking back to our address example, we can see each of these parts as being linked together like an address, each part is a server (or set of servers) that contains information on how to get to the next part of the address.
For example, the TLD servers will tell the resolver where it needs to go to get information on the domain name. The resolver will recursively go through this chain until it finds the authoritative name server for the URL being searched for.
The authoritative name server is going to be the place that stores the DNS record for the URL we’re looking for i.e the IP we should give to the browser to connect with.
Getting to the root of it all
The first place a resolver starts are the root servers. These root servers ( https://www.iana.org/domains/root/servers) are placed strategically around the world for performance reasons and contain information for all of the TLDs out there on the internet, these include .gov, .io, .netand the TLD we want, .com.
The resolver will ask the root server where we can find the TLD server for .com and the root server will respond with the IP address for the resolver to continue its search.
Next, the resolver will do the same thing with the TLD server, it will get the IP address for example and will continue following the chain.
Ok, so we have it, the location of the mysterious authoritative name server, so what do we do now?
Well, the authoritative name server contains all the DNS information for the domain in question, this is stored in what is known as a DNS record.
A DNS record may contain many records relating to the DNS resolution process, in our example, we’re interested in just one, the A (or AAAA) record, this record contains the IP address that we’re looking for.
This IP is sent back to the OS by the resolver which is then used by the browser to start the process of showing the web page to the user by establishing a connection with the webserver.
Cache you later
You might now be thinking “Hang on a minute, that’s a lot of stuff going on, how does it all happen so fast?”
Caching is the magic sauce here.
Sure, every time a domain name is resolved for the first time it has to go through all these stages, but after that, both the DNS resolver and your OS will remember where it went the last time (for a certain amount of time, known as the TTL).
Next time you visit that same website your OS will “remember” where it had to go last time so will give your browser that previous IP address, making the DNS process super quick.
The resolver also remembers previous resolutions so even if you’ve never been on a site before it will have access to the IP. For example, if the resolver has been to a .com TLD recently it will totally skip going to the root server to work that one out and likewise for any other domains.
Try it out!
So, this is all great in theory but let’s try it out.
It’s not just browsers that get to do DNS lookups. If you’re on a mac (a Unix based system) you can use the digcommand from the terminal. Using dig does essentially what our DNS resolver was doing above, recursively following the chain of referrals down until we find the authoritative name server, except this time from our own machine.
Try entering this into your terminal. +trace makes sure the command print out the journey the tool took and I’ve added +nodnssec to tidy up the output a little (full list of options can be found here).
dig www.google.com +trace +nodnssecAnd we get a little something like this:
You might notice that for each step in our journey we get multiple results. This is common when it comes to DNS lookups for high traffic sites, each time you do a DNS lookup you’re likely to get given a different one. This is an attempt to distribute traffic between different servers to stop any single one getting overloaded.
Why do I need to know all this?
So how is any of this any use to a FE engineer?
As an FE engineer, everything that happens in the browser should be your “bread & butter”, from the obvious things like javascript, CSS, HTML & web APIs to less known things like DNS resolution and how the browser parses HTML & CSS.
Having a well-rounded knowledge like this will help you with debugging issues, help with creating solutions for problems that are easier to read, more secure, performant and generally give you an overall better understanding of how the web works.
What’s next?
Up next we’ll be looking at how exactly the browser uses that IP address to start communicating with the webserver.
