The HTTP Protocol explained!
Hypertext Transfer Protocol, also known as HTTP is the communication protocol used by all of us daily when we access a website, in particular the World Wide Web.
Like many other protocols, HTTP is based on a client-server architecture. The client is the user, in this case, us, or the browser, and the server is the web server in which the website is hosted.
The browser regenerates a request in HTTP and sends it to the server that in turn replies with an HTTP response to the client (browser) based on the request received. HTTP usually runs on top of the transport protocol TCP/IP on port 80.
As previously mentioned, the HTTP protocol communication consists of clients performing requests and servers reply accordingly. Let’s analyze what makes an HTTP request and an HTTP response. It is important to mention that both request and response are composed of a header and a body section that are separated by a blank line.
HTTP Request and Reply Headers
GET / HTTP/1.1
Host: mregraoncyber.com
User-Agent: Chrome/87.0.4280.88
Referer: https://www.google.com/
Accept-Encoding: gzip, deflate
As you can see in the example above we have a request sent by me through the Burp browser. The associated reply is:
HTTP/1.1 200 OK
Date: Sun, 14 Mar 2021 13:34:56 GMT
Content-Type: text/html; charset=UTF-8
Connection: close
Vary: Accept-Encoding
Server: cloudflare
Content-Length: 506
<html><body>Yoo</body></html>
This is an example of an HTTP communication, with a request and a reply. Let’s try to see if we can understand what it is happening here.
The request:
GET / HTTP/1.1
As you can see, in the first line of the request we have the GET method being requested. The resource being request is / which basically is the home directory of my website, therefor the home page. And the HTTP version is HTTP 1.1.
Host: mregraoncyber.com
This field is the Host field and it declares the host to which the client is making the request to. This host will than be translated to the associated IP.
User-Agent: Chrome/87.0.4280.88
The User-Agent filed provides the server information about the browser that is being used to make the request.
Referer: https://www.google.com/
The Referer lets the server know the site from which the request originated from. As you can see there is a typo, it should be Referrer, just a funny remark.
Accept-Encoding: gzip, deflate
This field, the Accept-Encoding lets the server know the type of encoding supported by the browser for the communication.
The response:
HTTP/1.1 200 OK
This header section indicates the code status. In this case we have 200 OK which means the request was processed with errors.
Date: Sun, 14 Mar 2021 13:34:56 GMT
This is a timestamp of the response.
Content-Type: text/html; charset=UTF-8
This header indicates the Content-Type, in this case the file is text/html and uses UTF-8 as charset.
Connection: close
This header indicates whether the connection should remain open or be closed. In this example it is to close.
Vary: Accept-Encoding
This method is used to state whether the server accepts the suggested encoding formats. In this case it is a way to compress the information sent over the network to improve performance.
Server: cloudflare
Indicates the version of the server that is replying to the client, in this case is cloudflare.
Content-Length: 506
Indicates the length of the message body for this response.
There are several other methods for the HTTP protocol, such has GET, POST, HEAD, TRACE, etc. And there are also several HTTP request arguments, like Content-Length, Connection, among several others. With time, practice and continuous learning you will master them all!
HTTP Response Body
We have analyzed the headers of a request example. Now let’s analyse the body. The body has the HTML code that is rendered on the screen by the browser. In the example above the body is:
<html><body>Yoo</body></html> It is a very simple HTML, just for the example. A real website would certainly have a more complex body.
HTTP Protocol diagram
The HTTP uses the client-server architecture. As you probably know, this is a centralized approach in which a central server provides a service to a set of clients that interact with the server by performing requests. This type of architecture has its advantages and disadvantages both regarding performance and security. One real issue with HTTP centralization is the fact that if the server gets attacked by a DDoS attack, it could become offline and unable to provide the service to its clients. On the other hand, the fact that the service is centralized it is easy to manage and to protect the confidentiality of the user’s data because only the central server (should) have access to the information and no external unknown parties. The HTTP Client, is usually the web browser, that we use daily, such as Google Chrome, Firefox, Opera, among several others. The HTTP Server is the Web Server that is hosting the content that we want to access. One example is a web page, like this blog that you are reading now.
HTTP Protocol for security enthusiasts and Bug Hunters
This is one of the most important protocols for web security enthusiasts. This is the most widely used protocol on the Web. One very important tool that you probably should use to learn/manipulate this protocol and the associated requests is Burp Suite. There are others that you can use, but Burp is the one I am learning and the one it is widely recommended and used (I believe). Burp works as a proxy tool and allows you to see/manipulate HTTP requests. It is a great tool for Bug Hunters and allows you to better understand this protocol.
You can find several examples in this blog of CTF challenges that I solved in which I had to manipulate or understand in some way the HTTP protocol, a few examples are:
Like everything in life practice makes perfect and it allows us to understand what something is and what it is used for. I like CTFs because they help me with just that! Take a look at some of my writeups, hope you can find value in them. I will be posting more regularly.
Further reading about HTTP
HTTP is a complex and large protocol. It is being improved and changed regularly making it difficult to write in one post everything. I just hope I was clear and able to teach you something about HTTP. If you are curious and want to learn more about it, read more here. It is an amazing HTTP manual that I use every time I need to check something about HTTP.
Conclusion
HTTP is the protocol that is most widely used by a lot of people daily when they access the World Wide Web. This protocol allowed the development and growth of the Internet as we know and use it today. It is constantly being improved, and we soon should have a new version the QUIC, which is HTTP based on UDP instead of TCP. A subject for another post probably.
Keep on reading, learning and stay curious! HTTP is the back bone of the web, for security enthusiasts is a most know protocol, in particular if you want to learn about web security. Stay tuned for more posts and take a look at some of my web exploitation posts here.
Thank you very much for reading!
Cheers,
MRegra
Hey there would you mind sharing which blog
platform you’re working with? I’m looking to start my own blog soon but I’m having
a difficult time selecting between BlogEngine/Wordpress/B2evolution and Drupal.
The reason I ask is because your layout seems different then most blogs
and I’m looking for something unique.
P.S Apologies for being off-topic but I had to ask!
Hello, thanks for the feedback.
I will be writing a post about it soon.
Have a nice day,
Cheers,
MRegra
What’s up, after reading this amazing post i am as well glad to share my knowledge here with friends.