Scope: basic message examples use HTTP/1.1; telnet responses are historical records, not current site behavior. See RFC 9110 for semantics and RFC 9111 for caching.

Detailed explanation of HTTP protocol

HTTP is a stateless application-layer protocol defining request, response, and representation semantics. HTTP/1.1, HTTP/2, and HTTP/3 are standardized; the text message examples here mainly use HTTP/1.1 and do not directly represent HTTP/2 or HTTP/3 framing.

The main features of the HTTP protocol can be summarized as follows:

• 1. Support client/server mode.
• 2.Requests and responses: clients send a method, target, and fields; servers return a status, fields, and optional content. Performance depends on connection reuse, protocol version, and workload.
• 3. Flexible: HTTP allows the transmission of any type of data object.The type being transferred is marked by Content-Type.
• 4.Connection management: HTTP/1.1 supports persistent connections by default; HTTP/2 and HTTP/3 multiplex streams rather than requiring a new connection for every request.
• 5. Stateless: The HTTP protocol is a stateless protocol.Stateless means that the protocol has no memory ability for transaction processing.The lack of status means that if subsequent processing requires the previous information, it must be retransmitted, which may result in an increase in the amount of data transferred per connection.On the other hand, the server responds faster when it does not need previous information.

1. URL of detailed explanation of HTTP protocol

HTTP/1.1 and HTTP/2 normally use TCP; HTTP/3 uses QUIC over UDP. Stateless semantics and persistent transport connections are separate concepts.

The format of HTTP URL (URL is a special type of URI that contains enough information to find a resource) is as follows:

http://host[":"port][abs_path]

• http means to locate network resources through the HTTP protocol;
• host represents a legal Internet host domain name or IP address;
• port specifies a port number, if it is empty, the default port 80 is used; abs_path specifies the URI of the requested resource;
• If not given in the URL abs_path , then when it is used as the request URI, it must end with “/” Given in the form, usually the browser automatically completes this task for us.

eg:

• 1. Enter: www.guet.edu.cn The browser automatically converts to: http://www.guet.edu.cn/
• 2、http://192.168.0.116:8080/index.jsp

2. Request for detailed explanation of HTTP protocol

HTTP/1.1 messages generally contain a start line, fields, a blank line, and an optional body; the following format descriptions apply to HTTP/1.1.

1. Request line

Begins with a method symbol, separated by spaces, followed by the requested URI and protocol version, in the following format: Method Request-URI HTTP-Version CRLF Among them, Method represents the request method; Request-URI is a uniform resource identifier; HTTP-Version represents the requested HTTP protocol version; CRLF represents carriage return and line feed (except for the CRLF at the end, no single CR or LF characters are allowed).

请求方法（所有方法全为大写）有多种，各个方法的解释如下：
GET     请求获取 Request-URI 所标识资源的表示
POST    请求资源按自身语义处理提交的内容
HEAD    请求获取由Request-URI所标识的资源的响应消息报头
PUT     请求服务器存储一个资源，并用Request-URI作为其标识
DELETE  请求服务器删除Request-URI所标识的资源
TRACE   请求服务器回送收到的请求信息，主要用于测试或诊断
CONNECT 建立到目标服务器的隧道
OPTIONS 查询目标资源或服务器支持的通信选项

Example: Opening a normal HTTP page from the address bar usually issues a GET request:

eg:GET /form.html HTTP/1.1 (CRLF)

The POST method requires the requested server to accept the data attached to the request and is often used to submit forms.

eg：

POST /reg.jsp HTTP/1.1
Host: www.guet.edu.cn
Content-Type: application/x-www-form-urlencoded
Content-Length: 21

user=jeffrey&pwd=1234

HEAD returns no response body. Its fields generally correspond to the equivalent GET response, with some content-dependent fields permitted to be omitted. It can be used to inspect resource metadata.

2. Request header

Request header

3. Request text

Request bodies are not exclusive to POST; PUT/PATCH can also carry them. GET content has no generally defined semantics, so avoid depending on it. Content-Type describes encoding and Content-Length can indicate body length.

3. Response to detailed explanation of HTTP protocol

After receiving and interpreting the request message, the server returns an HTTP response message.

An HTTP/1.1 response contains a status line, fields, a blank line and optional content; HEAD responses and some status codes have no content.

1. The status line format is as follows:

HTTP-Version Status-Code Reason-Phrase CRLF Among them, HTTP-Version represents the version of the server HTTP protocol; Status-Code represents the response status code sent back by the server; Reason-Phrase represents the text description of the status code. The status code consists of three digits. The first digit defines the category of the response and has five possible values:

1xx：指示信息--表示请求已接收，继续处理
2xx：成功--表示请求已被成功接收、理解、接受
3xx：重定向--要完成请求必须进行更进一步的操作
4xx：客户端错误--请求有语法错误或请求无法实现
5xx：服务器端错误--服务器未能实现合法的请求
常见状态代码、状态描述、说明：
200 OK      //客户端请求成功
400 Bad Request  //客户端请求有语法错误，不能被服务器所理解
401 Unauthorized //请求未经授权，这个状态代码必须和WWW-Authenticate报头域一起使用
403 Forbidden  //服务器收到请求，但是拒绝提供服务
404 Not Found  //请求资源不存在，eg：输入了错误的URL
500 Internal Server Error //服务器发生不可预期的错误
503 Server Unavailable  //服务器当前不能处理客户端的请求，一段时间后可能恢复正常
eg：HTTP/1.1 200 OK （CRLF）

2. Response header

Server	服务器名字。Servlet一般不设置这个值，而是由Web服务器自己设置。
content-type 内容类型
location 将客户端重定向到另外一个URL地址，表示客户应当到哪里去提取文档。Location通常不是直接设置的，而是通过HttpServletResponse的sendRedirect方法，该方法同时设置状态代码为302。
content-disposition 指定一个被下载的文件名
content-length 服务器响应内容的字节数
set-cookie 在客户端创建Cookie
content-encoding 指定服务器响应内容的编码方式
Cache-Control 指定缓存策略
expires 用于制定缓存过期时间
ETag 响应表示的验证器；客户端可通过 If-None-Match 发起条件请求
Date 当前的GMT时间。你可以用setDateHeader来设置这个头以避免转换时间格式的麻烦。
Last-Modified 文档的最后改动时间。客户可以通过If-Modified-Since请求头提供一个日期，该请求将被视为一个条件GET，只有改动时间迟于指定时间的文档才会返回，否则返回一个304（Not Modified）状态。Last-Modified也可用setDateHeader方法来设置。
Access-Control-Allow-Origin 标识允许哪个域的请求
Access-Control-Allow-Methods 标识允许哪个方法的请求
Access-Control-Allow-Credentials: true 允许浏览器向脚本暴露带凭证的跨源响应，仍需满足其它 CORS 和 Cookie 条件
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Accept: aWAY+V/uyz5ILZEoWuWdxjnlb7E=

withCredentials affects credentials and associated Cookie handling in cross-origin XHR, not same-origin requests. Credentialed cross-origin access also requires appropriate server CORS configuration and remains subject to browser Cookie policies.

3. The response body is the content of the resource returned by the server.

4. Detailed explanation of HTTP protocol message headers

HTTP messages consist of client-to-server requests and server-to-client responses.Both request messages and response messages consist of a start line (for a request message, the start line is the request line, for a response message, the start line is the status line), a message header (optional), a blank line (a line with only CRLF), and a message body (optional).

HTTP message headers include ordinary headers, request headers, response headers, and entity headers.

Each header field is composed of name + ":" + space + value. The name of the message header field is case-independent.

1. Ordinary header

Among the ordinary headers, there are a few header fields that are used for all request and response messages, but are not used for the transmitted entities, only for the transmitted messages.

eg：

Cache-Control Used to specify cache instructions. The cache instructions are one-way (the cache instructions that appear in the response may not appear in the request) and are independent (the cache instructions of one message will not affect the caching mechanism of another message processing). HTTP1.0 A similar header field used is Pragma. Request directives include no-cache, no-store, and max-age. no-cache requires successful validation before reusing a stored response; it does not prohibit storage. no-store requires that this request and response not be stored. Response cache directives include: public、private、no-cache、no-store、no-transform、must-revalidate、proxy-revalidate、max-age、s-maxage.

eg：

For example, to allow storage but require validation before reuse, set:

response.setHeader("Cache-Control","no-cache");
// Pragma: no-cache is a legacy request directive, not a general response-cache substitute.

If storage must be prohibited, use Cache-Control: no-store instead; these directives are not interchangeable.

The Date common header field indicates the date and time the message was generated.

The Connection common header field allows sending connection-specific options.For example, specify that the connection is continuous, or specify the "close" option to notify the server to close the connection after the response is completed.

2. Request header

Request headers allow the client to pass additional information about the request as well as information about the client itself to the server.

Common request headers

Accept

The Accept request header field is used to specify what types of information the client accepts.e.g.: Accept：image/gif , indicating that the client wishes to accept resources in GIF image format; Accept：text/html , indicating that the client wishes to accept html text.

Accept-Charset

The Accept-Charset request header field is used to specify the character set accepted by the client.eg: Accept-Charset:iso-8859-1,gb2312.If this field is not set in the request message, the default is that any character set is accepted.

Accept-Encoding

Accept-Encoding lists acceptable content codings, such as Accept-Encoding: gzip, deflate; absence does not mean only unencoded content is accepted.

Accept-Language

The Accept-Language request header field is similar to Accept, but it is used to specify a natural language.eg: Accept-Language:zh-cn. If this header field is not set in the request message, the server assumes that the client can accept various languages.

Authorization

The Authorization request header field is mainly used to prove that the client has the right to view a certain resource.When the browser accesses a page and receives a response code of 401 (Unauthorized) from the server, it can send a request containing the Authorization request header field to ask the server to verify it.

Host (this header field is required when sending a request)

The Host request header field is mainly used to specify the Internet host and port number of the requested resource. It is usually extracted from the HTTP URL, eg:

We enter in the browser: http://www.guet.edu.cn/index.html

The request message sent by the browser will contain the Host request header field, as follows:

Host：www.guet.edu.cn

The default port number 80 is used here. If a port number is specified, it becomes: Host：www.guet.edu.cn :Specify port number

User-Agent

When we log in to the forum online, we often see some welcome messages, which list the name and version of your operating system and the name and version of the browser you are using. This often makes many people feel amazing. In fact, the server application obtains this information from the User-Agent request header field.The User-Agent request header field allows the client to tell the server about its operating system, browser, and other attributes.However, this header field is not necessary. If we write a browser ourselves and do not use the User-Agent request header field, then the server will not be able to know our information.

Example request header:

GET /form.html HTTP/1.1 (CRLF)
Accept:image/gif,image/x-xbitmap,image/jpeg,application/x-shockwave-flash,application/vnd.ms-excel,application/vnd.ms-powerpoint,application/msword,*/* (CRLF)
Accept-Language:zh-cn (CRLF)
Accept-Encoding:gzip,deflate (CRLF)
If-Modified-Since:Wed,05 Jan 2007 11:21:25 GMT (CRLF)
If-None-Match:W/"80b1a4c018f3c41:8317" (CRLF)
User-Agent:Mozilla/4.0(compatible;MSIE6.0;Windows NT 5.0) (CRLF)
Host:www.guet.edu.cn (CRLF)
Connection:Keep-Alive (CRLF)
(CRLF)

3. Response header

Response headers allow the server to pass additional response information that cannot be placed in the status line, as well as information about the server and further access to the resource identified by the Request-URI.

Common response headers

Location

The Location response header field is used to redirect the recipient to a new location.The Location response header field is often used when changing domain names.

Server

The Server response header field contains information about the software used by the server to handle the request.Corresponds to the User-Agent request header field.

Here is an example of a Server response header field:

Server：Apache-Coyote/1.1
WWW-Authenticate
401 响应应包含适用的 WWW-Authenticate 认证挑战；客户端随后可使用 Authorization 提交凭证。

eg：

WWW-Authenticate:Basic realm="Basic Auth Test!"  //可以看出服务器对请求资源采用的是基本验证机制。

4. Entity header

Both request and response messages can convey an entity.An entity consists of the entity header field and the entity body, but this does not mean that the entity header field and the entity body must be sent together. Only the entity header field can be sent.The entity header defines meta-information about the entity body (eg: presence or absence of an entity body) and the resource identified by the request.

Commonly used entity headers

Content-Encoding

Content-Encoding The entity header field is used as a media type modifier, and its value indicates the encoding of additional content that has been applied to the entity body, thus obtaining Content-Type The media type referenced in the header field must use the corresponding decoding mechanism. Content-Encoding This is the compression method used to record documents, eg: Content-Encoding：gzip

Content-Language

The Content-Language entity header field describes the natural language used by the resource.If this field is not set, it is assumed that the entity content will be available for reading in all languages. who.eg: Content-Language:da

Content-Length

The Content-Length entity header field is used to indicate the length of the entity body, expressed as a decimal number stored in bytes.

Content-Type

The Content-Type entity header field specifies the media type of the entity body sent to the recipient.

eg：

• Content-Type:text/html;charset=ISO-8859-1
• Content-Type:text/html;charset=GB2312
• Last-Modified
• The Last-Modified entity header field is used to indicate the last modified date and time of the resource.

Expires

The Expires entity header field gives the date and time when the response expires.In order to allow the proxy server or browser to update the pages in the cache after a period of time (when accessing the previously visited page again, load it directly from the cache to shorten the response time and reduce the server load), we can use the Expires entity header field to specify the time when the page expires.

eg：Expires：Thu，15 Sep 2006 16:23:12 GMT

HTTP1.1 clients and caches MUST treat other illegal date formats (including 0) as expired.eg: In order to prevent the browser from caching the page, we can also use the Expires entity header field, set to 0, and program in jsp

As follows:

response.setDateHeader("Expires", 0L);

5. Use telnet to observe the communication process of http protocol

Experiment purpose and principle:

Use MS's telnet tool to send a request to the server by manually inputting the http request information. After the server receives, interprets and accepts the request, it will return a response, which will be displayed on the telnet window, thus deepening the understanding of the communication process of the http protocol from a perceptual perspective.

Experimental steps:

• 1. Open telnet

• 1.1 Open telnet Run -->cmd-->telnet

• 1.2 Turn on telnet echo function set localecho

• 2. Connect to the server and send a request

• 2.1 open www.guet.edu.cn 80 //Note that the port number cannot be omitted

  HEAD /index.asp HTTP/1.0Host:www.guet.edu.cn

  /We can change the request method and request the content of Guilin Electronics homepage, and enter the message as follows/ open www.guet.edu.cn 80

  GET /index.asp HTTP/1.0 //Request resource content Host:www.guet.edu.cn

• 2.2 open www.sina.com.cn 80 //Input telnet www.sina.com.cn 80 directly at the command prompt HEAD /index.asp HTTP/1.0 Host:www.sina.com.cn

• 3 Experimental results:

• 3.1 Request information 2.1 and get the response:

HTTP/1.1 200 OK                                              //请求成功
Server: Microsoft-IIS/5.0                                    //web服务器
Date: Thu, 08 Mar 2007 07:17:51 GMT
Connection: Keep-Alive
Content-Length: 23330
Content-Type: text/html
Expires: Thu, 08 Mar 2007 07:16:51 GMT
Set-Cookie: ASPSESSIONIDQAQBQQQB=BEJCDGKADEDJKLKKAJEOIMMH; path=/
Cache-control: private

//资源内容省略

• 3.2 The response obtained by requesting information 2.2 is:

HTTP/1.0 404 Not Found       //请求失败
Date: Thu, 08 Mar 2007 07:50:50 GMT
Server: Apache/2.0.54 <Unix>
Last-Modified: Thu, 30 Nov 2006 11:35:41 GMT
ETag: "6277a-415-e7c76980"
Accept-Ranges: bytes
X-Powered-By: mod_xlayout_jh/0.0.1vhs.markII.remix
Vary: Accept-Encoding
Content-Type: text/html
X-Cache: MISS from zjm152-78.sina.com.cn
Via: 1.0 zjm152-78.sina.com.cn:80<squid/2.6.STABLES-20061207>
X-Cache: MISS from th-143.sina.com.cn
Connection: close

Lost connection to host

Press any key to continue...

Things to note:

1. If an input error occurs, the request will not succeed. 2. The header fields are not case-sensitive. 3. RFC 2616 has been superseded; use RFC 9110 linked at the start for HTTP semantics. 4. When developing background programs, you must master the http protocol

6. Technical supplements related to HTTP protocol

1. Basics:

High-level protocols include: File Transfer Protocol FTP, Email Transfer Protocol SMTP, Domain Name System Service DNS, Network News Transfer Protocol NNTP and HTTP protocols, etc.

There are three types of intermediaries: proxy (Proxy), gateway (Gateway) and channel (Tunnel). A proxy accepts requests according to the absolute format of the URI, rewrites all or part of the message, and sends the formatted request to the server through the URI identifier.A gateway is a receiving proxy that acts as a layer above some other server and, if necessary, can translate requests to the underlying server protocol.A channel acts as a relay point between two connections that do not change messages.Channels are often used when communication needs to go through an intermediary (such as a firewall, etc.) or when the intermediary cannot identify the content of the message.

Proxy (Proxy): An intermediate program that can act as a server or a client, establishing requests for other clients.Requests are passed internally or via other servers via possible translations.A proxy must interpret and if possible rewrite a request message before sending it.A proxy often acts as a portal for clients through a firewall. A proxy can also serve as a helper application to handle requests over a protocol that are not completed by the user agent.

Gateway (Gateway): A server that acts as an intermediary between other servers.Unlike a proxy, a gateway accepts requests as if it were the origin server for the requested resource; the requesting client is unaware that it is dealing with the gateway.

A gateway often acts as a server-side portal through a firewall. A gateway can also act as a protocol translator to access resources stored in non-HTTP systems.

Channel (Tunnel): is an intermediary program that acts as a relay between two connections.Once activated, the channel is not considered part of HTTP communication, although the channel may have been initiated by an HTTP request.When both ends of the relayed connection are closed, the channel disappears.Channels are often used when a portal (Portal) must exist or an intermediary (Intermediary) cannot interpret the relayed traffic.

2. Advantages of protocol analysis—HTTP analyzer detects network attacks

Analyzing and processing high-level protocols in a modular manner will be the direction of future intrusion detection. Common ports 80, 3128 and 8080 for HTTP and its proxies are specified in the network section using the port tag

3. Request content length and resource limits

Content-Length indicates the content length in bytes. Resource exhaustion depends on server size limits, read strategy, timeouts and concurrency controls; HTTP does not mandate buffering all content or make attacks untraceable.

4. Some ideas for using the characteristics of HTTP protocol to carry out denial of service attacks

SYN floods, IP fragmentation attacks and UDP reflection occur at different network layers and are not all HTTP flaws. Their impact on a web service depends on network and server resources, not a requirement to run a particular historical service.

5. HTTP fingerprint identification technology

The principle of Http fingerprinting is roughly the same: recording the subtle differences in the implementation of Http protocol by different servers to identify .Http fingerprinting is much more complicated than TCP/IP stack fingerprinting. The reason is that customizing the Http server configuration file and adding plug-ins or components make it easy to change the Http response information, which makes identification difficult; however, customizing the behavior of the TCP/IP stackThe core layer needs to be modified so it is easy to identify.

It is very simple to set up the server to return different Banner information. For open source Http servers like Apache, users can modify the Banner information in the source code, and then restart the Http service to take effect; for Http servers without open source code, such as Microsoft's IIS or Netscape, you can modify it in the Dll file that stores the Banner information.Modification has been discussed in related articles, so I won’t go into details here. Of course, the effect of such modification is still good. Another way to blur Banner information is to use plug-ins.

Common test requests:

1. HEAD / HTTP/1.0 is a basic request line; complete the fields and blank line as required.
2. Whether DELETE / HTTP/1.0 is allowed depends on server policy.
3. GET / HTTP/9.9 illustrates an invalid-version test; HTTP/3 uses binary frames and cannot be simulated by a text line.
4. GET/JUNK/1.0 lacks the required spaces and illustrates malformed syntax.

The Http fingerprint identification tool Httprint can effectively determine the type of Http server by using statistical principles and combining fuzzy logic techniques. It can be used to collect and analyze signatures generated by different Http servers.

6. Others

In order to improve the performance of users when using the browser, modern browsers also support concurrent access methods. When browsing a web page, multiple connections are established at the same time to quickly obtain multiple icons on a web page, which can complete the transmission of the entire web page more quickly.

HTTP/1.1 defaults to persistent connections. HTTP/2 and HTTP/3 change message transport further; historical HTTP-NG research is not the current next-generation standard.

HTTP/0.9 simple requests retrieved documents with GET and lacked modern fields. HTTP/1.0 introduced fields and status lines; parallel connections are a client policy. HTTP/1.1 defaults to persistence and permits pipelining. Six connections per domain is not a protocol rule; ordered pipelined responses still cause application-level head-of-line blocking. HTTP/2 uses binary frames, multiplexing and HPACK. Only pseudo-fields start with a colon, not regular fields. TCP head-of-line blocking remains. See RFC 9113. HTTP/3 uses QUIC to avoid loss on one stream directly blocking ordered delivery on another. A stream can still await retransmission, and congestion control is shared. See RFC 9114.

head-of-line blocking

Head-of-line blocking ( head-of-line Blocking) occurs when a TCP segment is lost, causing subsequent segments to arrive at the receiving end out of order.The subsequent segments will be retained by the receiving end until the first lost segment is retransmitted by the sending end and reaches the receiving end.This delayed delivery of subsequent segments ensures that the receiving application process receives the data in the order in which it was sent by the sender.This delay mechanism introduced to achieve complete ordering is very useful, but it also has disadvantages. Assuming that semantically independent messages are sent over a single TCP connection, for example, the server may send 3 different images for the web browser to display.In order to create the effect of these images being displayed side by side on the user's screen, the server first sends a fragment of the first image, then a fragment of the second image, and then a fragment of the third image; the server repeats this process until all three images are successfully sent to the browser. If a TCP segment of a fragment of the first image is lost, the client will keep all the data that has arrived out of sequence until the lost segment is retransmitted successfully.This not only delays the delivery of the first image data, but also delays the delivery of the second and third image data.

Pipeline

What is HTTP pipelining? HTTP 1.1 allows the optional use of request pipes over persistent connections.This is another performance optimization compared to keep-alive connections.Before the response arrives, multiple requests can be put into the queue. When the first request is sent to the server, the second and third requests can also start to be sent. Under high-latency network conditions, this can reduce the loopback time of the network and improve performance.