The requests are reverse proxied to a backend server and the corresponding request is logged there with status 408 (Request Timeout). The front end server (Apache. To extract and log the actual client IP address from the X-Forwarded-For header of a request using an Apache server, make the following changes to the server: Log into the Apache server. Go to /etc/httpd/conf or /usr/local/apache2/conf path and open the file httpd.conf. Be aware that there are already a number of 400/408 exceptions that will only appear in the access log today by default at LogLevel Info, and that appears to be true of the earlier revisions. So the suggestion to ignore the access log isn't really viable. Since it was deployed 4 years ago, apache has been running flawlessly. I checked my accesslog and sslaccesslog and it really seemed to start right after that specific update. I checked and there was not hint of a 408 errors during the previous month. I downgraded httpd with yum history undo and the 408 errors stopped flooding.
What is a 408 Request Timeout Error
This error means the server timed out waiting for the client after the client has initiated a request. From the W3 HTTP specs: 'The client did not produce a request within the time that the server was prepared to wait. The client may repeat the request without modifications at any later time.' Also see RFC2616.
Are 408 Request Timeout Errors a Problem?
Not necessarily, and 408 errors may not be indicative of a larger issue. In many cases 408 errors are just connections that hold Apache open for longer than allowed based on the timeout settings in the web server's configuration files.
Apache Access Log View
If Apache never enforced any timeout settings to close connections where the client has not communicated in a certain amount of time, then a single bad actor could flood the server with connections and not allow anyone else to connect.
In some cases these 408 errors come from systems looking for exploits. In recent years link previews and link prefetching have become popular and can also cause 408 errors as the services that implement such link previews (think Slack, social media sites, etc.) do not respect the standards and may leave server connections hanging after receiving the data they need (frequently the og-image, title, and description for the link preview). And link prefetching may just make the initial connection request prior to the user actually clicking the link, so a connection will be initiated on the server side which is left to die on the server side if the user never actually clicks the pre-fetched link.
Required reading about such problems with Google Chrome's prefetch implementation:
Related Apache Configuration Settings:
Related Apache Modules:
Apache Access Log 408
Slow Loris - if client connections are not timed out after a reasonable interval, an attacker can attempt to max out connection slots to the web server. Duck Duck Go for more info and ways to mitigate, and how to scan log files to identify possible attackers by IP address.
Sources and Related Resources
- 'http-status-code-408' tag wiki - Stack Overflow
Apache Access Log 408 Login
- Reverse Proxy Intermittant 408 Time Out Errors - Forum - Hiawatha webserver
- apache2 - A lot of 408 errors in apache logs - how to prevent them? - Webmasters Stack Exchange
- apache2 - Understanding “408 Request Timeout” on Apache with PHP - Stack Overflow