Apache Reverse Proxy 503 Service Unavailable

Posted on  by admin

Before this change, typing takes me to the Tomcat administration page; after it, I get '503 Service Unavailable'. I've temporarily turned off my firewall and removed AppArmor. Here's the relevant portion of modjk.log. Its functionality can be extended through modules that suit a wide range of use cases, from serving dynamic PHP content to acting as a forward or reverse proxy. The Apache HTTP Server Project is based on Rob McCool’s work on the HTTPd web server for the National Center for Supercomputing Applications (NCSA).

Greenhorn
posted 4 years agoApache Reverse Proxy 503 Service Unavailable
  • Optional 'thank-you' note:
I'm following this guide to setup Tomcat 8 on Ubuntu Server 16.04 using Apache2's mod_jk module as a reverse proxy:
https://www.digitalocean.com/community/tutorials/how-to-encrypt-tomcat-8-connections-with-apache-or-nginx-on-ubuntu-16-04
Everything works until the last step, which is to change the HTTP and AJP Connectors in server.xml to only listen on localhost. Here's the change I made to the AJP Connector (added localhost in the address):

Before this change, typing https://myhostname takes me to the Tomcat administration page; after it, I get '503 Service Unavailable'.
Here's the relevant portion of mod_jk.log
Saloon Keeper
posted 4 years ago
  • Optional 'thank-you' note:
You might want to check the Tomcat catalina.out and localhost log files to make sure that port 8009 didn't fail to open.
Although your proxy log message seems to be indicating that Apache might have been trying to connect using IPV6 and your allowable address (127.0.0.1) is an IPv4 address. If that's the case, I'd try and ensure that the Apache-to-Tomcat tunnel on port 8009 used IPv4, since I don't thing IPv6 gains you a lot on a loopback connection.

Some people, when well-known sources tell them that fire will burn them, don't put their hands in the fire.
Some people, being skeptical, will put their hands in the fire, get burned, and learn not to put their hands in the fire.
And some people, believing that they know better than well-known sources, will claim it's a lie, put their hands in the fire, and continue to scream it's a lie even as their hands burn down to charred stumps.

Greenhorn
posted 4 years ago
  • Optional 'thank-you' note:
Here are the logs based on your suggestion, it doesn't seem like there is an issue with port 8009 starting as line 36 of the catalina.out says this 'Initializing ProtocolHandler ['ajp-nio-127.0.0.1-8009']' but I'm not 100% sure:
Also, how do I ensure that the Apache-to-Tomcat tunnel on port 8009 uses IPv4. I didn't see anything I could change in the server.xml connector config to specify IP Type??
Apologies, I'm new to configuring tomcat and this is a bit of a learning process for me.
Thanks much!
Localhost.log:

catalina.out after running systemctl restart tomcat as root

Saloon Keeper
posted 4 years ago
  • Optional 'thank-you' note:

Apache Reverse Proxy Https

You can use the command 'netstat -tnlp' to list what programs are listing on which ports/protocols.
But it does appear that Apache is attempting to use IPv6 to communicate and your listening address on the connector is set for IPv4.
Unfortunately, Apache does its IPv6/IPv4 things with a certain amount of 'magic' these days, so it's not always easy to tell what protocols it's going to use. You'll probably have to read the Apache docs. I would expect that the mod_jk configuration options would deal with that, but unfortunately, the DigitalOcean tutorial doesn't show a functional and complete set of sample mod_jk directives
It's not totally essential that you limit the port 8009 listening address. That's not a public port, so access to it should be firewall-blocked both at the external entry to your site from the Internet, and in your case, on the local machine (iptables) itself. If someone managed to get far enough into your server to be able to send local requests inside the machine's firewall, you've probably already be totally eaten.
Also, are you the 'Bostich' that posted to the DigitalOcean site not long ago? We really would like to be notified when you cross-post stuff. Otherwise people get confused because sometime they'll answer on one site and sometimes on another.

Some people, when well-known sources tell them that fire will burn them, don't put their hands in the fire.
Some people, being skeptical, will put their hands in the fire, get burned, and learn not to put their hands in the fire.
And some people, believing that they know better than well-known sources, will claim it's a lie, put their hands in the fire, and continue to scream it's a lie even as their hands burn down to charred stumps.

Greenhorn
posted 4 years ago
  • Optional 'thank-you' note:
Tim,
Thank you for your clarifications! Yes that post on DO is mine.

Web servers such as LiteSpeed and Apache use various codes to tell browsers about responses. For example, if the webserver replies back with HTTP code 200, it means that everything is fine and the response generation was successful. There are many other response codes, but today we will discuss HTTP Error 503, and when this usually happens you get the following error on your browser: HTTP Error 503. The service is unavailable.

If you are not the administrator of the site, there is nothing much you can do as mostly this error indicates that there is something wrong on the server-side. You can either refresh the page, visit later or better notify an administrator of the site. However, if you are the administrator of the site, you can do much to figure out what is wrong and there are various reasons and ways to fix this error. There are multiple web servers, each may give you a slightly different error message, such as:

  1. 503 Error
  2. Http/1.1 Service Unavailable
  3. 503 Service Temporarily Available
  4. 503 Service Unavailable
  5. HTTP Error 503
  6. Service Unavailable – DNS Failure
  7. Error 503 Service Unavailable

Usually, the main thing to look for is error code which is HTTP error code 503. Today we will see how we can discuss various reasons and respective ways to fix the issue.

Server Side Issue

Before deep-diving into various reasons as to why this could happen, I would again like to mention that this is a server-side issue. All errors in the 5xx range are considered errors on the server side including 503 Service Unavailable Error. However, do keep in mind that 503 error means the server was able to process your web request and it was functioning properly but it chooses to return 503 error code because due to some problem/issue server is not able to process this request the way it should have.

Some times you will get the following error

Apache Reverse Proxy Headers

503 Service Unavailable – The server is temporarily busy, try again later!

It may really be a temporary error as the error message says, so wait some time and refresh the page. This can happen to high traffic sites, where enough resources are not available to handle the request. On the user end, be careful if you are seeing this error on the payment related pages, and make sure you won’t get charged twice.

If you are our customer and using our Apache as Backend feature, there is a chance that Apache is down. In this case frontend server which is OpenLiteSpeed, will give you a 503 error, as it is failed to connect to Apache. See if Apache is running

systemctl status httpd

If Apache service is not running, you can start with

systemctl start httpd

And see if your issue is resolved.

PHP FPM is down

If you get something like (Assuming you are using our Apache as backend feature or your stack includes PHP-FPM)

The server is temporarily unable to service your request due to maintenance downtime or capacity problems. Please try again later.

This means your PHP-FPM service is down. You can start php-fpm using

systemctl start php-fpm

In case you are our customer, there are multiple PHP-FPMs are available for different PHP versions and their respective commands are:

systemctl start php54-php-fpm

systemctl start php55-php-fpm

systemctl start php56-php-fpm

systemctl start php70-php-fpm

systemctl start php71-php-fpm

systemctl start php72-php-fpm

Reverse

systemctl start php73-php-fpm

If your issues are still not resolved, you can start to check various log files. In the case of CyberPanel and LiteSpeed (OpenLiteSpeed), log files to check are:

/usr/local/lsws/logs/error.log

/usr/local/lsws/logs/stderr.log

Unavailable

Apache Reverse Proxy 503 Service Unavailable Free

In the case of Apache

Apache reverse proxy https

/etc/httpd/logs/error_log

You can do an efficient search of log files using the grep command such as:

cat log_file_path grep error

cat log_file_path grep notice

This command will make sure that you only get the most relevant information, otherwise, you may also get info level messages in your log files and they are not relevant in this case.

Most of the time 503 errors can come due to a problem in your PHP code, or either PHP-FPM/LSPHP are not able to produce response thus server started giving you 503 error. It is always recommended to first create a phpinfo page and see if your PHP side is working fine. If you can see the phpinfo page, you can move forward to further debug the cause, otherwise, make sure LSPHP is working fine and external application is created properly or PHP-FPM is up and running.

Disable PHP OPCode Caching (xCache, ACP or eAccelerator)

On a default install of CyberPanel (OpenLiteSpeed or LiteSpeed Enterprise), OPCode caching is enabled. Sometimes different sorts of opcode caching can have compatibility issues with LSPHP (PHP). So if you are getting HTTP Error 503 Error better try to disable opcode caching. On CyberPanel you first need to find out which PHP version is used by your site. To find out your PHP version on CyberPanel run the following command

cat /usr/local/lsws/conf/vhosts/yourdomain.com/vhost.conf grep php

For example, the PHP version of your site is 7.2. Go to the configurations directory of PHP 7.2 and disable opcode caching.

cd /usr/local/lsws/lsphp72/etc/php.d

mv 10-opcache.ini 10-opcache.ini.bak

systemctl restart lsws

or

/usr/local/lsws/bin/lswsctrl restart

This will disable OPCode caching. If you are not on CyberPanel, you need to find the php.ini file for your PHP and disable OPCode caching. Usually, php.ini location is disclosed in phpinfo page. If your issue is still not resolved, you can move on to the next step.

No space left on /tmp

Some web application use /tmp directory to store temporary files (session data etc). If /tmp is full you can get HTTP Error 503 Error. Use the following commands to inspect /tmp directory space

df -h

df -i

PHP memory_limit reached

memory_limit is a php directive that specifies how much memory a PHP script is allowed to allocate. Sometimes your application might be exceeding this limit, thus failed to produce response for the web server resulting in HTTP Error 503 Error. As explained above, first find out the PHP version used by your site. Then you can directly increase memory_limit from CyberPanel interface.

Login to your CyberPanel Dashboard then from left sidebar PHP -> Edit PHP Configs

  1. Select PHP version to change the value of memory_limit directive.
  2. Set the new value of the directive.

Finally, scroll down and click Save Changes.

max_execution_time reached

max_execution_time is similar to memory_limit. So if your PHP script exits early without producing response again you will get the same error. You can follow the same procedure as described above to fix max_execution_time as well. Make sure to set it to a high enough value so that your script is properly executed.

Reverse

We’ve pretty much discussed all the possible causes of “HTTP Error 503. The service is unavailable”. However, we recommend moving your sites to CyberPanel, because CyberPanel use LiteSpeed servers. This means in low-cost VPS you can host more sites, and using LSCache WordPress plugin you can avoid many such errors including HTTP Error 503. Because if your pages are cached, PHP engine is not used, thus giving other applications more resources to run. So in a low cost server you can run multiple sites at super fast speed and avoid such errors. You can learn in our OpenLiteSpeed vs NGINX comparision post as to why you would use CyberPanel and OpenLiteSpeed.

You can also get our managed vps and let us do this for you. We offer 3 days trial (no credit card required, plus free migration and fully managed support)