Reading time:

5 minutes

Word count:

936

Don’t get me wrong, I like the simplicity of the user interface in Atlassian Stash, what is too painful for me is the networking configuration between it and Apache HTTPD server. It really should be very simple, but I find that mod_proxy drops more connections than a butterfingered wicket keeper on the field.

Apparently, this configuration in Mod Proxy should work e.g /etc/apache2/mods_available/proxy.conf

<IfModule mod_proxy.c>
ProxyPass        /stash https://localhost:7990/stash keepalive=On connectiontimeout=300 timeout=300
ProxyPassReverse /stash https://localhost:7990/stash keepalive=On
</IfModule>

This is based on the official Atlassian Stash documents, the difference is I have added keepalive=On. The HTTP connection should be kept alive for the long time.

And yes, I have already looked around the official Apache documents on mod_proxy and played around with the connectiontimeout, timeout and ttl parameters to no avail. I found the keepalive parameter and that helped alot.

I also found that on my virtual environment that is provided by my hosting company that Tomcat tooks a very long time to launch. I suspected also my hosting company VPS (virtual private server) was closing connections from time to time, but when I asked them, they answered that they do not do that with their customer’s policy. So the apache is losing the HTTP connection or the Tomcat server is dropping it.

Ok, I have seen the product warnings in the actual output of the shell script file:

... Detecting JVM PermGen support...
PermGen switch is supported. Setting to 256m

Using STASH_HOME:      /etc/acme/protect/innocent
Using CATALINA_BASE:   /opt/atlassian-stash-2.10.1
Using CATALINA_HOME:   /opt/atlassian-stash-2.10.1
Using CATALINA_TMPDIR: /opt/atlassian-stash-2.10.1/temp
Using JRE_HOME:        <strong>/opt/jdk1.7.0_51</strong>
...
Success! You can now use Stash at the following address:https://localhost:7990/
<strong>If you cannot access Stash at the above location within 3 minutes</strong>, or encounter any other issues starting or stopping Atlassian Stash, please see the troubleshooting guide at:
https://confluence.atlassian.com/display/STASHKB/Troubleshooting+Installation

When I checked the logs of the embedded Tomcat server with the Atlassian Stash product, I see very slow times:

<strong>INFO: Starting Servlet Engine: Apache Tomcat/7.0.47</strong>
<strong>May 19, 2014 9:45:53 AM org.apache.catalina.core.AprLifecycleListener init</strong>
INFO: The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: /opt/atlassian-stash-2.10.1/lib/native:/magic/etc/acme/protect/git/stash/repository/lib/native
May 19, 2014 9:45:53 AM org.apache.tomcat.util.digester.SetPropertiesRule begin
WARNING: [SetPropertiesRule]{Server/Service/Engine/Host} Setting property 'xmlValidation' to 'false' did not find a matching property.
May 19, 2014 9:45:53 AM org.apache.tomcat.util.digester.SetPropertiesRule begin
WARNING: [SetPropertiesRule]{Server/Service/Engine/Host} Setting property 'xmlNamespaceAware' to 'false' did not find a matching property.
May 19, 2014 9:45:55 AM org.apache.coyote.AbstractProtocol init
INFO: Initializing ProtocolHandler ["http-bio-7990"]
May 19, 2014 9:45:55 AM org.apache.catalina.startup.Catalina load
<strong>INFO: Initialization processed in 3734 ms</strong>
May 19, 2014 9:45:55 AM org.apache.catalina.core.StandardService startInternal
INFO: Starting service Catalina
May 19, 2014 9:45:55 AM org.apache.catalina.core.StandardEngine startInternal
INFO: Starting Servlet Engine: Apache Tomcat/7.0.47
May 19, 2014 9:49:01 AM org.apache.coyote.AbstractProtocol start
INFO: Starting ProtocolHandler ["http-bio-7990"]
May 19, 2014 9:49:01 AM org.apache.catalina.startup.Catalina start
<strong>INFO: Server startup in 186153 ms</strong>
May 19, 2014 9:49:21 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.8-atlassian-12 07/11/2013 01:08 AM'
May 19, 2014 9:49:21 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.8-atlassian-12 07/11/2013 01:08 AM'
May 19, 2014 9:49:21 AM com.sun.jersey.api.wadl.config.WadlGeneratorLoader loadWadlGenerator
INFO: Loading wadlGenerator com.sun.jersey.server.wadl.generators.WadlGeneratorApplicationDoc
May 19, 2014 9:49:21 AM com.sun.jersey.api.wadl.config.WadlGeneratorLoader loadWadlGenerator
INFO: Loading wadlGenerator com.sun.jersey.server.wadl.generators.WadlGeneratorGrammarsSupport
<strong>May 19, 2014 9:49:21 AM com.sun.jersey.api.wadl.config.WadlGeneratorLoader loadWadlGenerator</strong>
INFO: Loading wadlGenerator com.atlassian.plugins.rest.doclet.generators.resourcedoc.AtlassianWadlGeneratorResourceDocSupport

There is a clue here, inspect the timing between the WADL generation and the actually server up time. It is 4 minutes longer than necessary. Why is this WADL generation not pre-generated or cached on each server run into a database? There is should be 4 minutes start up time on a Tomcat server for an enterprise application in 2014!

So I have to two issues with Stash. First it runs like a dog (“a slow dog, I have know many fast real dogs, mongrels that can just tear up the track”) and second it has a long start up time. It is not emotional but a mystery. Stash is not quite reliable for me to recommend it to others or to clients.

Postscript

I am currently using the following settings in the proxy.conf

<IfModule mod_proxy.c>
ProxyPass        /stash https://localhost:7990/stash keepalive=On connectiontimeout=300 timeout=300
ProxyPassReverse /stash https://localhost:7990/stash 
</IfModule>

And I have two system cron jobs to reboot Apache and Stash every few hours, which works something like this:

10 8,11,13,15,19 * * * /etc/init.d/apache2 restart
08 8,11,13,15,19 * * * /etc/init.d/Stash.sh force-kill-restart

I will file a ticket with Atlassian support, because I should have to continuously reboot services like this.
First, I will upgrade to the very latest version of Stash 2.12.
I just tried to upgrade from 2.10.1 to 2.12.3. I backed out of it, because I need access to the special port and I cannot do it from here remotely.