PEAT Online Course -
Test-Driven Development
using Kotlin (DELUXE EDITION)

  • Theory 'n' practice
  • Test-driven method
  • Acceptance criteria
  • Learn to code mastery
  • Ace that job interview
  • Free updates
AVAILABLE NOW
VIMEO ON-DEMAND

Staying Alive with Atlassian Stash

19 May 2014 3 comments

5 minutes

936

Don’t get me wrong, I like the simplicity of the user interface in Atlassian Stash, what is too painful for me is the networking configuration between it and Apache HTTPD server. It really should be very simple, but I find that mod_proxy drops more connections than a butterfingered wicket keeper on the field.

Apparently, this configuration in Mod Proxy should work e.g /etc/apache2/mods_available/proxy.conf

<IfModule mod_proxy.c>
ProxyPass        /stash https://localhost:7990/stash keepalive=On connectiontimeout=300 timeout=300
ProxyPassReverse /stash https://localhost:7990/stash keepalive=On
</IfModule>

This is based on the official Atlassian Stash documents, the difference is I have added keepalive=On. The HTTP connection should be kept alive for the long time.

And yes, I have already looked around the official Apache documents on mod_proxy and played around with the connectiontimeout, timeout and ttl parameters to no avail. I found the keepalive parameter and that helped alot.

I also found that on my virtual environment that is provided by my hosting company that Tomcat tooks a very long time to launch. I suspected also my hosting company VPS (virtual private server) was closing connections from time to time, but when I asked them, they answered that they do not do that with their customer’s policy. So the apache is losing the HTTP connection or the Tomcat server is dropping it.

Ok, I have seen the product warnings in the actual output of the shell script file:

... Detecting JVM PermGen support...
PermGen switch is supported. Setting to 256m

Using STASH_HOME:      /etc/acme/protect/innocent
Using CATALINA_BASE:   /opt/atlassian-stash-2.10.1
Using CATALINA_HOME:   /opt/atlassian-stash-2.10.1
Using CATALINA_TMPDIR: /opt/atlassian-stash-2.10.1/temp
Using JRE_HOME:        <strong>/opt/jdk1.7.0_51</strong>
...
Success! You can now use Stash at the following address:https://localhost:7990/
<strong>If you cannot access Stash at the above location within 3 minutes</strong>, or encounter any other issues starting or stopping Atlassian Stash, please see the troubleshooting guide at:
https://confluence.atlassian.com/display/STASHKB/Troubleshooting+Installation

When I checked the logs of the embedded Tomcat server with the Atlassian Stash product, I see very slow times:

<strong>INFO: Starting Servlet Engine: Apache Tomcat/7.0.47</strong>
<strong>May 19, 2014 9:45:53 AM org.apache.catalina.core.AprLifecycleListener init</strong>
INFO: The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: /opt/atlassian-stash-2.10.1/lib/native:/magic/etc/acme/protect/git/stash/repository/lib/native
May 19, 2014 9:45:53 AM org.apache.tomcat.util.digester.SetPropertiesRule begin
WARNING: [SetPropertiesRule]{Server/Service/Engine/Host} Setting property 'xmlValidation' to 'false' did not find a matching property.
May 19, 2014 9:45:53 AM org.apache.tomcat.util.digester.SetPropertiesRule begin
WARNING: [SetPropertiesRule]{Server/Service/Engine/Host} Setting property 'xmlNamespaceAware' to 'false' did not find a matching property.
May 19, 2014 9:45:55 AM org.apache.coyote.AbstractProtocol init
INFO: Initializing ProtocolHandler ["http-bio-7990"]
May 19, 2014 9:45:55 AM org.apache.catalina.startup.Catalina load
<strong>INFO: Initialization processed in 3734 ms</strong>
May 19, 2014 9:45:55 AM org.apache.catalina.core.StandardService startInternal
INFO: Starting service Catalina
May 19, 2014 9:45:55 AM org.apache.catalina.core.StandardEngine startInternal
INFO: Starting Servlet Engine: Apache Tomcat/7.0.47
May 19, 2014 9:49:01 AM org.apache.coyote.AbstractProtocol start
INFO: Starting ProtocolHandler ["http-bio-7990"]
May 19, 2014 9:49:01 AM org.apache.catalina.startup.Catalina start
<strong>INFO: Server startup in 186153 ms</strong>
May 19, 2014 9:49:21 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.8-atlassian-12 07/11/2013 01:08 AM'
May 19, 2014 9:49:21 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.8-atlassian-12 07/11/2013 01:08 AM'
May 19, 2014 9:49:21 AM com.sun.jersey.api.wadl.config.WadlGeneratorLoader loadWadlGenerator
INFO: Loading wadlGenerator com.sun.jersey.server.wadl.generators.WadlGeneratorApplicationDoc
May 19, 2014 9:49:21 AM com.sun.jersey.api.wadl.config.WadlGeneratorLoader loadWadlGenerator
INFO: Loading wadlGenerator com.sun.jersey.server.wadl.generators.WadlGeneratorGrammarsSupport
<strong>May 19, 2014 9:49:21 AM com.sun.jersey.api.wadl.config.WadlGeneratorLoader loadWadlGenerator</strong>
INFO: Loading wadlGenerator com.atlassian.plugins.rest.doclet.generators.resourcedoc.AtlassianWadlGeneratorResourceDocSupport

There is a clue here, inspect the timing between the WADL generation and the actually server up time. It is 4 minutes longer than necessary. Why is this WADL generation not pre-generated or cached on each server run into a database? There is should be 4 minutes start up time on a Tomcat server for an enterprise application in 2014!

So I have to two issues with Stash. First it runs like a dog (“a slow dog, I have know many fast real dogs, mongrels that can just tear up the track”) and second it has a long start up time. It is not emotional but a mystery. Stash is not quite reliable for me to recommend it to others or to clients.

Postscript

I am currently using the following settings in the proxy.conf

<IfModule mod_proxy.c>
ProxyPass        /stash https://localhost:7990/stash keepalive=On connectiontimeout=300 timeout=300
ProxyPassReverse /stash https://localhost:7990/stash 
</IfModule>

And I have two system cron jobs to reboot Apache and Stash every few hours, which works something like this:

10 8,11,13,15,19 * * * /etc/init.d/apache2 restart
08 8,11,13,15,19 * * * /etc/init.d/Stash.sh force-kill-restart

I will file a ticket with Atlassian support, because I should have to continuously reboot services like this.
First, I will upgrade to the very latest version of Stash 2.12.
I just tried to upgrade from 2.10.1 to 2.12.3. I backed out of it, because I need access to the special port and I cannot do it from here remotely.

3 Comments

  1. Hi Peter,

    I’m the dev lead for Stash, I’m sorry to read that you had trouble getting Stash up and running. The application definitely shouldn’t take 4 minutes to start up. It should take less than a minute to start up (assuming the required resources are available).

    I’ll look into your suggestion regarding the KeepAlive value you had to set to make sure our documentation is up to date.

    I also would like to ask you to raise a support ticket at https://support.atlassian.com/ so that we can help you to fix the slow startup problem, 4 minutes definitely hints at something going wrong here.

    Cheers,
    Stefan

    Comment by ssaasen — 20 May 2014 @ 1:35 am

  2. I had a quick look at our internal config and we have been running with Apache in front of Stash for a couple years now without specifying the keepalive setting for the proxy -> backend connections (we do enable keepalive though for the client -> proxy connections). So again, sounds like something is not quite right and our support team could help you troubleshoot this.

    Cheers,
    Stefan

    Comment by ssaasen — 20 May 2014 @ 1:47 am

  3. Actually, I had the keepalive setting only for the proxy -> backend connection as a first step. I will file a ticket with the support team. Thanks Stefan for getting back to me.

    Comment by Peter Pilgrim — 20 May 2014 @ 9:46 am

RSS feed for comments on this post.

Sorry, the comment form is closed at this time.

Hey all! Thanks for visiting. I provide fringe benefits to interested readers: checkout consultancy, training or mentorship Please make enquiries by email or
call +44 (0)7397 067 658.

Due to the Off-Payroll Working plan for the UK government, I am enforcing stricter measures on contracts. All potential public sector GOV.UK contracts engagements must be approved by QDOS and/or SJD Accounting. Please enquire for further information.