May 11 2009

10 Steps To A Stable and Performant Web Application - Step 3

Posted by Mike Brunt at 9:31 AM
11 comments
- Categories: ColdFusion | JRun-J2EE

This piece will detail the third step we take when working with clients to diagnose and remedy performance issues. Before continuing, here are links to the previous two steps, for reference purposes.

Step 1 - Inspect Production ColdFusion Logs (no older than 7 days).

Step 2 - Enable Enhanced ColdFusion Logging (Metrics Logging).

In step 3 we will enable verbose garbage collection logging, these details apply to the Sun Java Virtual Machine (JVM). We will already have some indication of how the total and free memory is behaving from the data produced by the metrics logging we enabled in step 2. 

Here, once again, is an example of metrics logging output...

01/05 12:17:09 metrics Web threads (busy/total/wait): 1/28/0 Sessions: 1 Total Memory=690048 Free=158060

01/05 12:18:09 metrics Web threads (busy/total/wait): 1/28/0 Sessions: 1 Total Memory=690432 Free=215408

01/05 12:19:09 metrics Web threads (busy/total/wait): 1/28/0 Sessions: 1 Total Memory=689984 Free=164688

01/05 12:20:09 metrics Web threads (busy/total/wait): 1/27/0 Sessions: 1 Total Memory=688000 Free=93655

01/05 12:21:09 metrics Web threads (busy/total/wait): 1/27/0 Sessions: 1 Total Memory=688832 Free=58571

01/05 12:22:09 metrics Web threads (busy/total/wait): 1/27/0 Sessions: 1 Total Memory=689792 Free=83309

01/05 12:23:09 metrics Web threads (busy/total/wait): 1/27/0 Sessions: 1 Total Memory=690496 Free=105955

The Memory= metric shows the total memory available to the Java Virtual Machine (JVM) heap and the Free= metric shows how much memory is still available to the heap, unused.  In this case, we can already see that free memory gets very low from time to time, what we want to look at next, is the detail of what the generations of memory in the JVM heap are doing.

One note, it is good to consider that Java is not simply a web development language and that the JVM and the way it behaves is designed to support all kinds of Java applications.  This is why it is always good to "tune" the JVM because the base default settings are typically not suitable for ColdFusion applications.

There are three overall parts, called "generations" which subdivided the total memory in the Sun JVM.  They are as follows...

The New or Young Generation - This is where all objects created in an application first go, to be very basic when someone clicks on something an object is created in memory and goes into the New-Young generation.  At this point there will be "references" to that object from the application and ongoing garbage collections check all objects for references.  After around 40 to 45 attempts to collect an object, in the New-Young generation it is moved to the next generation, if it cannot be collected, that generation is called the Old or Tenured generation..  One point about the New-Young generation is that in almost every case we observe this generation at 100% in ColdFusion applications.  In many of those cases we cannot set that generation high enough to avoid 100% usage, this comment applies to 32-bit installs.

The Old or Tenured Generation - As mentioned in the section above, objects that are not collected whilst in the New-Young generation, after 40-45 attempts, are moved into the Old-Tenured generation. This generation is effectively only collected when there is a Full Garbage Collection (Full GC).  Full GC's, in the current family of Sun JVM's are "stop-the-world" events in the sense that for the duration of the Full GC the JVM stops doing all else.  Therefore, too many Full GC's of too long a duration will impact performance, negatively and that is one of the things we always look for.

The Permanent Generation/PermGen - This is where the classes required by JRun-ColdFusion are stored along with some of the cached objects in the ColdFusion application layer. The thing that we want to avoid here is this generation permanently sitting at 100% with no room to grow.  In a lot of cases, we have seen out of memory errors caused by the permanent generation running at 100%.

The way that we enable verbose garbage collection, is by passing 4 arguments to the JVM via the jvm.config file.  This is an example of those arguments...

-XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintHeapAtGC -verbose:gc -Xloggc:cfwhispererGC.log

This set of arguments will produce logging output which will go into a file called "cfwhispererGC.log" in the {drive}\JRun4\bin directory on an Enterprise multiple instance install and into the {drive}\Coldfusion8\runtime\bin directory on a standard install of Coldfusion.  The actual logging will look like this...

{Heap before GC invocations=1281 (full 2):

PSYoungGen total 55360K, used 55360K [0x2bef0000, 0x2f7d0000, 0x2f7d0000)

eden space 52608K, 100% used [0x2bef0000,0x2bef0000,0x2f250000)

from space 2752K, 67% used [0x2f250000,0x2f422630,0x2f500000)

to space 2816K, 0% used [0x2f510000,0x2f510000,0x2f7d0000)

PSOldGen total 466048K, used 464447K [0x0f7d0000, 0x2bef0000, 0x2bef0000)

object space 466048K, 99% used [0x0f7d0000,0x2bd5fde8,0x2bef0000)

PSPermGen total 46848K, used 46842K [0x037d0000, 0x06590000, 0x0f7d0000)

object space 46848K, 99% used [0x037d0000,0x0658e878,0x06590000)

This tells us that there have been a total of 1281 garbage collections of which 2 were Full GC's. {Heap before GC invocations=1281 (full 2):

It also shows us that the New-Young generation is at 100% used - PSYoungGen total 55360K, used 55360K [0x2bef0000, 0x2f7d0000, 0x2f7d0000)

In addition, that the Permanent generation is at 99% used - PSPermGen total 46848K, used 46842K [0x037d0000, 0x06590000, 0x0f7d0000) object space 46848K, 99% used [0x037d0000,0x0658e878,0x06590000)

We use these findings to change the arguments passed via the jvm.config file to the JVM to adjust memory settings and segments.  One very important in sight here, there are dramatic differences between 32-bit and 64-bit installations.  On 32-bit installs we can never use more than 1.4GB for the JVM heap, total.  On 64-bit installs that number becomes 32GB so as mentioned earlier that is a totally different place to be.  If you are getting new hardware you should make sure it will be 64-bit.

Comments

Aaron Longnion

Aaron Longnion wrote on 05/13/09 7:59 AM

@Mike - great to get so much detailed information from you on this series. I just read step 4, too. I appreciate it!

One note with 64-bit, that's probably so obvious it's not worth mentioning, is to make sure you have enough physical memory to support the amount of memory you allocate to the JVM, even though 64-bit supports up to 32G for the heap.

The reason I bring this up is because I just made that mistake on Monday of this week. We updated to CF8 (64bit) on 64bit RedHat (thanks for the recommendation at Adobe MAX Europe 2008!), and each physical server only had 4G physical memory. Each the JVM args were set to use up to 1G for each CF instance, and (do the math) we had 5 instances of CF running on each physical server. Not good! You don't have a problem *at first*. But after about 5 hours of high traffic, the query cache filled up to 8000 (we've since reduced that setting), and the template cache got high... we'll we actually fully crashed the RedHat 5.3 (64bit) servers (never done that before). We had to have our IT guys go into the rack room and physically reboot the machine.

Anyway, lesson learned and fixed with only one outage after the huge infrastructure upgrade. But wanted to share, and remind folks to check your math. I wish I had. ;-)
Mike Brunt

Mike Brunt wrote on 05/14/09 8:00 AM

@Aaron you are absolutely spot on and this is not such an obvious point and if you are with a hosting company this can get hairy and expensive but we should always think about a good deal RAM with 64-bit systems otherwise we are losing a big advantage. What we have found is 32GB RAM for CF-Web Servers and 64GB RAM for Database servers are good numbers.
buy pinterest pins

buy pinterest pins wrote on 06/20/13 9:17 AM

This generation is effectively only collected when there is a Full Garbage Collection.
Real Estate Oakville

Real Estate Oakville wrote on 06/25/13 12:31 AM

Our team of professionals provides our clients with a very convenient one-stop opportunity for their Mississauga real estate needs. Our team includes: a professional photographer, a professional writer, stager, assistants, competent lawyers, experienced mortgage lenders and brokers, invaluable home inspectors, trustworthy renovators and handyman and the list goes on. Call us today to discuss your real estate goals.
buy facebook fans

buy facebook fans wrote on 07/17/13 5:53 AM

many form of web application now days coming in the market. and like short stack getting very popular application development.
rolex replica

rolex replica wrote on 08/20/13 7:02 PM

Anyway, lesson learned and fixed with only one outage after the huge infrastructure upgrade. But wanted to share, and remind folks to check your math. I wish I had.
dackkif

dackkif wrote on 10/22/13 9:58 AM

vous pouvez également regarder dans les évaluations en lien avec des caractéristiques spécifiques qui à leur tour améliorent la simplicité avec efficacité dans les chaussures. Même si crampons Nike sont vraiment un phénomène relativement nouveau mercurial vapor pas cher produit ou service, ils sont capables de lutter naturellement avec les marques les plus matures particuliers. La technologie que la http://www.newmercurial9.com marchandise utilise est tout simplement Chaussure de foot Nike Hypervenom superbe , il peut même facilement atteindre ou dépasser l'efficacité dans les autres marques reconnues . Il peut être recommandé l'utilisation des chaussures de trainings.Indoor est presque exactement le même que le faible minimiser shoes.However de tennis informel , il ya des questions standards spécifiques qui sont nécessaires simplement par chacun d'eux quand ils http://www.chaussurefootballpaschere.com se dirigent hors et loin du terrain. Une des nombreuses exigences de ce sport est un couple de nouveaux petits souliers de la ligue. Ces éléments sont essentiels pour être utilisé parmi les joueurs en raison de nombreuses raisons. Cela donne lieu à la signification crampons de foot sur crampons de football à faible coût. re essentiel de ces personnes » , si vous voulez être décrit comme une portion de la partie. crampon nike hypervenom Ou peut-être si presque rien d'autre, suivre les divers autres participants.
chaussure foot mercurial

chaussure foot mercurial wrote on 10/28/13 2:23 AM

Ce blog contient des articles merveilleux.
jeuxde918

jeuxde918 wrote on 11/17/13 11:34 PM

For a greater understanding of the quality replica brand watches that we sell, please browse through the testimonials on the site.
xuehua

xuehua wrote on 05/20/14 11:28 AM

They may be set up in boot styles to provide better hold http://www.chaussurefootballpaschere.com
throughout materials the place that the players enjoy. It's important http://www.nouveaufootball.com with regard to basketball shoes to have this specific component to be able to avoid these people through falling or perhaps getting back in mishaps.

Write your comment



(it will not be displayed)



Leave this field empty: