May 5 2009

10 Steps To A Stable and Performant Web Application - Step 2

Posted by Mike Brunt at 7:18 AM ColdFusion | JRun-J2EE

In the first of the 10 blog postings we will put out here, we looked at the first of 10 Steps To A Stable and Performant Web Application.  As mentioned, we always want to know what is happening in production and this initially involves looking at existing logs that are no more than 1 week old.  In this blog posting, we will look at adding Metrics Logging and also enhanced log splitting out using log.level.

Irrespective of whether we have or use ColdFusion Server Monitor, SeeFusion or Fusion Reactor; we always recommend the turning on of Metrics Logging and leaving it on in production.  The overhead is insignificant and the help we can get at times of server stress is priceless in some cases.  All the settings in this article can be used whether this is a Stand Alone install of ColdFusion or an Enterprise install, the configuration files are simply in different places. One very important point I want to make here, if you purchased the Enterprise license please ensure you install ColdFusion in the Multiple Instance mode, even if you will only have one instance at first, otherwise you are miising out on one of the most powerful features of the Enterprice license, the ability to create Multiple Instances.  We will edit the jrun.xml file to enable metrics logging and it is located, as follows, depending on the install of ColdFusion you have (assuming CF8 although, the only difference on MX6.1 and MX7 is the CF install directory name on the Standard install.

On a Standard install - {drive}Coldfusion8\runtime\servers\coldfusion\SERVER-INF

On an Enterprise Multiple Instance install - {drive}\JRun4\servers\coldfusion\SERVER-INF

In the jrun.xml file, locate this entry (it will somewhere around line 129)

<!-- This Service provides metrics information  -->

<!-- ================================================================== -->

<!--  To enable metrics: uncomment this service and in LoggerService set metricsEnabled to true -->

<!--<service class="jrunx.metrics.MetricsService" name="MetricsService">   

<attribute name="bindToJNDI">true</attribute>

</service>

-->

Uncomment this section if it is commented out, after that it will look like this...

<!-- This Service provides metrics information  -->

<!-- ================================================================== -->

<!--  To enable metrics: uncomment this service

      and in LoggerService set metricsEnabled to true -->

<!-- -->

<service class="jrunx.metrics.MetricsService" name="MetricsService">

  <attribute name="bindToJNDI">true</attribute>

</service>

Around line 159 locate this entry, if this is set as shown here "false", change this to "true".

<!-- You may also need to uncomment MetricsService if you want metrics enabled -->

<attribute name="metricsEnabled">false</attribute>

The next section has various items that can be set to different values.

<attribute name="metricsLogFrequency">60</attribute> - This entry sets the frequency of metrics logging to 60 second intervals which I have found to be a good number.

<attribute name="metricsFormat">Web threads (busy/total): {jrpp.busyTh}/{jrpp.totalTh} Sessions: {sessions} Total Memory={totalMemory} Free={freeMemory}</attribute> - This line contains a number of very important settings.  One important point before going any further is that in a non Enterprise Stand-Alone install we need to remove the "jrpp" from the metric variable declaration, so jrpp.busyTh needs to to be changed to busyTh and so on.  If this is an Enterprise multiple instance install, jrpp.busyTh is fine.  Here is an explanation of each element in this line.

Web threads - In the default version of this file there are two web threads that will be shown, (busy/total): {jrpp.busyTh}/{jrpp.totalTh}, however there are more, here is a rundown of each.

jrpp.listenTh - This is available threads that have not yet been used for anything, or were used but cyled back through different thread states to become idle again.

jrpp.idleTh - This is the state a thread returns to after being used or busy.

*jrpp.delayTh - Threads in this state are handling an incoming request but they cannot run because all available threads are busy.  This is not a good thing and usually causes degradations in performance of the server overall.

*jrpp.busyTh - This is all threads that are currently actively serving incoming requests.  In a well tuned server this number will stay low and continually cycle up and down.

*jrpp.totalTh - This is the total of all threads in every state.

jrpp.droppedRq - This shows the number of requests that were dropped because of some occurrence or other, perhaps a request time-out setting for instance.

jrpp.handledRq - This is the number of requests successfully handled.

jrpp.handledMS - This is the total amount of time spent successfully handing requests.

jrpp.delayMs - This is the total amount of time spent by threads in a delayed state.

jrpp.bytesIn - This is the total amount of data from requests coming in.

jrpp.bytesOut - This is the total amount of data sent out in response to incoming requests.

*freeMemory - This is the amount of free memory in the heap at the time of the metrics snapshot.

*totalMemory - This is the total amount of memory in the heap at the time of the metrics snapshot.

*sessions - Number of active jsessions at the time of the metrics snapshot.  This is Java sessions and not standard ColdFusion sessions.

sessionsInMem - This is the total number on jsessions in memory, including those that are not in use any more but have not yet timed out.

The ones we tend to use most often are *.

Please bear in mind that all of these metrics relate to what happened in the last 60 seconds, as defined by this setting <attribute name="metricsLogFrequency">60</attribute>.  Also, if you have a Standard non-Enterprise install, leave off the "jrpp." suffix.

At this point the metrics would end up in the {drive}Coldfusion8\runtime\coldfusion-out.log on a Standard install or in the JRun4\logs\{instance}-out.log on an Enterprise multiple instance install.  This is very useful as if there is some sort of problem the metrics will be in the -out.log at the exact time of any-all problems logged there, however it is also good to split out that log into discreet parts, in our experience.  For instance, creating a separate log for metrics only can be very useful for trending.  to identify possible memory leaks, for instance.  In order to do this we make one last change to the jrun.xml file as follows...

Around line 166 is this entry <attribute name="filename">{jrun.rootdir}/logs/{jrun.server.name}-event.log</attribute>.

We add a log.level command as follows...<attribute name="filename">{jrun.rootdir}/logs/{jrun.server.name}-{log.level}-event.log</attribute>

The net result of all of this are log entries which look like this...

29/04 23:29:04 metrics Web threads (busy/total/wait): 1/24/0 Sessions: 0 Total Memory=1012800 Free=943151
29/04 23:30:04 metrics Web threads (busy/total/wait): 2/24/0 Sessions: 0 Total Memory=1012800 Free=928446
29/04 23:31:04 metrics Web threads (busy/total/wait): 3/25/0 Sessions: 0 Total Memory=1012800 Free=919736
29/04 23:32:04 metrics Web threads (busy/total/wait): 1/26/0 Sessions: 0 Total Memory=1015104 Free=963036

This is just after a ColdFusion restart and all looks OK at this point.


01/05 12:17:09 metrics Web threads (busy/total/wait): 1/28/0 Sessions: 1 Total Memory=690048 Free=158060
01/05 12:18:09 metrics Web threads (busy/total/wait): 1/28/0 Sessions: 1 Total Memory=690432 Free=215408
01/05 12:19:09 metrics Web threads (busy/total/wait): 1/28/0 Sessions: 1 Total Memory=689984 Free=164688
01/05 12:20:09 metrics Web threads (busy/total/wait): 1/27/0 Sessions: 1 Total Memory=688000 Free=93655
01/05 12:21:09 metrics Web threads (busy/total/wait): 1/27/0 Sessions: 1 Total Memory=688832 Free=58571
01/05 12:22:09 metrics Web threads (busy/total/wait): 1/27/0 Sessions: 1 Total Memory=689792 Free=83309
01/05 12:23:09 metrics Web threads (busy/total/wait): 1/27/0 Sessions: 1 Total Memory=690496 Free=105955

This is around 12-13 hours later during busier times and we can see a good amount of memory is in use and total memory has decreased.

So Metrics logging is a very useful tool overall for getting started in the right direction with troubleshooting, in our next post we will look at Verbose Garbage Collection logging.

 

 

 

 

Comments

Mike Brunt

Mike Brunt wrote on 05/05/09 11:04 AM

I am once again apologizing because some of these detailed file examples do not display well in the browser. I just hope you can get the information you need from this piece.
Damon Gentry

Damon Gentry wrote on 06/19/09 9:20 AM

Mike,

I recently enabled metrics logging for a CF8 standalone server. I have specified a metricFormat that includes 'handledRq' and 'handledMs'.

I'm noticing that the values logged to the file are cumulative and are not reset at the metricsLogFrequency setting of 60 seconds.

Have you noticed this before on a standalone server? I'm not seeing this behavior on J2Ee deployments.
superdry sale

superdry sale wrote on 11/18/11 6:07 PM

Every superdry sale fans convinced superdry will be next huge designer. The colors of superdry uk are just amazing because they are universally flattering! It is a great alternative for people who are tired of buying out the entire superdry outlet line and are finished shopping at Our Superdry uk sale store.
expert  technique seo

expert technique seo wrote on 11/01/12 9:41 PM

Merci pour ce blog vraiment enrichissant. Cela me paraît fort intéressant. Continuez !

expert seo et backlinking http://superreferencementgratuit.wordpress.com agence marketing
delhiseoconsultancy

delhiseoconsultancy wrote on 06/17/13 3:11 AM

Your website is very chatty. It will be useful for all of us. You have completed a work. I will come here again to inspect new updates. Thanks for posting.
replica rolex

replica rolex wrote on 08/20/13 6:56 PM

I have to admit this is a very good article and worth reading !As mentioned, we always want to know what is happening in production and this initially involves looking at existing logs that are no more than 1 week old. In this blog posting, we will look at adding Metrics Logging and also enhanced log splitting out using log.level.
Replica Watches

Replica Watches wrote on 08/26/13 7:43 PM

I was longing to read such kind of informative write ups from a very long time.
get soundcloud downloads

get soundcloud downloads wrote on 10/23/13 1:40 AM

I am glad that it turned out so well and I hope it will continue in the future because it is so worthwhile and meaningful to the community to share something which we could use in the future. Thanks a lot.
personalfinancemagazine

personalfinancemagazine wrote on 03/07/14 5:18 AM

Simply both of these little modifications on it’s own inside your amusement costs can provide a person an additional $350 monthly for the individual financial cost management.
atlanta business insurance

atlanta business insurance wrote on 03/07/14 5:18 AM

Every time insurance industry profits decline sharply, the industry declares an "insurance crisis" - rates go up sharply, deductibles rise and underwriting guidelines tighten.
united states laws

united states laws wrote on 03/07/14 5:18 AM

The Immigration Reform and Control Act of 1986 deals with illegal immigration. This Act imposed tough criminal sanctions on employers hiring illegal aliens. The Illegal Immigration Reform and Immigrant Responsibility
replica Franck Muller watches

replica Franck Muller watches wrote on 03/15/14 4:38 AM

However, today's post is about a form of racing that is very young. Jewelry lovers whose space is limited will appreciate the clever design of the ultra jewelry organizer.
www.bloghostingproviders.com

www.bloghostingproviders.com wrote on 03/27/14 6:09 AM

A hosting provider is a company or a person that "hosts" your blog on his or her own server space. When you wish to set up your own blog, choosing the hosting provider is probably one of the most confusing tasks.
Search Engine Optimization For Law Firms

Search Engine Optimization For Law Firms wrote on 04/23/14 8:33 AM

I recently allowing metrics logging for a CF8 standalone server, seriously I'm not seeing this behavior on J2Ee deployments.
griffin law firm

griffin law firm wrote on 05/08/14 4:53 AM

Your composed work style has been surprised me. Thankful to you, greatly eminent post.
home inspection services

home inspection services wrote on 05/08/14 4:54 AM

What's up Dear, are you positively heading off to this site page on a standard establishment, if so thusly you will completely get sublime knowledge....
homes by pinnacle

homes by pinnacle wrote on 05/08/14 4:54 AM

This could be positively astounding and trust it'll work for me. Accepting that it takes a shot at purpose of me then i'll be over appreciative to you. Thankful before long.
home sense realty

home sense realty wrote on 05/08/14 4:55 AM

This article is obviously delightful, every one be fit for effectively be mindful to it, Thanks a significant measure.
humble homes

humble homes wrote on 05/08/14 4:55 AM

Your style is sincerely remarkable diverged from distinctive individuals I have examined stuff from. Thankful concerning posting it
essay writing company

essay writing company wrote on 06/05/14 12:52 PM

Website building is a unique way to earn money online rather than going to work in factory and earn, if you are brilliant coder then you can make a responsive website that can make earn through online traffic.
freedebtconsolidationquotes.com

freedebtconsolidationquotes.com wrote on 07/09/14 1:31 AM

Irrespective of whether we bear or manipulate ColdFusion Server Watch, SeeFusion or Alliance Reactor; we always suggest the turning on of Metrics Logging besides leaving it on in output.
questo articolo

questo articolo wrote on 08/06/14 8:47 AM

This is very useful as if there is some sort of problem the metrics will be in the -out.log at the exact time of any-all problems logged there, however it is also good to split out that log into discreet parts, in our experience. For instance, creating a separate log for metrics only can be very useful for trending. to identify possible memory leaks, for instance. In order to do this we make one last change to the jrun.xml file as follows..
Dr. Thomas Nabors

Dr. Thomas Nabors wrote on 08/21/14 1:39 AM

The web application is good. However, it will always need proper documentation of the process.
Ask Ask What

Ask Ask What wrote on 09/15/14 6:45 AM

Nice article and I will definitely bookmark it. I really like your writing style so I will visit this blog again in the future!
Amazon SmallVille

Amazon SmallVille wrote on 10/20/14 3:44 PM

That’s a discussion that speaks as a central focusing on the topic. Keep it up! thank you
HVAC learning

HVAC learning wrote on 11/23/14 9:32 AM

This does not surprise me. We need to bring back crucifixion for HVAC online courses . Come on wimps, let's give Mike Brunt his props. Too bad HVAC learning do not have the facility to think ahead and to think clearly but instead are ruled by emotion alone. Thinking like this is possibly our only hope.
findinvisaligner

findinvisaligner wrote on 12/24/14 5:26 PM

Excellent information, I think illustrations can do wonders, especially for those who are following it.
click

click wrote on 12/26/14 10:30 PM

I am happy to find this post very useful for me, as it contains lot of information. dgeggbfbfgb
Multi-Lab Macor

Multi-Lab Macor wrote on 02/10/15 12:48 AM

Multi-lab offer BNP 2 Machinable Aluminium Nitride in blocks, rods, sheets and finished components for industries including aerospace.
Cash in your gadgets

Cash in your gadgets wrote on 02/10/15 12:48 AM

Find out how much you could earn for your old laptop with Cash in your Gadgets.
SMT assembly

SMT assembly wrote on 02/10/15 12:49 AM

Trizo is a UK based contract electronics manufacturer. Providing PCB, Electronic & SMT Assembly & Design. We also offer Surface Mount PCB Assembly
Beverly Diamonds Ripoff Report

Beverly Diamonds Ripoff Report wrote on 04/20/15 1:23 PM

You have shared beneficial information and I think it will be very helpful to me for a stable and performant web application. I have read and understood thoroughly about the procedure to get such web application. Thank you so much!
clothing manufacturer

clothing manufacturer wrote on 07/04/15 4:22 AM

Every time insurance industry profits decline sharply, the industry declares an "insurance crisis" - rates go up sharply, deductibles rise and underwriting guidelines tighten.
http://www.cavoeno.fr

http://www.cavoeno.fr wrote on 08/24/15 3:49 AM

est watch to be inspired by military history reinterprets pocket watches bomber pilots secured to their legs durin
batright.com

batright.com wrote on 09/24/15 4:45 PM

Some great tips, despite the fact that I'm really not into web app creation, but website creation.
Postpartum underwear

Postpartum underwear wrote on 02/27/16 12:38 AM

carercn Exercise may also give you some Postpartum underwear relief from backache.
It can help with sleep problems, too, as long as you don't exercise too close to bedtime.
Carnrn,keep step go with incontinence underwear carercn
include pregnancy and childbirth,Postpartum underwear health conditions such as diabetes and obesity,
and changes that accompany aging.
iphone 7 cases

iphone 7 cases wrote on 03/18/16 5:33 AM

great article <a href="http://www.bestiphone7cases.com">iphone 7 cases</a> thank you