charlie arehart wrote on 08/12/08 5:13 PM

Mike, rather than leave people to guess, can you detail what you mean about the problem of "crash protection capability in another product" causing a server to "suddenly go off line once they reached 10 threads in use"?

Since you're not talking about CF8's alerts (a feature they have akin to crash protection), and I don't think SeeFusion offers something akin to it, I suspect you're referring to FusionReactor.

But I can't think what CP feature could cause the server to go offline once 10 threads are used. It could send a notice, or it could abort threads, or it could queue future threads while the problem remained. I suppose if someone did the latter and didn't expect it, the queuing of requests could seem like "going offline". But hey, powerful tools come with responsibility. Let's not blame the tools for being misused.

Now, the "memory tracking" of CF8's monitor might be in a different class: it really can kill a server under even moderate load--but then judicious use of oft-missed filtering features could temper that.

But you conclude "If we are allowing something to shut-down our servers that is not a solution but a band-aid and I personally would not be trusting something to shut-down my production instances." Again, what are you referring to that's "shutting down" the server?

I wonder if instead you're referring to FusionReactor's Enterprise Scripting feature, which allows a monitoring server to control a monitored server if it detects it becoming unresponsive. [Some may say, "buy, Charlie, you sound defensive, jumping to the conclusion that it's FR". I'm just using my familiarity with the features of the 3 tools to sort through what wasn't said. :-) ]

In that case, yes, it can shut down a server that it deems has become unresponsive (more at http://www.fusion-reactor.com/fr/featurefocus/monitoredServerScripts.cfm). Again, though, it's just a tool. Many have long used home-grown or third party features like this (ping tools, which will restart the CF service if it fails to respond.) As with the CP abort/queue features, all such tools do need to be used carefully.

Let me be clear: I, too, tend to stay away from the features that either abort or queue requests, or restart the server, at least when first using a tool. I'd much rather use them the way you describe, to gather diagnostic info. And indeed, the main points of your perfect tool are what all three of the major tools do.

I just fear that it's possible someone reading your note could conclude that the "other unnamed" monitor tool is something to fear and avoid. That would be an unfortunate conclusion. All 3 of the major CF monitoring tools have their strengths and weaknesses, and all can be configured to have very low overhead and impact, just serving to provide info to help make better diagnosis and analysis.

But you go on to say that such tools "should do nothing more than just monitor the server". Well, that's certainly a philosophical difference among the tools. Even CF8's monitor can do more than just monitor, in its alerts features. People should be careful with them, yes, but it seems a bit much to assert that the tools shouldn't offer the feature at all.

Still I agree that it's good and fair to warn people (especially newcomers to a tool) to be careful of the more powerful features, and for that I'm glad to see you offer your thoughts here. Always thought-provoking! :-)

Mark Kruger wrote on 08/12/08 6:47 PM

Mike,

I agree with you pretty wholeheartedly that server monitors should monitor and not be tasked with killing thread or recovering resources automatically.

This sort of reminds me of the many uses of try/catch. It can be an appropriate tool for catching items outside of our control - or it can be a catchall for sloppy code :) In the same way I fear that sometimes the automation of alerts and actions that mitigate problems are really work arounds that mask deeper problems.

Of course if you are someone whose business is shared hosting (like the folks at Edge web for example) then sometimes the condition of the code is out of your control as well - so I guess I can see some circumstances where it would be a godsend. Still, over all I think you ar spot on.

-Mark

Sharon wrote on 08/15/08 5:23 PM

Do you have a Coldfusion User group meeting? When and where? If not who would like to have a monthly meeting?

charlie arehart wrote on 08/16/08 2:11 PM

Hey Mike, thanks for getting back to me. For some reason I wasn't getting all the comment emails for this thread (I got some, just not all).

As for whether and when to use FR's queuing, this is discussed at length in a back and forth exchange on the FR google group, in this thread:

http://groups.google.com/group/fusionreactor/browse_thread/thread/2f57342847400193/

The bottom line may be that CF started exposing its queuing features after FR (which has offered it for 6, 7, and 8). There is also a new feature in FR3 to monitor queued requests, something I don't see being offered in the CF8 monitor (which even if it does offer it, is only on CF8 Enterprise).

But as I said above, each tool has its strengths. I get why SF doesn't even offer the power to control requests. As I noted, even though FR and the CF 8 monitor both offer it, they are tools that must be used carefully--if at all. I tend to be in the later camp, too. Just didn't want anyone to think that just because FR had such power, its use otherwise should be discouraged or impuned. :-)

Hey, one last thing: any chance you may tone down the intensity of the Captcha you use? It's pretty tough to read. Many have gone to less obtrusive captchas (less lines, less circles), and not seen an increase in spam. Just a suggestion. If that's using Lyla as the captcha, I blogged about simple changes in its XML file that you could consider:

http://carehart.org/blog/client/index.cfm/2006/10/7/lyla_captcha_simplified_xml_file

Your call, of course. Just trying to help if you'd not considered this.

charlie arehart wrote on 08/18/08 2:42 PM

Hey Mike, a couple of thoughts. First, about monitors and their features to control requests, you say "providing ways to use information from the JVM to adjust server behavior could be somewhat dangerous". I realize it may seem we were coming to agreement on the above, but I just feel I need to respond to this. And while I'm at it, a couple other thoughts have come to mind. All this is hoping to help folks reading the thread, not trying to pick any fight, of course. I know you know that. :-)

I just find curious the assertion that FR is "getting information from the jvm". It's tracking requests as they come and go, like SeeFusion, just watching how long they take to run, how many are running, and available memory. It then also offers options to queue or terminate requests if set limits for those are exceeded. (I guess you could say the memory stat comes from the JVM, but not the rest. Just not sure what the key point may have been there, so I'm kind of guessing.)

Now, the crux of this entry and the comments have been about tools giving users too much control to be able to shoot themselves and create more damage than good, right? Well, I didn't press the point before, but SeeFusion supports killing requests manually too, just like FR and the CF8 monitor. And that can certainly be misused or abused. Just being clear about things.

Maybe your concern is with such control being automated, and I have agreed that that requires still more care.

But then I'd ask: do you have the same concern about load balancing/failover and other monitoring solutions, which ping the server and often also offer options to restart a non-responsive service (or might instead just declare it out of the cluster, even when it may still be up)? Just seems worth getting that out there, as again we don't want to fault any one tool as being too onerous in its efforts to provide for keeping things running smoothly. It's easy with any such tool for it to be more trouble than value if it's not used carefully.

As for the training classes, thanks for asking.If anyone's interested, they're offered at http://www.fusion-reactor.com/fr/traininginformation.cfm. (I wonder if anyone may read your question as really being a surreptitious way to get me to admit I have close alliances to FR. I don't think that's what you meant, but I don't hide it. I speak about FR as much as about the CF8 monitor, and in the past also spoke about SeeFusion. Again, I see them all bringing value to the table, so my comments here in support of FR are just from one member of the community to the others.)

Your blogs (this and the old ones at Alagad, Teratech, and webapper) are among the few that discuss a lot of these server issues, so people are thirsty for info on the topics. I just want to add to the conversation and help keep people fully informed. We're both here to help. :-)

Finally, about the captcha, the sad truth is that some of them will come from humans simply trying to increase their search engine ranking by dropping a URL into a comment (or in the website field here, even if not in the comment). We as bloggers just need to eradicate those pests when they pop up. But really, then, the heavy captcha is hurting far more people than its helping, as clearly the spammers work through it regardless. Only if you were being hit by automated spammers would a weaker captcha start to hurt. Just a thought.

Cheers.