Nov 7 2009

Is The Database A Poor Step-Child?

Posted by Mike Brunt at 9:57 AM
8 comments
- Categories: Web Servers | .NET | DataBase | CloudComputing | ColdFusion | JRun-J2EE

Databases are so important to us, in many ways, they are the number one dependency we deal with in ColdFusion and yet, in many ways,  we treat them as ignored step-children and I fear the advent of Hibernate ORM in CF 9 will make matters even worse.  If client variables are in use, this becomes even more critical, tuning the database will always certainly yield more benefits than any amount of JVM tuning.  By tuning I mean looking at indexing, affinities, CPU allocation, hard-drive layout, etc.  All of theses things can yield immediate benefits or cause immediate pain if they are not right.

Typically, when we get new clients we try to persuade them to spend some time on our database review and report offering but most of the time they decline, wanting to focus all time/spend on the CF-web server, eventually we are typically able to persuade them to let us look at the database.  They would genuinely benefit greatly if we started out at the database.  Here are the main things we look at and report on...

  • Table Indexing
  • Back-up Redundancy Schema
  • Security Settings - Ports etc
  • Security Settings - User Accounts
  • Use of Views
  • Use of Stored Procedures
  • CPU Affinity
  • CPU Allocated to Database
  • Service Pack Level
  • Data File Location
  • Log Files Location
  • Compatibility Level (SQL Server Specific)
  • Heavy/bad processes such as Auto_Shrink
  • Data Type usage (looking for the most efficient data types)
  • Unnecessary Services
  • Long Running SQL

 

 

 

Comments

Bucky Schwarz

Bucky Schwarz wrote on 11/07/09 11:39 AM

I agree completely. Hell, even before you get to the tuning of the actual database, how much better would so many apps run if better SQL was written. It seems like instead of adopting the mindset that a good CF developer should write good CF as well as good SQL, we instead go in the opposite direction and say "don't worry about the SQL, ColdFusion 9 will handle that for you." I think the result is going to be the database becoming a sort of 'black box' where magic just kind of happens for you. What happens when the magic doesn't work? You end up digging around in the database and figuring how that works, THEN figuring out how to make the database work with Hibernate. What happens when you have to start writing complex queries with HQL? At that point, is it even worth it to use Hibernate?

But back to your original point, I think the database is absolutely the red-headed stepchild, and I'm not really sure why. The whole point of web apps is managing data and presenting it in a useful and meaningful way. It's kind of what web developers do: manage data and give it to the layman in a way that makes sense, so he doesn't have to go digging around in a database. A lot of people just hate dealing with databases, which should make you happy since I guess it gives you more business opportunities ;)
Mike Brunt

Mike Brunt wrote on 11/07/09 11:59 AM

@Bucky, thanks for taking the time to add just a great comment, you expressed it far better than I did. Sometimes I have people ask me why I spend so much time focused on the database as they thought I was JVM specialist, well I hope I am a performance specialist and of course as all data passes through ColdFusion, guess what takes the blame? I have two very good DBA colleagues one in the SQL Server world the other in Oracle and they do not like ORM centric work.
TJ Downes

TJ Downes wrote on 11/07/09 5:35 PM

I'd venture to say that the relational database is nearing the end of its life for many types of application development. OO development has made object storage more feasible and common, and applications will be better for it, in my opinion :)
Mike Brunt

Mike Brunt wrote on 11/08/09 10:13 AM

@TJ thanks for your comment and no doubt there are other mechanisms afoot, CouchDB for instance. From experience with clients, I still see mostly RDBM's and I think it will be some time, if ever they go away completely. What I do see happening relatively quickly is the adoption of caching mechanisms, such as memcached, to move data out of its slowest responding place; the hard drives.
Mike Kaplan

Mike Kaplan wrote on 11/09/09 6:14 AM

Don't forget about defragging your indexes. We have a weekly job that checks all indexes and defrags those that surpass a certain level of logical scan fragmentation. DBCC SHOWCONTIG and DBCC INDEXDEFRAG are powerful weapons in making sure your indexes keep doing what they're supposed to be doing, particularly in a system with a lot of data being added and edited.
Peter Bell

Peter Bell wrote on 11/09/09 6:16 AM

I think this is one of those cases where people try to generalize rules that are applicable to specific situations. The purpose of an ORM is to speed development of applications. The purpose of database tuning is to optimize performance of applications.

The vast majority of apps written never get heavy load - of course Mike, you probably don't get called in to do performance consulting on the many apps which don't need to perform! For some apps cost/effort/time to market is less important than the ability to immediately handle high loads (especially when replacing legacy systems - green field apps seldom need to handle huge loads on day 1 unless there's a lot of marketing or pent up demand to drive traffic).

For a lot of apps an ORM allows you to get to market quicker, then if you have a performance problem, load testing will identify bottlenecks (usually in the DB) which you can remove one at a time until the app performs acceptably for the current load - sometimes through DB only changes, sometimes through replacing specific hibernate queries with HQL or even with direct SQL through JDBC (although that raises a host of other issues), and sometimes through better caching strategies in the app level.

I think it's important to realize that the DB is often the source of performance issues, but also to understand that there are plenty of apps where cost and time to market are more important than performance. If there weren't, we'd write all of our apps in Java - not ColdFusion, Groovy or Ruby which are nowhere near as performant for many operations.
Mike Brunt

Mike Brunt wrote on 11/09/09 2:55 PM

@ Mike, thanks for the pointers and tips I am sure they will help many.

@ Peter, thank you for your comments and insights. You are right of course that most of the time we are called in for large applications where performance has become an issue. My view on this is that there are cost-effective ways to engineer and test applications before they are released, irrespective of size and it is important to do so. Mainly because small applications can grow big rather quickly if they become compelling enough. Load testing should be part of all application deployments in my opinion.
Gary Funk

Gary Funk wrote on 01/10/10 7:50 PM

HI Mike. Ray Camden and I just went over his databases for BlogCFC and CFBloggers. I just did a small entry on my blog dealing with MyISAM and InnoDB. A lot of programmers never give the database a second tought.

Write your comment



(it will not be displayed)



Leave this field empty: