Posted by on Dec 17, 2010 in SQL Family, SQL Musing | 0 comments

On the 7th day of SQL…

Gaston - arrogant dude from Beauty and the Beast. Because I can

Following in Tim Ford’s (blog|twitter) footsteps, I also went non-technical with my favorite post for this awesome twelve days of SQL, which is one of Brent Ozar’s (blog|twitter) collection of bad great ideas that I’m very honored to be a part of.     I struggled to pick a favorite for this series since, well, there are many of them and it’s hard for me to just pick one,  but I hold this particular near and dear to my heart since this is a simple fact that we all know, but sometimes overlook (me included).

This post serves to remind me that, as a Production DBA, I need to be aware of everything that touches my servers and databases.   I’m not talking about fancy Change Management process (that could be a separate post on its own) but it’s the awareness that other groups sometimes don’t have, or hey, sometimes we as a DBAs forget too.

I’m talking to maintain a centralized log about every change that happens on the server.

 

How many times have we modified an index on the fly and forgot to document that change?   It’s not really a code change, is it?   Well, being a Production DBA on a very busy server, every change matter.    On that day I noticed an anomaly in our monitoring software, a spike where there shouldn’t be one.  Red letters that said something had changed and I needed to look to see what it was.  It was not impacting customers, but it could if I let it go.  That is why I set my monitoring thresholds so I know before they do that something needs attention.  Anyway, rather than go through a thorough diagnostics, log analysis, and alerting the team to start culling through code changes, I simply popped open list of the changes that had been made to the server and even though it wasn’t in red and flashing, the root cause popped right out and sure enough, if we didn’t take action, customers would know.  The fact I had a running log of even the subtle changes made to the server allowed me to quickly deploy a fix to a couple indexes and had the devs increase our cache time and voila, the site sailed through a record Thanksgiving traffic with 100% availability and response times that may make others jealous.

Don’t get me wrong, this is not just a need to keep track of the things we DBA’s and our developers do, we also need to be aware of other teams small tweaks on the server or network, such as, oh I don’t know, replacing the network card or swapping the processor, or something silly like that.  Even subtle difference like changing the network cable or plugging it into a different switch port can have huge impacts and I need to know!   For the network or sysadmin teams, that might be nothing, but that could have huge impact to our DB performance if we are not aware of it.  Ever try and troubleshoot a port speed mismatch without the support of the netadmin?  Or that pesky (but well intentioned) SAN admin who thinks you can do just fine with a RAID5 instead of a RAID10?  Not to mention tools are almost designed so that notification is an afterthought, that SAN admin can do the whole thing on-line without your knowledge and there is no down time, huh.  Did you ever think the marketing hype of “no down time” would be abused?  How many times have you heard an admin claim, “no one will know” only to hear help desk phones start ringing without warning?  Subtle changes can make a big difference in the big picture, and don’t even get me started on load testing in the test environment.  Anyway…

My favorite post is written by Jonathan Kehayias (blog|twitter).  He is a co-author of this awesome book, Professional SQL Server 2008 Internals and Troubleshooting (every SQL Professional should have this book – and no, Brent didn’t make me say this).  I have gotten to know him during SQL PASS 2010 when he dropped by to my office with Brent and whipped out some fancy extended events code to help me out identify some of the performance issue that I experienced.   He’s a well respected member of the SQL Community, an intelligent fella’ , MVP and I would love to just pick his brain anytime I can.

 

Without further ado, here’s the post:

http://sqlblog.com/blogs/jonathan_kehayias/archive/2010/08/03/there-is-no-such-thing-as-a-small-change-to-a-production-database.aspx

Coming next, a dear friend of mine who I met in person during another of Brent’s bad idea, SQL Cruise.   Karen Lopez (blog|twitter) is incredibly talented and has an ocean of knowledge about database design/architecture.   She’s very active in the community, a renowned speaker, an icon for Women in IT and an awesome person to hang out with (even at a character dining in Disney.)     Please check on her blog tomorrow to see what her favorite post of the year.

Enjoy

Share
Read More