A friend of mine sent me a link to an article that appeared in Redmond Developer News in January that should be required reading. I wish all my prospects read this. The sad part it’s a true story. So let me take you through the story a bit....

Dev Disaster: Death by Delete
One database, one delete statement, one big disaster for a small chain of pet stores.
by Alex Papadimoulis

Rick worked for a regional ISP that provided hosting services. The customer base consisted primarily of consumers and small businesses, so the ISP offered a lot of "a la carte" services for things like SSL, authentication and database access.

A small chain of pet stores in the tri-state area, MegaPetCo, approached Rick's company because it was trying to expand into other cities and states. Having outgrown its current provider, MegaPetCo wanted Web space, a database and a few e-mail boxes. After careful consideration, it opted for the $74.99/ month Advanced Package that included a 10MB database, 100MB of disk space and 100GB of bandwidth.

God does it not sound good. An advance package for $74.99 a month. They got  a 10MB database, 100MB of disk space and 100GB of bandwidth. This sounds too good to be true. You can run a small retail chains infastructure for $74.99. What is the very old saying  Caveat Emptor or in english  "Buyer Beware. Or to quote that american showman in the1800's P.T. Barnum "There is a sucker born every minute". As a sidenote even though P.T. Barnum was credited with saying this, people who knew him at the time felt he most likely never said that. That he was more likely to say "A Customer is born every minute".

As this story goes on by  Alex Papadimoulis...

After the switchover, it became apparent that what MegaPetCo needed far exceeded what it had ordered. For starters, its database ballooned to several gigabytes within the first month. On top of that, its Web traffic was far higher than the company's allotted bandwidth and other Web sites on the shared server were getting bogged down.

Given that the typical database grows 3 to 5X its size every 3 years, I am not surprised by the line  " its database ballooned to several gigabytes within the first month."

The other line that struck home " On top of that, its Web traffic was far higher than the company's allotted bandwidth and other Web sites on the shared server were getting bogged down."

Now for the drum roll...

MegaPetCo opted to pay for overages, and its first monthly bill was astronomical: $4,281.

Now there a shock. The first monthyl bill to run the  small chain of retail stores was significantly more than $74.99 a month. Now there a real shock. The story gets even better.

MegaPetCo had no problem paying outrageous hosting bills, but the company was upset that its Web site ran incredibly slow. Although Rick's company had long since dedicated a "shared" hosting server for MegaPetCo, it would still take upward of a couple seconds for a page to load.

Lets fast forward through the story...

How little he really knew. The shared database for this Web site -- a 10MB file which was meant to power nothing more than a couple of blogs and small-traffic forums -- was the database for the entire company. The Web site itself accounted for less than 5 percent of the total data in MegaPetCo's database. The rest was home-office stuff, including POP sales registers, payroll, HR, inventory, tax records, and a kind of dynamic storage for invoices and maintenance tickets.

The database was incredibly simple: a single table with hundreds of columns. It probably had humble beginnings as a spreadsheet and organically grew into a vast monolith over the seven or so years that MegaPetCo was in business. 

 

Fast forward a little more...

All told, the database had millions and millions of rows -- had being the operative word.

Fast Forward a little more...

One day, a developer was optimizing the database and removing records that MegaPetCo no longer needed.

 

All it took was a single, poorly formed delete query to wipe out each and every row in the database table.

To say the developer panicked is akin to describing a quadruple amputation as a mere flesh wound. MegaPetCo's sales immediately ground to a halt. Along with everything else. Payroll, logistics, reporting, purchasing, you name it. Its Web site didn't work -- but that was the least of the company's worries.

The status of its backups was bleaker still: None. Zip. Zilch. Nada.

Fast Forward even more...

Unfortunately, the single delete query proved to be far more than MegaPetCo could bear. Within a few months,

the company filed for bankruptcy and was forced to close every one of its stores -- laying off several hundred people along the way.

 

 

 

 

 

 

 

 

 

 

 

 

I do not do this story justice. I would encourage you to read the orginal story and fill in the many blanks I left. Here is a link to the original story...

Dev Disaster: Death by Delete by Alex Papadimoulis

So Many Lessons to Be Learned

I could go on for hours on this story. It has so many great lessons to be learned. 

Lesson number 1.  If it sounds to good to be true its most likely not true

Here is a business that employeed several 100 people. It had multiple locations. Where was the common sense. Remember the initial offer. 

After careful consideration, it opted for the $74.99/ month Advanced Package that included a 10MB database, 100MB of disk space and 100GB of bandwidth.

Do you realy think the actual price was going to be $74.99 a month. 

Lesson number 2.  The Hosting Provider

Hosting providers are in the business of Rack, Power & Pipe. If it’s Rack, Power & Pipe they excel at it. Its been my experience they provide at best mediocre Database Administration Services. 

The typical way they deal with performance problems with your business applications is to tell you need more hardware. Surprise, Surprise they make more money for that. The other way they deal with performance problem is to blame everyone else but themself. Its the application you are running. A good Database Administration As a service "TM" firm will get you answers to why you are having performance problems. The facts are the facts, and a good DBA, with the right tools can get you the facts to prove who is causing the performance problem and why. 

When I look at the backup strategy they are using. They approach the backup strategy in a one-size fits all model. They are not well versed in the many options available to you in how you can backup a database.  In fact when we audit database backup in general we find problems with about 20% of them. A system admin is not a DBA. A DBA is not a system admin.

Lesson 3. A Developer is Not a DBA.

In this story...

One day, a developer was optimizing the database and removing records that MegaPetCo no longer needed. All it took was a single, poorly formed delete query to wipe out each and every row in the database table.

To say the developer panicked is akin to describing a quadruple amputation as a mere flesh wound. MegaPetCo's sales immediately ground to a halt. Along with everything else. Payroll, logistics, reporting, purchasing, you name it. Its Web site didn't work -- but that was the least of the company's worries. The status of its backups was bleaker still: None. Zip. Zilch. Nada.

First of all. A good DBA firm would have made sure a proper DBA backup was in place. I am not surprised that the hosting provider was not backing up the Database properly. This business employed a few hundred people. They could afford a proper DBA service who would have made sure there was a backup in place before this happened. In fact when I look at the fact they were spending over 5000 a month. They should not have been messing around. By saving pennies, they are now out of business.

Or what a novel idea, we make a test database and run this query in test before we go live. 

Lesson 4. You Get What You Pay For 

Ntirety is in the business of Database Administration As a Service "TM".  There are over 50 companies that claim to do what we do. There is a big difference between hiring a a true service or a single DBA. There is a big difference between hiring a consulting company who uses DBAs who might be sitting on the bench and a true service. We have put a quote in front of someone and they have come back and said. We know this local guy who is a DBA that will do it for a lot less. Well will he be available on Christmas day if your database has a problem. What happens if he has an accident and is in the hospital a few days? What happens if you need to deply a technology suite he or she is not familiar with. Will they be proactive? What happens if they come across a problem that is over their head or they have never seen before?

My point there is a big difference between a service and firms or people who a sort if in this business but not really.

When your database is down your mission critical business applications don’t work. You are out of business. So think about that before you go with the low bidder.

I could go on and on. I think you get the point !

Posted Michael Corey,

Founder & CEO, Ntirety

www.ntirety.com

My Personal Twitter Account: Michael_Corey

Ntirety Corporate Twitter Account: Ntirety