backtop


Print 22 comment(s) - last by tech329.. on Jun 23 at 5:05 AM

Web host says sites are unrecoverable

It's every web administrator's worst nightmare -- your online presence is totally destroyed in a service outage.  That's precisely what happened when Australian domain registrar and web host Distribute.IT was attacked.

Over 4,800 websites were reportedly lost when the hackers struck last Saturday, as four servers were reportedly left unrecoverable.  The company comments:

The overall magnitude of the tragedy and the loss of our information and yours is simply incalculable; and we are distressed by the actions of the parties responsible for this reprehensible act.

At this time, We regret to inform that the data, sites and emails that were hosted on Drought, Hurricane, Blizzard and Cyclone can be considered by all the experts to be unrecoverable," it said.

While every effort will be made to continue to gain access to the lost information from those hosting servers, it seems unlikely that any usable data will can be salvaged from these platforms.

In assessing the situation, our greatest fears have been confirmed that not only was the production data erased during the attack, but also key backups, snapshots and other information that would allow us to reconstruct these servers from the remaining data.

The company promises to help customers "transfer your hosting and email needs to other hosting providers."

For large site owners that likely won't be a problem as they likely have save backup copies of their homepage.  For smaller operators, though, this could be very bad news, as many of them don't have the resources to save backup copies. 

Writes one customer in a local forum, "[The hack] has probably killed my business."

The question remains why Distribute.IT was penetrated so easily and thoroughly.  It is also baffling why they chose not to back up their data off-site as most hosting firms do.

As the potential for abuse of the stolen private information of website owners is great, these factors may play a key role in possible future legal proceedings by site owners against the company.


Comments     Threshold


This article is over a month old, voting and posting comments is disabled

Dude, where's my data?
By SiliconJon on 6/22/2011 9:24:26 AM , Rating: 3
What incompetent bafoon was in charge of their backups? They don't even need to go offsite to keep their backup data offline, unless they were hit with an EMP or incredibly patient logic bomb seed planted long ago, though offsite is of course still essential in case of a physical disaster. But that's not my my point...how on earth does one have such an immense single point of total failure/loss?

Wait, nevermind, I know how...I've worked in several offices and have seen it in each and everyone of them(in many a department) - we here in Amerika tend to run on cliques and resemble something closer to a Kakistocracy.




RE: Dude, where's my data?
By StevoLincolnite on 6/22/2011 9:44:12 AM , Rating: 2
From what I read a few days ago, the hacker wiped the backups. (Which are on-site)
However, an intrusion such as this would probably have come from some disgruntled ex-employee who knew the layout of the hardware and hence backup systems.
Or a hacker stumbled onto the back-up systems and had a field day thinking all his Christmas's had come at once...

Hopefully some of these websites (Which are small businesses) can use Google cache to retrieve a copy of their website.


RE: Dude, where's my data?
By Dribble on 6/22/2011 10:22:20 AM , Rating: 2
I always thought backups meant tapes in a fireproof safe, preferably off-site? Surely your only backup for data can't be another disk in another machine on the same site - then even a simple fire would wipe out everything.


RE: Dude, where's my data?
By amanojaku on 6/22/2011 10:41:00 AM , Rating: 2
quote:
I always thought backups meant tapes in a fireproof safe, preferably off-site?
That's how it used to work, when companies didn't have much to backup besides a database. That model is expensive today since everything (servers and desktops) gets backed up, restoration from tape takes too long and is prone to data corruption, and offsite facilities like Iron Mountain are out of the control of the IT staff.

The newest method of backup is the VTL, or virtual tape library. It's a NAS or SAN that pretends to be a tape device, with greater performance and improved restoration times. You can even do data de-duplication, defrag, etc... Except that the data isn't taken offline, so anyone with network access to the VTL can do all sorts of bad stuff.

http://en.wikipedia.org/wiki/Virtual_tape_library

There's also the possibility that the backup procedures were lacking. I've seem many environments that left tapes in the backup servers, which an authorized user could erase. And sometimes did, by accident.


RE: Dude, where's my data?
By greylica on 6/22/2011 11:30:26 AM , Rating: 2
What I do here: Generate My Backup, compress with 7-ZIP, and then record it into BD-R -> Library.
If you ask me for a snapshot of DB of January 2006, 5, I will great you with a DVD of the week and the backup is in good state.
All of them are stored with dissecant, in appropriate cases. And when they get 5 years old, I start to switch media from DVD to Blu Ray.
Never lost any backup, never put my job and career at risk. If there's offline media to use, (and very cheap for what they do) let's use it !
The infrastructure to achieve those levels of security is cheaper than Tape Library, even in cases where we have lot's of data.
We have for example a partition with 128GB of documents, even older docs are there. After a full (Ultra) 7-ZIP compression, (Octa Core, 6 Hours of compression), we have 40GB of data split to two U$3,00 BD-R. It's one of our main Backups, and it's done once a week, because there aren't lot's of changes on them over a week (traced). The production DB is backed up once a day, and after compressed, 100GB of data is converted to a simple U$ 0,50 DVD storage.
And I say, we ddin't store them only in the facility, some times, a copy of those backups go to the home of the enterprise owner, avoiding problems like fire, etc.
There's no excuse for online backups only, sorry...


RE: Dude, where's my data?
By Mitch101 on 6/22/2011 12:00:49 PM , Rating: 2
Its the same as home users. LAZY or CHEAP.

Nobody buys or does a backup until they lose information and no one should expect much in way of backup or support from those $2.99 a month hosting packages that a lot of users sign up for.


RE: Dude, where's my data?
By Samus on 6/23/2011 1:21:08 AM , Rating: 1
Wow. I'm an IT consultant and each and every one of my clients have two forms of backup: onsite (USB HDD, Iomega Rev, etc) + offsite (Kryptonite, DataGuard, Symantec...)

The onsite backup is rotated, in many cases daily but some at random, by an employee such as a secretary, instructed to rotate using drives/disks in a small, fireproof lockbox next to the server that has a basic electric keypad.

Point it, these are small, ~10 employee, businesses in greater Chicago area, and this article is telling me a fucking WEB HOST had an inferior backup plan? Dude, 2TB USB drives cost like $100 bucks, go get a couple and rotate the damn things daily on your way out the door for the evening. It takes 20 seconds!! While your at it spending a few bones on the hard drives, get a $50 electronic locking firebox at harbor freight, and if you really want to splurge, which you should, spend a few hundred bucks a year on a popular offsite backup like Kryptonite.

At worst, you'll lose maybe 3 days of recent data.


RE: Dude, where's my data?
By Etsp on 6/22/2011 11:38:21 AM , Rating: 2
Desktops can be reinstalled, usually with clones. Servers can be reconfigured. End-User data can NOT be rebuilt. Customer data can NOT be rebuilt. It's fine to be selective of what you put into an offline backup system, but it is in no way OK to NOT HAVE ONE, and in no way is it OK to NOT TEST IT REGULARLY.

As an aside, what kind of a moron designs a backup system to backup end-user desktops? That's a waste of resources if I've ever heard it. Have users save their data on a network share that gets backed up. Have a clone of their PC configuration ready in the event of a hardware failure. Now, suddenly, there is no need to include end-user PC's in a regular backup. (Upper Management/Executive PC's are a different story... CYA there.)


RE: Dude, where's my data?
By amanojaku on 6/22/2011 12:36:47 PM , Rating: 2
@greylica, no executive in a large company is going to do what you described. Yes, it is inexpensive. No, it is not simple for a c-level or VP with a head like a box of rocks.

@Etsp, I agree that you must have a backup solution, and that it must be tested. I have a NAS and I've pulled drives hot, added larger drives to resize the volume, pulled power cords, etc... just to see what happens.

As to desktop backup being unnecessary... That depends on the company. Yes, a NAS or SAN should be the default location for user data. This way the desktop/laptop/thin client becomes a generic processing node, and backup is typically (but not always) done at the array level. Centralized storage is the default model in VDI environments running Citrix XenDesktop or VMware View, as well.

However, offline and/or remote users cannot work with this model. A person on a plane does not (yet) have access to company resources, and low bandwidth to remote locations restricts file transfer and modification over the WAN for large files. Worse, these users tend to have sensitive data, so backup becomes a necessity. Small brokerage firms with offices across the globe fall into this category, with a total head count of 100 folks, 40-50 in NYC, 10-20 in London, 10-20 in Hong Kong or Singapore, and 1-5 person-offices everywhere else. Worse, these organizations don't have or want to spend the money for an enterprise NAS/SAN, which usually starts at $250K.


RE: Dude, where's my data?
By BZDTemp on 6/22/2011 6:03:59 PM , Rating: 2
quote:
There's also the possibility that the backup procedures were lacking.


Exactly - stupid mistakes are made. I've seen a 600 house holiday resort lose all their booking data because their Mini caught fire and their backup tape was lying on top of the machine (they fortunately had paper copies of all the reservations send out - it only took 6 man months to get them back in a computer).


RE: Dude, where's my data?
By deathwombat on 6/23/2011 4:10:04 AM , Rating: 2
That would give them the HTML output, but not the code (PHP, ASP, etc.) that generated it. If all I had was a copy of my website's HTML, and not the source code, I would have to manually edit each of the thousands of pages on the site in order to make any changes. It would be 10 to 100 times more work, and would probably kill the site.

Of course, that being said, I keep multiple backups of the site code, files, and databases. Knowing that something like this can happen makes me glad that I do.


So what good
By michal1980 on 6/22/2011 9:29:13 AM , Rating: 5
are these hackers doing?

What benefit to the world do these hacks bring?

IMHO, they're nothing but thugs destroying other peoples work.




RE: So what good
By chmilz on 6/22/2011 10:04:20 AM , Rating: 3
I can tell you what good it's supposed to do, but so far hasn't: Created an internet security revolution.

Obviously I don't condone any attacks that hurt innocent parties, but this whole cyber war going on right now has the entire online world caught with their pants down about how little they care about security and protecting private information. Expect identity theft and fraud to skyrocket while companies continue to be complacent.


Sorry, but it's ridiculous nowadays.
By greylica on 6/22/2011 10:05:23 AM , Rating: 4
As I read the entire article, the statement ''key backups are lost'' seems to me as very incompetent. A good administrator will never leave i'ts backup online, a good administrator will leave a ''copy'' of the backup online, only to save time, and another 2 or 3 offline, either inside optical media ( Blue Ray ), or even in external hdds, Tapes or anything else, and when those disaster occurs, will work on the copy of the backup to restablish the systems. Also, some key systems using SAS, SATA or SCSI Raid controllers could be marked as write protected, also some systems as linux and unix, and also some HDDS.
Sorry, but as an good IT admin, I recovered my enterprise twice, one from a strike that took our equipment, and another from a power fluctuation that kills nobreak, power supply and 2 HDDs that where in raid. It took me only 4 hours to restablish those systems. No online backups, no chance for errors of this magnitude. Sorry, this is unnaceptable.

Guys, learn here:
There are plenty of offline backup medias out there, there is no excuse for leaving backups online.
Same for giant enterprises, no matter how you're big.

Greylica.
Ivan Paulos Tomé.
T.I. admin.




By Ben on 6/22/2011 12:23:50 PM , Rating: 2
Agreed. This applies to the hosting company as well as all of it's customers.

It's unfortunate, but it takes a big data loss to make people proactive about data backup.


By BugblatterIII on 6/22/2011 2:13:43 PM , Rating: 2
The backups may not have been online. It could be that the web servers were part of the network, and once the hackers had compromised the web servers they gained access to the entire network.

Where I work we've made sure our web servers aren't on the network at all. It was one of the first decisions we made.

But even if the backups weren't online they should have been off-site.

For years companies have been getting away with cutting corners on security. Hackers are obviously accountable, but the companies that get hacked should be too.


Distributed
By xSauronx on 6/22/2011 10:03:21 AM , Rating: 4
Their backups were not.
/isn't it ironic?




Easily?
By nafhan on 6/22/2011 9:47:12 AM , Rating: 3
quote:
The question remains why Distribute.IT was penetrated so easily and thoroughly.
Sounds like their backup strategy may have had some serious shortcomings and "thoroughly" penetrated certainly seems to be the case, but do we know their systems were penetrated "easily"? It seems like a reasonable assumption, but I don't really see any info regarding the sophistication of the attack.




Not surprised
By Lanister on 6/22/2011 12:37:54 PM , Rating: 2
What do companies expect when they slash their IT budget to the absolute bare minimum they can get away with? Old tech, cheap employees with little to no experience, what could possibly go wrong?




It's their own fault
By repatch on 6/22/2011 1:15:39 PM , Rating: 2
"For smaller operators, though, this could be very bad news, as many of them don't have the resources to save backup copies."

Seriously? "smaller operators" can't run rsync? Seriously?

Look, I feel sorry for what happened from a downtime perspective, but data loss is a non issue. First off, the fact that the hosting company didn't have offsite backup is criminal. Second, what kind of company would have a site that their business relies on that they don't independently back up? Any of my sites go down I can be up with a new provider in a matter of hours if need be. Sure, the site might be a day or 2 old, but I'd still be up.




By undummy on 6/22/2011 5:32:52 PM , Rating: 2
All anyone has to do is take a job at ANY major company and see the incompetence in those IT or IS departments.

I hate the hackers and don't approve of their attacks. But, they do bring out issues that should have been addressed years ago.

Too bad so sad...to any company that still doesn't know how to take care of their data. All the IT degrees and certificates and 'common sense' still is all but forgotten.




Worst Nightmare
By tech329 on 6/23/2011 5:05:57 AM , Rating: 2
This is soemthing that every admin loses sleep over.

My solution is simple. I do backups to an offsite location and initialize those backups from that site. Not from the site being backed up. Thus there are no scritps etc or any other info that allows to contact the backup location. And as a final piece of my paranoid pie I have that backup site devised so all communication is one way with no connections initiated from the outside in allowed. It might be a pain if I need the data ever but it can't be touched. And lastly the backup site is a static IP. No DNS, no URL etc etc.




"Intel is investing heavily (think gazillions of dollars and bazillions of engineering man hours) in resources to create an Intel host controllers spec in order to speed time to market of the USB 3.0 technology." -- Intel blogger Nick Knupffer














botimage
Copyright 2014 DailyTech LLC. - RSS Feed | Advertise | About Us | Ethics | FAQ | Terms, Conditions & Privacy Information | Kristopher Kubicki