39
Voxera
6y

Around 27 hours at new customer location.

They had a server failure due to incompetence.

They had fired their own IT guy and called us 6 months later because the server stopped responding.

First diagnostic. 2 drives are dead in a raid 5 with one hot spare. Raid controller then proved to be broken once the disks was replaced.

Waiting for new raid controller and installing.

Backup non existing, no one changed dat tape during the 6 months without IT. The tape was just a transparent plastic band, no media left.

Raid config is stored in static ram on controller, no backup!

Several hours in tech support to find out how to rebuild raid config from existing disks.

Proves to be impossible to rebuild raid set due to some checksum failures.

More hours with support to enable some diagnostic read only mode to mirror low level content to external drive.

Then many more hours to copy parts of the tree until it gets an error, restart after that and go on.

In the end we got around 70% back.

During this time I manage to be in contact with the raid manufacturers all support centers, one in europe, one in the us and one in Taiwan, switching each time one if them closed for the night.

The customer later declined a steady support contract due to us being to expensive ;)

Some just don’t want to learn.

Comments
  • 5
    RAID-5 with 2 drives dead.. RIP whatever was on it :') it surprises me that you were able to get anything out of it at all, as RAID-5 only allows for one drive to fail. Good thing that they've fired that incompetent moron. Hopefully they've learned their lesson now. Backups, backups and more backups!
  • 3
    That sounds awful. D:

    Well, at least the next time you had not to be the ones supporting them.
  • 2
    @Condor the hot spare got used before the second crashed.

    This was not an over night failure but had been brewing for months before boiling over :)
  • 2
    Raid is not a backup. Thus, the data on the disks was not considered important anyway.
  • 3
    @electrineer they had a backup, they just did not know they had to switch tapes.

    Dat tapes usually should not be used more than 50 rewrites as they degrade.

    The tape in the drive had gone through at least 150.
  • 0
    @Voxera Crazy story!
    It baffles me how they can be so irresponsible, not replace the IT guy and then even call you too expensive...
Add Comment