Linux/Recovering an ext3 hard disk

Contents 1 Recovering a damaged ext3 hard disk 1.1 Intro 1.2 DD_RESCUE the important files first 1.3 1- Check the disk with a disk checking utility

Recovering a damaged ext3 hard disk

This article shows you how to get your data back, see if you have hardware problems with your disks, repair in place, or extract the data to a new location.

I use Debian Lenny, with very slight modification you can make this work with Fedora, Redhat, SUSE and any other, Ubuntu, being a modified debian will probably work with same exact instructions if you append SUDO or run sudo su in terminal.

For those of you who know what to do but need some hints, it is all in the points, For those of you who do not yet understand how it works, you can read on.

1. DD_RESCUE the important files first
2. Disk check utility - fsck - EX: fsck/dev/sdb1
3. Mounting and Unmounting disks - mount -t ext3 /dev/sdb1 /adirectory or by editing the fstab file
4. Tune the file system, Removing the Journal (Effectively converting the EXT3 to an EXT2) - EX: tune2fs -O ^has_journal /dev/sdb1
5. debugfs <- use it in Read only mode, this thing is like brain surgery, 1 mistake and your FS is playing chess in heaven with all the other dead disks
6. Getting your data out before it goes down in flames EX: dd_rescue /dev/sdb1 /directory/mydisk.img
7. An even better way to get your data out, starting with the good parts, EX: dd_rhelp /dev/sdb1 /directory/mydisk.img
8. Checking the hardware

In this short tutorial, i will show you how to repair ext3 and ext2 disks, if repair does not work say due to a hardware error, there are always.

ways to get the good data out before the disk goes completely busted.

Intro

Not long ago, My NAS storage computer (Pentium D) got a bad disk, partially because the front fan that does spot cooling for the disks stopped and no one noticed, The disks were so hot they almost burned my hand.

Surely, i can't expect the disk to function after getting this hot in a vibrating environment with 8 other disks occupying the Computer case, all the vibration and the heat combined with the prolonged operation must have damaged the disk.

So, time to refresh my memory concerning disk repair, Just as if MAXTORs were still around. (I know nothing about maxtor since seagate took over, they may have improved)

DD_RESCUE the important files first

Before starting with anything, Sometimes there is one important file that you MUST have before you work on a failing or corrupted hard drive, For example, a XEN file with your clients data that you have not backed up in such a long time.

So if you can mount the failing hard drive (If it would mount), but you have I/O errors (IO / I0) like "/dev/sda2: Attempt to read block from filesystem resulted in short read while reading block 533" or link is slow to respond please be patient, Start by trying to get that file out of the fire first.

To do that, First install DD_RESCUE by issuing the command

apt-get install ddrescue

Then, go for that file directly

dd_rescue /path/on/failing/disk/file.img /path/on/good/disk/file.img

This surely requires that your good disk has the space to accommodate the file, if not, see how you can compress the DD_RESCUE output

1- Check the disk with a disk checking utility

fsck /dev/sdb1

or for an ext2 or ext3 hard disk

e2fsck /dev/sdb1

Now, if the above is asking too many questions and you need to answer yes to them all, you can use the -y switch, use this switch with caution

e2fsck -f -y -v /dev/sdb1

if this does not work, and exits along the way, we will need to remove the Journal from the EXT3 to make it an EXT2

tune2fs -O ^has_journal /dev/sdb1

Then try running the command from step 1 again

Still no luck ? let us extract all the data from the hard drive to an image file on another drive and fix it there, the other hard drive must be larger than the source hard disk, the amount of data on the source is irrelevant.

5- To make an image of /dev/sdb1 to a file, I can simply issue a dd_rescue command like this, You are advised to use the one in step 6 and not this one

dd_rescue /dev/sdb1 /directory/mydisk.img

and an image of the disk should get copied there, dd_rescue is a nice tool, But it expects you to know what you want to do and how you want to use it, if this is not the case, you can simply use dd_rhelp below and dd_rhelp will do all the magic.

6- dd_rhelp

NOTE: dd_rhelp is not really low level software, it is just the smart way to invoke dd_rescue.

dd_rhelp is a bash script that uses dd_resecue to copy the good parts of the drive first, so if your hard disk is taking it's few last spins, you can get as much good data as possible before stressing it out trying to read the busted sectors / Blocks,

To use dd_rhelp, you need to download it and copy the file dd_rhelp to a directory in the included path such as /usr/local/bin and then from the command line invoke it as follows

dd_rhelp /dev/sdb1 /destination/mydisk.img

It will try to copy the contents of sdb1 to /destination/mydisk.img that you can later mount as a drive, It will also create a log file named mydisk.img.log

Now that i have an image, i would like to check it and make it a valid ext2 or ext3 image, so i execute the command

fsck -y /destination/mydisk.img

Now that the check is done, Let us hope it did the magic so we can mount the system and get our files out !

to mount the image, We issue the commands

mkdir /backupmount
mount -o loop /destination/mydisk.img /backupmount
mount -o loop -t ext3 /destination/mydisk.img /backupmount

If, for any reason you need the disk's info... here it is, if you are following up and everything is fine this is NOT NEEDED

fdisk -l /dev/sdc

resulted in

Disk /dev/sdc: 320.0 GB, 320072933376 bytes
255 heads, 63 sectors/track, 38913 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0xf05f0548

Device Boot      Start         End      Blocks   Id  System

/dev/sdc1               1       38913   312568641   83  Linux

So, my disk or what i could get from it is now mounted on /backupmount, and i am copying the data somewhere else and will later DELETE the disk image file as soon as i have my data out to some drive i have, All my files were in the Lost+Found directory on the Mounted disk image

7- So every sign in the world says this hard drive is busted, But i have this strange belief that Western Digital hard drives never get busted, i am 99% sure that i am wrong and that it is busted, But something inside me says it can't be, so i want to check the hardware, and test it well.

There is a tool that can tell you how your hardware is doing and if it can be trusted, But to use it and be sure of the results, You need to destroy all the data on the disk by writing and then reading and then writing and so on, this is done by the w switch of the badblocks software, you can use the n switch that does not destroy your data, but it is much much slower, you can not use both n and w together, And i probably need to tell you that badblocks is a miserably slow tool, but the limitation is a hardware one not from the software itself.

The utility badblocks is the tool of choice for testing bad sectors, here is how we can make it check the disk and allow it to delete data while doing so

badblocks -swv /dev/sdb1

This would check disk sdb, the first partition, in DESTRUCTIVE read-write mode and display progress by writing out the block numbers as they are checked. As mentioned here about the -w switch, All data will be LOST !! Make sure to not make it execute silently or you would get a feeling it got stuck or halted.

And what do you know, Not a single bad block ! it took almost forever to finish the test, But it made me much more comfortable that all my disks are not dying !

Hope this helps