Cosmos/DAS2 - Recover Nutritious Rice Data

From CompBio
Jump to: navigation, search

Introduction

The Direct Attached Storage (DAS) unit attached to cosmos, das2, has a corrupt file system. The data on the file system is related to the Nutritious Rice project.

The DAS unit itself is a Promise VTrak 8110 populated with seven ST31500341AS 1500GB drives. Six of these drives are configured as a RAID5 volume and the seventh drive is a hot spare.

The file system corruption happened as a result of an unclean system shutdown of the host cosmos.

We've ordered eight drives so that we can fully duplicate the original array, and have spares. If all goes well we will upgrade the drives in das1 (same chassis as das2) with 1500GB drives and then setting up a highly available NFS server using DRBD and heartbeat.

Plan

We're going to attempt to duplicate the drives, then run fsck against the duplicated drive array. If that fails we will try a few other TBD approaches for recovery. If all else fails we'll send the disks offsite for recovery.

status

  • 2011-01-02

I ran the following script to recover gz files that were in the base directory /das/lost+found

#!/bin/bash
# this only deals with gzip files, in /das/lost+found, that have a previous name
recovery_dir="/das/recovered"
lost_dir="/das/lost+found"
cd $lost_dir
total_ctr=0
recovered_ctr=0
for i in `find . -maxdepth 1 -type f`
do
   let total_ctr=$total_ctr+1
   fileinfo="$(file $i)"
   # check for previous name
   echo $fileinfo |grep -q -w "gzip"
   zip_check=$?
   echo $fileinfo |grep -q ", was "
   if [ $? == 0 ] && [ "$zip_check" == "0" ] ; then
      prevname="$(echo $fileinfo | awk -F"," '{print $2}' |awk -F '"' '{print $2}')"
      newname="$prevname.gz"
      echo "$i has previous name, $prevname, will be renamed to $newname"
      # see if we should move it to the recovered directory
      if [ ! -f "$recovery_dir/$newname" ] ; then
         echo "First sighting of file, moving to $recovery_dir/$newname"
         mv "$i" "$recovery_dir/$newname"
         let recovered_ctr=$recovered_ctr+1
      else
         # if the file already exists, create a prevname.$ctr file
         ctr=0;
         while [ true ] ; do
            let ctr=$ctr+1
            if  [ ! -f "$recovery_dir/$newname.$ctr" ]  ; then
               echo "not first sighting of file, determining new filename: $recovery_dir/$newname.$ctr"
               mv "$i" "$recovery_dir/$newname.$ctr"
               let recovered_ctr=$recovered_ctr+1
               break
            fi
         done
      fi
   fi
done

echo "Total files: $total_ctr"
echo "recovered files: $recovered_ctr"
echo "remaining files: $(expr $total_ctr - $recovered_ctr)"
  • 2010-12-30

The new disks have been installed to the array. Here is the logical drive as defined by the controller.

PDM Auto Start Option: Disabled
ID  Name                 RAID  SBS  CAP    PD IDs            Status
--------------------------------------------------------------------------
1   array0                5    64K  7500GB 1-6               Functional
--------------------------------------------------------------------------


Here are some relevent system logs:

/var/log root@cosmos % grep das boot.log
Dec 30 14:04:19 cosmos lvm.static:   1 logical volume(s) in volume group das_vg now active
Dec 30 14:04:52 cosmos vgchange:   1 logical volume(s) in volume group "das_vg" monitored 
Dec 30 14:04:52 cosmos lvm2-monitor: Starting monitoring for VG das_vg: succeeded 

Some output from vgdisplay shows the array is recognized by LVM

/var/log root@cosmos % vgdisplay
  --- Volume group ---
  VG Name               das_vg
  System ID             
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  1
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                1
  Open LV               0
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               6.82 TB
  PE Size               4.00 MB
  Total PE              1788139
  Alloc PE / Size       1787823 / 6.82 TB
  Free  PE / Size       316 / 1.23 GB
  VG UUID               rpEvTs-LZP0-udkH-fRqf-z5BS-nXdz-L5EH9U
   

Also, there was a log from a previous fsck attempt, and the logs of the fsck attempt ran after duplicating the disks have the exact error messages, so I'm confident transferring the data to the new drives worked.

Old fsck log:

~/das_debug root@cosmos % ls -l log.e2fsck-n-v
-rw-r--r--  1 root root 6255463 May 31  2010 log.e2fsck-n-v
~/das_debug root@cosmos % head -21 log.e2fsck-n-v 
fsck 1.35 (28-Feb-2004)
e2fsck 1.35 (28-Feb-2004)
Couldn't find ext2 superblock, trying backup blocks...
Block bitmap for group 10928 is not in group.  (block 0)
Relocate? no

Inode bitmap for group 10928 is not in group.  (block 0)
Relocate? no

Inode table for group 10928 is not in group.  (block 1236831645)
WARNING: SEVERE DATA LOSS POSSIBLE.
Relocate? no

Block bitmap for group 10929 is not in group.  (block 0)
Relocate? no

Inode bitmap for group 10929 is not in group.  (block 0)
Relocate? no

Inode table for group 10929 is not in group.  (block 0)
WARNING: SEVERE DATA LOSS POSSIBLE.

New fsck log:

~/das_debug root@cosmos % ls -l fsck.out.20101230 
-rw-r--r--  1 root root 2252762 Dec 30  2010 fsck.out.20101230
~/das_debug root@cosmos % head -20 fsck.out.20101230 
fsck 1.35 (28-Feb-2004)
Couldn't find ext2 superblock, trying backup blocks...
Block bitmap for group 10928 is not in group.  (block 0)
Relocate? no

Inode bitmap for group 10928 is not in group.  (block 0)
Relocate? no

Inode table for group 10928 is not in group.  (block 1236831645)
WARNING: SEVERE DATA LOSS POSSIBLE.
Relocate? no

Block bitmap for group 10929 is not in group.  (block 0)
Relocate? no

Inode bitmap for group 10929 is not in group.  (block 0)
Relocate? no

Inode table for group 10929 is not in group.  (block 0)
WARNING: SEVERE DATA LOSS POSSIBLE.
~/das_debug root@cosmos % 
  • 2010-12-29


The eSATA multiplier has four ports, so I was able to copy two disks at a time, I did them serially. 1,2 then 3,4 then 5,6 then 7.

Following is the output from the commands run to copy the data from drives 1,3,5 and 7. The output for drives 2,4 and 6 will follow this output. I used different windows to execute the commands, but drive 1 was coping to 1a at the same time 2 was copying to 2a.

It's interesting to see that drive 7, which was the only drive done one at a time, had an average of 50MB/s while the other drives, which were done two at a time, had a total average of 63MB/s.


root@mike-Studio-XPS-1645:~# ddrescue -c 1048576 /dev/sda /dev/sdb rice01.dd.log

Press Ctrl-C to interrupt
Initial status (read from logfile)
rescued:         0 B,  errsize:       0 B,  errors:       0
Current status
rescued:     1500 GB,  errsize:       0 B,  current rate:   47430 kB/s
   ipos:     1500 GB,   errors:       0,    average rate:   32083 kB/s
   opos:     1500 GB,     time from last successful read:       0 s
Finished                   
root@mike-Studio-XPS-1645:~# ddrescue -c 1048576 /dev/sda /dev/sdb rice03.dd.log


Press Ctrl-C to interrupt
Initial status (read from logfile)
rescued:         0 B,  errsize:       0 B,  errors:       0
Current status
rescued:     1500 GB,  errsize:       0 B,  current rate:   31620 kB/s
   ipos:     1500 GB,   errors:       0,    average rate:   31710 kB/s
   opos:     1500 GB,     time from last successful read:       0 s
Finished                   
root@mike-Studio-XPS-1645:~# ddrescue -c 1048576 /dev/sda /dev/sdb rice05.dd.log


Press Ctrl-C to interrupt
Initial status (read from logfile)
rescued:         0 B,  errsize:       0 B,  errors:       0
Current status
rescued:     1500 GB,  errsize:       0 B,  current rate:   40654 kB/s
   ipos:     1500 GB,   errors:       0,    average rate:   31988 kB/s
   opos:     1500 GB,     time from last successful read:       0 s
Finished                   
root@mike-Studio-XPS-1645:~# ddrescue -c 1048576 /dev/sda /dev/sdb rice07.dd.log


Press Ctrl-C to interrupt
Initial status (read from logfile)
rescued:         0 B,  errsize:       0 B,  errors:       0
Current status
rescued:     1500 GB,  errsize:       0 B,  current rate:   56916 kB/s
   ipos:     1500 GB,   errors:       0,    average rate:   50354 kB/s
   opos:     1500 GB,     time from last successful read:       0 s
Finished                   
root@mike-Studio-XPS-1645:~#

Now, drives 2,4 and 6, nothing exciting to see here.

root@mike-Studio-XPS-1645:~# ddrescue -c 1048576 /dev/sdc /dev/sdd rice02.dd.log


Press Ctrl-C to interrupt
Initial status (read from logfile)
rescued:         0 B,  errsize:       0 B,  errors:       0
Current status
rescued:     1500 GB,  errsize:       0 B,  current rate:   71145 kB/s
   ipos:     1500 GB,   errors:       0,    average rate:   32048 kB/s
   opos:     1500 GB,     time from last successful read:       0 s
Finished                   
root@mike-Studio-XPS-1645:~# ddrescue -c 1048576 /dev/sdc /dev/sdd rice04.dd.log


Press Ctrl-C to interrupt
Initial status (read from logfile)
rescued:         0 B,  errsize:       0 B,  errors:       0
Current status
rescued:     1500 GB,  errsize:       0 B,  current rate:   56916 kB/s
   ipos:     1500 GB,   errors:       0,    average rate:   31596 kB/s
   opos:     1500 GB,     time from last successful read:       0 s
Finished                   
root@mike-Studio-XPS-1645:~# ddrescue -c 1048576 /dev/sdc /dev/sdd rice06.dd.log


Press Ctrl-C to interrupt
Initial status (read from logfile)
rescued:         0 B,  errsize:       0 B,  errors:       0
Current status
rescued:     1500 GB,  errsize:       0 B,  current rate:   40654 kB/s
   ipos:     1500 GB,   errors:       0,    average rate:   31983 kB/s
   opos:     1500 GB,     time from last successful read:       0 s
Finished                   
root@mike-Studio-XPS-1645:~# 
  • 2010-12-28

Spent the last couple days waiting for the data transfer to finish. Decided to use ddrescue instead of dd, it seemed better suited to the task at hand.

sample output:


root@mike-Studio-XPS-1645:/home/mike# ddrescue -h
GNU ddrescue - Data recovery tool.
Copies data from one file or block device to another,
trying hard to rescue data in case of read errors.

Usage: ddrescue [options] infile outfile [logfile]
You should use a logfile unless you know what you are doing.

Options:
  -h, --help                    display this help and exit
  -V, --version                 output version information and exit
  -b, --block-size=<bytes>      sector size of input device [default 512]
  -B, --binary-prefixes         show binary multipliers in numbers [SI]
  -c, --cluster-size=<sectors>  sectors to copy at a time [128]
  -C, --complete-only           do not read new data beyond logfile limits
  -d, --direct                  use direct disc access for input file
  -D, --synchronous             use synchronous writes for output file
  -e, --max-errors=<n>          maximum number of error areas allowed
  -F, --fill=<types>            fill given type blocks with infile data (?*/-+)
  -g, --generate-logfile        generate approximate logfile from partial copy
  -i, --input-position=<pos>    starting position in input file [0]
  -m, --domain-logfile=<file>   restrict domain to finished blocks in file
  -n, --no-split                do not try to split or retry failed blocks
  -o, --output-position=<pos>   starting position in output file [ipos]
  -q, --quiet                   suppress all messages
  -r, --max-retries=<n>         exit after given retries (-1=infinity) [0]
  -R, --retrim                  mark all failed blocks as non-trimmed
  -s, --max-size=<bytes>        maximum size of input data to be copied
  -S, --sparse                  use sparse writes for output file
  -t, --truncate                truncate output file to zero size
  -T, --try-again               mark non-split, non-trimmed blocks as non-tried
  -v, --verbose                 verbose operation
Numbers may be followed by a multiplier: b = blocks, k = kB = 10^3 = 1000,
Ki = KiB = 2^10 = 1024, M = 10^6, Mi = 2^20, G = 10^9, Gi = 2^30, etc...

Report bugs to bug-ddrescue@gnu.org
Ddrescue home page: http://www.gnu.org/software/ddrescue/ddrescue.html
General help using GNU software: http://www.gnu.org/gethelp

root@mike-Studio-XPS-1645:~# ddrescue -c 1048576 /dev/sda /dev/sdb rice01.dd.log

Press Ctrl-C to interrupt
Initial status (read from logfile)
rescued:         0 B,  errsize:       0 B,  errors:       0
Current status
rescued:     1500 GB,  errsize:       0 B,  current rate:   47430 kB/s
   ipos:     1500 GB,   errors:       0,    average rate:   32083 kB/s
   opos:     1500 GB,     time from last successful read:       0 s
Finished                   
root@mike-Studio-XPS-1645:~#           



  • 2010-11-20 order submitted to purchasing
  • 2010-11-22 order approved
  • 2010-11-30 order delivered. Disks are in room 222. need to order SATA controller to duplicate disks.
Personal tools
Namespaces

Variants
Actions
Navigation
Toolbox