Fixing A Corrupt Exchange Database (29-12-2004)
2nd Christmas day Exchange nightmare - a true story !
The disk system that contains my Exchange databases had a hardware failure and damaged the database.
Although the information store was still running and the mailbox store was mounted, I notice the corruption as the backup was failing after 400mb (normally my exchange backup is 10gb.)
No big deal, I thought. I'll just restore the database from a previous backup and roll forward with the transaction logs. No data loss, just a little downtime
Then I found that a good recoverable backup was a few days old. It so happens that the corruption occurred on Christmas eve. Nobody checks Exchange events or backup logs during the holidays!
I figured that before I move mailboxes to a new database or try and fix the current one, I better image the disk right away, before my disk system will fail completely and I will be left with nothing.
I figured wrong!
After reboot the database would not mount.
Before, the database was corrupt but at least it was running.
The event ID looked like this:
Event ID 475
Information Store (3384) First Storage Group: The database page read from the file "C:\Exchsrvr\mdbdata\priv1.edb" at offset 5570543616 (0x000000014c07c000) for 4096 (0x00001000) bytes failed verification due to a page number mismatch. The expected page number was 1359995 (0x0014c07b) and the actual page number was 1546806883 (0x5c326663). The read operation will fail with error -1018 (0xfffffc06). If this condition persists then please restore the database from a previous backup. This problem is likely due to faulty hardware. Please contact your hardware vendor for further assistance diagnosing the problem.
So... my Exchange database is unstartable and my backups are relatively old. What do I do next?
Exchange includes two very sophisticated utilities that came to my rescue: Eseutil and Isinteg.
Here is how to repair an Exchange database that won't start:
1. Make a copy of the database files before you repair them. (cause you never know)
If you're not sure where your database files are, or what they are called, you can find out in Exchange System Manager by accessing the database properties. The Database page lists the paths and names.
2. Verify that you have sufficient disk space to do the repair. As a general rule of thumb, you should have the equivalent of 20% of the database size. If you don't have that much free space on the drive where the database files are, you can use command line switches to redirect the temporary files created during repair to a different drive.
3. Run Eseutil in /P (repair) mode.
The easiest way to do this is to have both database files (.EDB and .STM) in the same directory (which they usually are). If they're in different places, you're going to have to point to the files on the command line.
Eseutil is located in the \exchsrvr\bin directory
Example:
eseutil /p c:\exchsrvr\mdbdata\db1.edb /sd:\exchsrvr\mdbdata\db1.stm /te:\temp.edb
This command line will repair DB1.EDB located on C: along with its matching .STM file located on D: and will put the temporary file on the E: drive.
If your streaming database file (.STM) is not matched to the database file (.EDB) or it has a problem that is blocking repair, you can add the /i switch to the repair command line.
The /i option ignores the signature mismatch error in the check phase. The database and streaming file will receive new signatures in the repair phase.
Repair can take a while--hours. When it finishes, it will leave you with a very detailed log file of what it did called
Not quit finished yet, there are two more steps to complete.
4. Run Eseutil in /D (defragment) mode.
Repair may leave index and space allocation problems in the database. Along with compacting the physical size of the file, defragmentation rebuilds the indexes and space trees.
To defragment the database, you need space equivalent to 110% the size of the database.
As with repair, you can redirect the temporary file to a different drive if necessary, but this will take considerably longer.
5. Run Isinteg in -fix -test alltests mode.
Note: when you run Eseutil, you can move database files to temporary locations to make repairs. But to run Isinteg, you must put the database back in the location from which it is normally mounted.
At the end of an Isinteg fix run, you will probably see hundreds of warnings. This is normal as Isinteg was originally created as an internal test utility. Just make sure that at the end of a successful Isinteg run, you have zero errors reported. If there is even one error, you should run Isinteg again.
If a few runs of Isinteg do not decrease the error count down to zero, then you should not rely on this database in production. You should move mailboxes from it.
Some say you should expect to spend an hour per gigabyte of data to get through the whole repair process.
It took me 3 hours on a 9gb database though.
6. It is now safe to go home and get a heart attack.
