Under My Hat

Tip: How to shrink a volume beyond half its size using DiskPart or Disk Manager regardless of Microsoft’s NTFS restrictions and $MFTMirr

Posted on 01 August 2010

Giveaway

If you use Disk Manager to shrink a volume, or when you go through the command line interface of DiskPart to do the same, you will often find out that it isn’t possible to shrink all available free space. And if you have a lot of free space, you cannot even ever shrink smaller than half the size of the disk. This article explains how you can solve this issue.

Method 1: how to shrink a volume with Disk Management

There are multiple ways of shrinking a volume. If you use a tool like Partition Magic, then this article does not apply to you, as it will take care of the problems that Windows XP, Vista or Windows 7 puts you in.

The easiest way is to use Control Panel > System and Security > Disk Management. Right-click the disk you want to shrink and select “Shrink volume…”. The system will now calculate the amount available to shrink and show that in a messabox, which has the following warning:

You cannot shrink a volume beyond the point where any unmovable files are located. See the “defrag” event int he Application log for detailed information about the operation when it has completed.

If you click Shrink, the volume (or partition) will be shrunken to the amount you requested. This will be succesful if there aren’t any unmovable files in that section of the volume.  If shrinking fails or the requested amount is not reached, you may find a warning in the eventlog as follows:

EventID	261
Provider	Microsoft-Windows-Defrag
Level	Warning (3)
Log Name	Application
Task Category	None
Keywords	Classic
General description	Error: a file blocked volume shrink on volume OS (V:).Diagnostic details: - The last unmovable file appears to be: \Windows\winsxs\Manifests:$I30:$BITMAP - The last cluster of the file is: 0x2b868c1 - Shrink potential target (LCN address): 0x1f54ac3 - The NTFS file flags are: —-S - Shrink phase: <check> To find more details about this file please use the “fsutil volume querycluster \\?\Volume{886db015-6083-11df-98f3-806e6f6e6963} \Windows\winsxs\Manifests:$I30:$BITMAP” command.

The file mentioned in the Event Log, “$BITMAP” is a hidden system file that’s unmovable and part of the necessary NTFS disk architecture. If you’d ever find a way to move that file by hand, you end up losing your disk due to MFT loading failures or worse. In case you wondered: it has nothing to do with pretty pictures, also sometimes called bitmaps.

Method 2: how to shrink a volume with DiskPart

After your first attempt proved unsuccessful, you decide to use the old-fashioned commandline, hoping this gets you further. Or perhaps this is your game anyway and you did this in the first place.  The tool DiskPart works slightly different then your average commandline in that it has its own command interface. If you use an operating system older then Windows Vista, Windows 2008 or Windows 7, you may need to download DiskPart, which is part of the SDK of your operating system.

The following commands are what you need to shrink a volume:

• LIST VOLUME: lists the volume and their numeric ID. Use the ID or your volume for the next command;
• SELECT VOLUME ###: replace ### with the number of the previous list, choose carefully!
• SHRINK: without any parameters will shrink the currently selected volume by its maximum, or;
• SHRINK DESIRED=50000: if you don’t want to shrink the maximum, specify the amount you want to shrink, note, this is not the desired target amount, but the amount that will be taken off your volume!

More information on how to shrink a volume usin this method or the previous method, can be found at Bleeping Computer here (the section on DiskPart is at the bottom). When you are done, you may receive the following error:

DISKPART> shrink desired=50000

Virtual Disk Service error:
The specified shrink size is too big and will cause the volume to be
smaller than the minimum volume size.

Or the following error:

DISKPART> shrink

DiskPart has encountered an  error: The parameter is incorrect.
See the System Event Log for more  information.

I asked about the first error on Serverfault.com and the second error was asked and answered here, but it comes down to the same reasons as with method 1 and 2: you need to defragment the drive to get rid of the error. But when you defragment the drive, it will not move the unmovable files which are in the middle of the disk….

After this, and after some hair-tearing, you may have tried method 3:

Method 3: backing up all data and restoring it

By now you should’ve come to the conclusion that something unmovable cannot be moved on your disks. But since you are in charge of your disks and you don’t want to delete your data, you figure: why not simply back it all up and then restore it. You choose System Backup and after you’re done, you find a nice VHD file that is smaller than the real size of your disk. So, happy with the prospect, you erase your volume and try to restore. Unfortunately, you have created a real System backup, and after you’ve restored (the system will tell you that the whole volume is needed), the same unmovable files are back in place.

Lessons learned about backup: it’s a true backup, but it’s time consuming and cannot be used with shrinking a volume.

Method 4: use the VHD (Virtual Disk) that the backup created and mount it as virtual disk

You can, indeed, use a VHD disk as if it were a real disk. You can even boot from it, or use it as the basis of a virtual Windows OS using either VMWare Viewer or Windows Virtual PC.  Because the VHD internally compresses all free space, not only the largest contiguous block, this makes sense. After this, you can remove the original volume and you can continue using the VHD (if you so prefer).

You can use the following commands with DiskPart to create the virtual disk and make it accessible:

SELECT VDISK="D:\WindowsImageBackup\CompName\Backup 2010-07-20 223352\4d591e27-f187-11dc-88fa-806e6f6e6963.vhd"
ATTACH VDISK
LIST VOLUME
SELECT VOLUME [New Volume Number]
ASSIGN LETTER=X

Tip: move the windows image backup to a more convenient location and a more convenient name. The quotes are necessary when the name contains spaces.

This actually JUST WORKS! In most cases…

The means it works when the disk is smaller than 128GB or 137GiB. That’s a hard limit of VHD, caused by historical reasons of the pre-ultra-large disk time. I won’t go into it, but suffice it to say, that some day Windows XP needed a new service pack to allow running at all on disks larger than this size. Because Virtual PC must be able to run Windows XP without these service packs, they took the shortcut and decided to hardwire this limit into VHD.

If your disk, after all the shrinking, is still above 128GB, the DiskPart commands will issue the following error:

DiskPart has encountered an  error: The parameter is incorrect.
See the System Event Log for more  information.

Again, we’re back at where we started, at least if our original disk was larger than 256GB, because in that case, some of these unmovable system files are right in the middle! Time to move on to more drastic measures and stop using Microsoft’s tools:

Method 5: use third party software to do the defragmenting

We know now that the issue is with unmovable files and the impossibility of Defrag to free up a large contiguous disk space. So perhaps there are some third party tools that can help us here.

I tried UltraDefrag and PerfectDisk (which comes in a free full functional 30 day version), but at first it seemed that these didn’t work either. The problem is: most of the defragmenting tools out there are using the same Windows Defragmentation API as Windows itself uses.

The report of UltraDefrag is left, the report of PerfectDisk is on the right (click to enlarge). Both clearly have trouble with typical NTFS system files that are part of the base structure of every disk. While technically these files could be moved (Microsoft has made that possible since Windows XP), doing so is intricate and dangerous and not available directly from the Defrag API that Microsoft provides. Maybe that’s the reason why most available defrag tools simply don’t do it.

The simple conclusion is, that these cheap and more expensive defrag tools are just a waste of your money, they only offer a more flexible interface to the already available API of Microsoft. But, there’s one exception,PerfectDisk. Risking to sound like a commercial ad, read on to the next section, Method 6:

Method 6: The Solution, using PerfectDisk in offline mode

I called the PerfectDisk company with my questions and explaining my problems so far. They pointed me at a simple tutorial, in which numer 5 (run a boot-time defrag) was of particular interest. If your disk is the OS disk and is currently active, this is your only option. But if you select this on an mounted, but not startup volume (whether it contains an OS or not doesn’t matter), PerfectDisk will offer you to take the disk offline and perform the defragmentation without restarting. The solution step-by-step:

Before you start: if you have an OS disk that is not active (i.e., you boot from another OS disk), reboot to the OS disk that you want to shrink and follow step 1 and 2 on that booted OS disk. Afterwards, restart and go back to your current OS.

Step 1: turn off system restore

This step will remove all your system restore points. Prior to doing this, you may want to create a full system backup (but prior to shrinking a volume, making a backup is always a good idea!), which will allow you to go back, including the restore points.

• Windows ME: disable system restore;
• Windows XP: turning off system restore;
• Windows Vista: turning off system restore;
• Windows 7: turning off system restore (also applies to Windows Vista, step-by-step visual guide);
• Frequently asked questions about system restore (to answer one: it eats easily 10% of your disk space!)
• Windows 2003 Server: normally this version doesn’t have system restore installed, but if you do, read this and either uninstall, or use the tips of Windows XP;
• Windows 2008 Server: doesn’t have system restore, but can use volume shadow copies, see this article on how to configure it.

Step 2: disable hibernation

This step is rather harmless, but solves issues with moving the hibernation file and improves speed. You don’t necessarily need to do this, but I highly recommend it. If you later want your hibernation file back, simply turn it back on. It only improves performance when returning from a hibernate session.

• Windows ME: how to disable hibernation;
• Windows XP: disable hibernation;
• Windows Vista, Server 2008 and Windows 7: disable/enable hibernation;
• Windows 2003: can only do this with powercfg command line;
  

Step 3: perform a first defragmentation with PerfectDisk or another defrag tool

If you haven’t downloaded PerfectDisk, you should do now, and run the installer. It will not install bloatware, nagware, spyware or adware, at least none that I have seen. It’s rare that system utilities that you pay for are really good, but this one seems to live up to its promises quite well. After startup, make sure it doesn’t do any default defragmenting (takes way too long for all disks), so make sure everything is set to “Manual”. Then:

1. In the main screen, select your disk;
2. Select Consolidate Free Space in the dropdown box in the top left;
3. Click the green play button and make yourself some coffee.

After you’re done, the Drive Map may look something like the following, where the red circle-like mark shows the unmovable parts. If you don’t see these and the whole end part is white, it means that you’re lucky and the whole disk has its maximum free space already. Note that fields marked MFT, MFT Zone and Metadata are unmovable. If you want to know what’s in a particular sector, you can rightclick it and select Show files. For the Metadata section below, this screen is placed on the right:

Here, we see the cause of all the trouble, $MFTMirr. This is an essential file and contains a copy of the first four records of the MFT Table. You’ll never notice it, the operating system will use it when these first four entries in the MFT Table have become corrupted, and even then, not even a warning will be issued. For safety reasons, this mirror image must be on a physically different location from the MFT Table. That’s why Microsoft places this in the middle of the drive. The same safety reasons prevent it from being moved easily (there’s another spot on the drive, inside the MFT Table metadata, that points to this location).

Step 4: perform a second (and third) offline defragmentation with PerfectDisk

The last league:

1. Select the Boot Time button on the Defragmentation tab;
2. If the disk is not the currently booted OS disk, you’ll get the message as in the screenshot, I suggest you select Defrag Now. Otherwise, if the currently defraggable disk is the bootdisk, select Defrag On Next Boot.
3. This defrag takes much longer, so make yourself a can of coffee.

After this defragmentation has finished, the final drive map or drive layout may look something like the following. The Metadata section containing the obnoxious $MFTMirr has been moved up a bit. If it still not completely at the end of your data. To solve that, you need to shrink the volume, and then do the offline defrag again. In my case, I didn’t mind a little free space, I just needed to get the whole size below 128GB, which I managed now. See Method 1 and Method 2 above for how to shrink the volume.

Conclusion

Microsoft has gone through great lengths to ensure NTFS integrity. One measure was to make sure that copies of vital data exist on both the beginning and the middle of the drive. In severe circumstances, this may save your day and in general, this is good design. Unfortunately, not being able to move this has backfired a bit in cases where you want to shrink your volume. Maybe nobody ever wants to shrink volumes, but considering the amount of reports on the subject, I take it that shrinking does happen and this $MFTMirr file should simply not be in our way.

I’ve come to appreciate the high level of security and safety that NTFS brings, but it really shouldn’t be this hard to do some simple repartitioning tasks. At least, thanks to PerfectDisk (haven’t found an open source alternative that can do this), I have managed to keep my sanity and old boot sections can now be turned into VHD’s of less then the required 128GB of size. That means: I can run Vista on top of Windows 7, which is what this was all about to begin with.

 

Additional comments powered by BackType