When ntfsclone'ing to a file, the target file was truncated to the volume
size. This is not useful on file systems which support sparse files. In
the case of ntfs-3g this leads to prevent optimizations specific to
appending data. So when a sparse output file is detected, it is emptied
to benefit from subsequent appending of data.
The upcoming libgrypt-1.6 drops the "module" interface which was used
by ntfsdecrypt for decrypting files which were encrypted with the "DESX"
algorithm. This algorithm is a Microsoft variant of DES with a key size
of 128 bits, and is not natively supported by libgrypt. The module interface
made possible to declare an external algorithm so that all the encryption
modes could be processed the same way whether the algorithm was internal
or external.
This patch makes DESX a specific case, so that the module interface is
not needed any more. It is compatible with current libgrypt and upcoming
libgrypt-1.6
When trying to undelete a file which was modified several times the
same day, it is difficult to tell which one is the latest unless the
modification time is displayed.
Traditionally the backup boot sector is not set by ntfsresize because
the exact partition size is not yet known. A chkdsk is triggered to
subsequently insert it.
However ntfsresize is frequently activated by a partition editor which
has an exact knowledge of the wanted layout, and the backup boot sector
can be inserted by ntfsresize. This is only done if the target partition
size is defined with no unit suffix, and it is a multiple of the sector
size. Anyway the backup boot sector cannot overwrite useful data as it
is inserted between the file system size and the target partition size.
The usa sequence is used to make sure the sectors containing metadata
are fully written. The values are always put at the end of 512-byte
chunks even if the sectors are greater.
When the MFT is partially located beyond the end of a resized partition,
it has to be relocated after the other files have been processed
because the MFT data is needed to do the relocations. When the MFT runlist
is split over extents, all of them have to be processed.
When the beginning of the MFT is beyond the end of the resized
partition, a specific procedure has to be used to relocate this part
of the MFT and adjust the bitmap accordingly. On a test run, these
updates should not be done.
The long long printing formats (such as %lld or %llx) have to be
translated to %I64 on older Windows systems, and the buffers to
receive the translated format must be able to hold the tool banners
which can be quite lengthy.
The size argument of ntfsresize is the target partition size. This is
not the file system size as returned by df(1) or statvfs(3) because a
sector is reserved for the backup boot sector, and the remaining sector
count might not be a multiple of clusters per sector.
When a sector is unreadable, consider the whole sector as bad, instead
of trying to find which 512-byte parts of it are bad. This was causing
unwanted retries for devices with 4K sectors.
Older msvcrt.dll (on XP and earlier) did not support "%ll" print editing
formats frequently used by ntfsclone. So translate them to "%I64" when
running on Windows. This format appears to be supported by all Windows
versions. Error messages from libntfs-3g are still not translated.
This mainly means using the functions in win32_io.c instead of the
standard msvcrt.dll ones in order to be able to truncate files and
create sparse ones.
This mainly means using the functions in win32_io.c instead of the
standard msvcrt.dll ones in order to be able to truncate files and
create sparse ones.
The summary of 'runcount.runs' and 'runcount.fragments' was displayed in
non-verbose mode. However these variables are not updated in non-verbose
mode, so they would always be 0.
We now only display this summary in verbose mode.
The filtering of files to undelete is usually done through the regex
library. This patch offers an alternate way when such a library is
not readily available (typically on Windows).
When ntfsundelete scans the $MFT for possible deleted files, it may
examine extries which have never been used, producing error messages
which most users do not understand. This patch silences such messages.
Allow the "--ignore-fs-check" option to be used when cloning a file system
which has allocation errors. This is useful when there is a fear that
repairing the file system might create further damage, or when the repair
has to be made on another computer.
This patch adds a new option -n to ntfsclone for simulating a restore
without writing anything. This is useful for checking the consistency
of an image without destroying the original partition.
Upgrade the Win32 interface (win32_io.c) which was designed for Cygwin
so that it can be used for using the ntfsprogs utilities on native Windows.
Two new entries are added for truncating a file and creating a sparse
file, both of which not being supported through msvcrt.dll.
When creating a metadata image, all user data is supposed to be wiped off,
but this was wrong for $MFT and directory indexes when they were
fragmented. This patch cleans the tails of metadata fragments.
The MS_* flags originated from system constants. However the flags
passed to ntfs_mount were really unrelated to the system constants and
many new MS_* flags had to be introduced as different features were
added to the library. Those flags had no counterparts in any system
APIs, so using the same naming scheme is inappropriate.
Instead, let's namespace these flags similarly to what has already been
done in ntfsprogs/libntfs earlier. This avoids any possible conflicts
with system constants.
The values of the flags themselves are kept the same as earlier, so
backward compatibility is retained.
The variable 'flags' was modified along the way and did not contain the
original flags at the end as assumed by the last printout. Fixed by
storing the original state of the flags in a temporary const variable.
Since Windows 8 a new stream is added to the upper case file to record
a CRC of the upper case data. This way, if the file does not match the
one defined on the current Windows system, chkdsk can tell whether the
file is damaged or just out of date, and an error is only displayed if
the file is damaged.
The $Info data has been checked to be ignored by Windows XP, Windows
Vista and Windows 7. If not present, chkdsk for Windows 8 complains about
a wrong upper case file.
The named attributes sizes were wrongly computed in mkntfs. This did not
lead to visible errors so far owing to 8-byte alignments in attribute
records (current names $SDS, $R, $O, etc).
ntfsfix now checks and update the backup boot sector on the last sector of
the partition (instead of the sector next to end of the file system). After
an ntfsresize the file system generally does not use the maximum size because
of different roundings and required alignments in the resizing and the
repartitioning.
Defined new options --new-serial and --new-half-serial to set a new
serial number (either the given one or a random one) to a file system.
Useful for mounting the original and the cloned file system at the
same time.
Defined new options --new-serial and --new-half-serial to set a new
random serial number when cloning or restoring a file system.
Useful for mounting the original and the cloned file system at the
same time.
Clearing of user data was wrong when extracting the metadata into a
special format image. In this situation the clearing has to be done
without mounting the file system.
When extracting the metadata, unused data at the end of MFT records is
supposed to be cleared. This was done for the base record of each file,
but not for the extent records.
Clearing the bad cluster list was done by truncating $BadClus:$Bad,
this turned out not to be reliable because chkdsk does not adjust
the size of $BadClus:$Bad when declaring a cluster bad.
This avoids name collisions with Mac OS X system headers (specifically
/usr/include/sys/queue.h). It's quite possible that other operating
systems also have similarly named macros in their system headers since
the function/macro names are very generic.
'wipe_mft' didn't write any $MFTMirr entries for newly manufactured
entries (i.e. entries that were unused and therefore competely wiped and
replaced with a newly initialized copy).
The dereferencing of 'offset' and 'usa' from the MFT record wasn't an
le16* dereference but a u8* dereference, leading to only the least
significant byte (little-endian systems) or the most significant byte
(big-endian systems) being part of the value. (So while this bug could
go unnoticed on little-endian systems for volumes with small values of
'usa', it caused even more significant problems on big-endian systems.)
Fixed by properly casting the address to le16* before dereferencing.
'wipe_mft' was hardcoded to use the USA size of a 1024 byte MFT record
when creating new blank MFT entries. It now calculates the USA size
based on vol->mft_record_size.
This bug led 'utils_mftrec_in_use' to use 'cached' data before it had
been initialized (i.e. its contents was undefined), producing incorrect
results for the first 3584 MFT entries.
When the cluster size is bigger than the index block size, the index
block size unit is 512 (not the sector size) instead of the cluster size.
The partitions formatted by mkntfs and used by ntfs-3g were not
interoperable with Windows when the cluster size is bigger than 4K
and the sector size is not 512.
When undeleting a file whose name cannot be recovered, this patch
defines a name to which the inode number is appended to facilitate
a bulk recovery (recovering the unwanted deletion of a subtree).
When a file is deleted, ntfs-3g deletes the name from the MFT record,
so the name is not available when the file has to be undeleted.
This patch tries to recover the name when it has not been overwritten.
This is mostly possible when data is non-resident and not fragmented.
Under some rare and obscure circumstances probably unrelated to ntfs-3g,
a part of the runlist of MFT describes its own location, therefore
it cannot be loaded. This patch relocates the MFT extent to inode 15
to fix this. Note : chkdsk cannot fix it and destroys all the files.
When the lower bound of an ntfs partition is moved down this patch
recreated new metadata in the expanded space without copying the
actual data. The existing code for moving the upper bound of the
partition has been kept unchanged.
Logging of fixup errors for uninitialized inodes cause unnecessary
worries and suspicion of malfunctions in ntfs-3g. This patch silences
these loggings in ntfsclone and ntfsresize which have to analyze all
inodes, including the uninitialized ones.
The label changing code in ntfslabel was cleaned up and modified to use the more
advanced functionality of libntfs-3g instead of using older custom code to
resize and create resident attributes.
The core label changing functionality was also moved into the library so it can
be reused by other programs.
After the '--enable-extras' patch, EXTRA_PROGRAMS and bin_PROGRAMS were both defined when ENABLE_EXTRAS was enabled.
This created a conflict, where at least automake 1.10 attempted to emit targets for the binaries twice, resulting in failure when running 'autogen.sh'.
Conflict was resolved by defining the names of the binaries in a non-reserved variable EXTRA_PROGRAM_NAMES, which is then included either in EXTRA_PROGRAMS or bin_PROGRAMS but never in both.
When the normal boot sector is not usable, ntfsfix tries to use the last
sector as a boot sector replacement. This implies getting the sector size
and reading both full-sized boot sectors.
When creating a partition image, ntfsclone write an extra byte to
describe each cluster, this causes two unneeded ntfs-3g calls per cluster,
and inefficiency when imaging to ntfs.
The recorded allocated size depends on unnamed data being non resident,
which depends on MFT record size, hence on sector size.
The allocated size was wrong for $AttrDef when sector size is 4K bytes
(chkdsk silently fixes it).
A recent patch added copying the backup boot sector when cloning,
saving or restoring a partition, but when the total number of sectors is
not a multiple of the number of sectors per cluster, the last cluster
containing the backup boot sector is shorter than usual.
When this happens, the present patch avoids overflowing from partition
when accessing the last cluster.
Note: The NTFS_MNT_FORCE, in addition to what MS_RECOVER does, also bypasses the check for the 'dirty' bit in libntfs' ntfs_mount.
However, this check does not exist in libntfs-3g (libntfs-3g will not check or change the dirty bit, being confident that it can handle volumes marked as 'dirty'), so in essence the same behaviour is achieved with MS_RECOVER.
libntfs (unlike libntfs-3g) sets the volume dirty bit automatically on ntfs_mount (if not already set) and also automatically clears the volume dirty bit on ntfs_umount (if it was not already set before mount). The 'WasDirty' flag is set to indicate that the volume was already dirty when it was mounted, so setting it means bypassing the 'clear dirty flag' behaviour on unmount (it does not mean 'set dirty flag on unmount').
NVolSetWasDirty is accordingly replaced with the actions that were intended:
- If the intention was to leave the dirty bit set on unmount, we explicitly set the bit if it is not already set.
- If the intention was to simply update the 'WasDirty' bit to be consistent with earlier changes, we just comment out the statement.
This is not a perfect conversion. The VOLUME_IS_DIRTY flag reflects the _current_ state of the volume dirty bit and not the mount-time state.
However, since libntfs-3g (as opposed to libntfs) does not automatically change the dirty bit on mount and unmount (only when ntfs_volume_write_flags is called explicitly), and these tests are done directly after a mount (ntfsclone.c, ntfscp.c, ntfsresize.c, ntfswipe.c, utils.c) or when the volume is in an appropriate state (ntfsfix.c), the result will be the same.
Jean-Pierre André
Erik Larsson
Szaka