Since Windows 10, the cluster size may be greater than 128 sectors, and
it has to be recorded as a power of 2 in the boot sector. Hence there
are two possible ways of cluster size. Accept both ways leading to
valid values.
On linux the request argument of ioctl() is defined as an unsigned long,
but the fuse protocol squashes it into a signed int. As a consequence
the value received by ntfs-3g may appear as negative and different from
the value defined by the corresponding macro.
So define the request argument as unsigned long in ntfs-3g. It has
however to be fed as unsigned from fuse until the fuse protocol is
updated.
An earlier patch enabled updating a file proper id without changing
the other id (birth, volume, domain). However the first time the id
is set, these other ids have to be zeroed.
The alignment of times set in an extended attribute value cannot be
asserted, and this cause alignment errors on some CPUs (met on ARM).
Be safe by copying them in a properly aligned array.
From Windows 10 Creators edition, the cluster size limit has been
extended to 2MB. This has implied redefining the boot sector field
"sectors_per_cluster" so that values greater than 128 can be recorded.
When extents are needed to store the runlist of the MFT, the first one
must be located in record 15 so that its location can be determined from
the part in the base extent. As this record is always marked in use,
determining whether it is not really in use requires a specific logic.
Since its 2017 edition, Windows 10 has stopped mirroring $MFT to the
full size of $MFTMirr leading to mounts of partitions with big clusters
to be rejected because of mismatches. With this patch, only 16 records
are checked, though mirroring is still done for all records in $MFTMirr.
Object ids can be used to locate files which have been move to another
volume. This is only possible when the birth ids are recorded, but in most
cases files reside on their birth volume and their birth ids are not set.
The patch enables setting a file id without changing its birth id, by
setting an extended attribute "system.ntfs_object_id" limited to 16
bytes.
Windows 10 brought security descriptors which contain ACE's related to
new ways of accessing files or directories. These ACE are now accepted
with minimal consistency checks. They are still ignored for translating
permissions and for Windows-type inheritance.
When setting a security descriptor on an NTFS v1.2 format file in an
NTFS v3.0+ volume, NTFS-3G would migrate $STANDARD_INFORMATION to the
new format, which requires extending its size from 48 to 72 bytes. If
this happened while the file's MFT record was almost full, and none of
the file's attributes could be made non-resident, and the file did not
have an attribute list attribute, then the operation would unexpectedly
fail with ENOENT. Fix this by adding an attribute list to the file in
this situation.
(contributed by Eric Biggers)
Windows places filenames with a trailing dot or space in the Win32
namespace and allows setting DOS names on such files. This is true even
though on Windows such filenames can only be created and accessed using
WinNT-style paths and will confuse most Windows software. Regardless,
because libntfs-3g did not allow setting DOS names on such files, in
some cases it was impossible to correctly restore, using libntfs-3g, a
directory structure that was created under Windows.
Update ntfs_set_ntfs_dos_name() to permit operating on a file that has a
long name with a trailing dot or space. But continue to forbid creating
such names on a filesystem FUSE-mounted with the windows_name option.
Additionally, continue to forbid a trailing a dot or space in DOS names;
this matches the Windows behavior.
(contributed by Eric Biggers)
Now that the size of the reparse point attribute is no longer used by
the FUSE drivers to populate st_size for symlinks and junctions, it no
longer needs to be returned by ntfs_make_symlink().
(contributed by Eric Biggers)
valid_reparse_data() would read past the end of the reparse point buffer
if it was passed a malformed reparse point that had the tag for a
mountpoint or a symlink but had a data buffer smaller than expected.
Fix this by validating the buffer size.
(contributed by Eric Biggers)
If an output buffer was provided, ntfs_utf16_to_utf8() limited the
output string length without the terminating null to 'outs_len'. This
was incorrect because a terminating null was always added to the string,
causing a buffer overrun if the output string happened to have exactly
the maximum length. This was a longstanding bug. Fix it by leaving
space for a terminating null.
(contributed by Eric Biggers)
utf16_to_utf8_size() was not guaranteed to fail with ENAMETOOLONG if the
computed length was greater than @outs_len. This could cause a buffer
overrun in ntfs_utf16_to_utf8().
(contributed by Eric Biggers)
The validation contained an off-by-one error. The
expression '(u32)(usa_ofs + (usa_count * 2)) > size' used 'usa_count'
after it had been decremented to skip the update sequence number entry.
Consequently, the code could read out of bounds, up to two bytes past the
end of the MST-protected record.
Furthermore, as documented in the comment in layout.h for "NTFS_RECORD"
and also on MSDN for "MULTI_SECTOR_HEADER", the update sequence array
must end before the last le16 in the first logical sector --- not merely
before the end of the record.
Fix the validation and move it into a helper function, as it was done
identically in the read and write paths.
Signed-off-by: Eric Biggers <ebiggers3@gmail.com>
Currently, applications that wish to access security descriptors have to
explicitly open the volume's security descriptor index ("$Secure") using
ntfs_open_secure(). Applications are also responsible for closing the
index when done with it. However, the cleanup function for doing,
ntfs_close_secure(), cannot be called easily by all applications because
it requires a SECURITY_CONTEXT argument, not simply the ntfs_volume.
Some applications therefore have to close the inode and index contexts
manually in order to clean up properly.
This proposal updates libntfs-3g to open $Secure unconditonally as part
of ntfs_mount(), so that applications do not have to worry about it.
This proposal updates libntfs-3g to open $Secure unconditonally as part
of ntfs_mount(), so that applications do not have to worry about it.
ntfs_close_secure() is updated to take in a ntfs_volume for internal use,
and ntfs_destroy_security_context() is now the function to call to free
memory associated with a SECURITY_CONTEXT rather than a ntfs_volume.
Some memory leaks in error paths of ntfs_open_secure() are also fixed.
Signed-off-by: Eric Biggers <ebiggers3@gmail.com>
A number of functions in libntfs-3g are generally useful, but are tied to
extended attributes support and are not included when the library is
built on platforms without extended attributes support.
This proposal updates libntfs-3g to always include these functions.
The only tricky part is dealing with the XATTR_CREATE and XATTR_REPLACE
flags. These flags are defined in <sys/xattr.h>, so they must be
redefined on platforms without extended attributes support.
Signed-off-by: Eric Biggers <ebiggers3@gmail.com>
- Update documentation for COLLATION_RULES
- Document how ntfs_names_full_collate() compares names
- Update comments and DEBUG code to reflect that ntfs_names_full_collate()
always access 'upcase', even in CASE_SENSITIVE mode
- Remove unneeded assignments to 'c1' and 'c2' in IGNORE_CASE mode
Signed-off-by: Eric Biggers <ebiggers3@gmail.com>
Although ntfs_log_trace() is defined to a no-op in non-DEBUG builds,
ntfs_attr_name_get() is not. This function performs a string conversion
and a memory allocation, so it is nice to have the call to it compiled
out when not needed.
Signed-off-by: Eric Biggers <ebiggers3@gmail.com>
It was possible for ntfs_attr_name_get() to set errno due to a wide
character string that could not be converted to a multibyte string. This
caused ntfs_delete() to fail.
Fix by checking for a nonzero return value specifically from
ntfs_attr_lookup(), rather than assuming that nothing else sets errno.
Signed-off-by: Eric Biggers <ebiggers3@gmail.com>
Switch to the standard autoconf AC_HEADER_MAJOR macro which takes care
of the ugly details like when to use mkdev.h and when to use sysmacros.h.
(requires <sys/types.h> to be included)
Also include these in all files that use major/minor/makedev funcs.
(Contributed by Mike Frysinger)
The upper case value for 0x1d79 is 0xa77d, so the difference is 0x8a04,
which overflows in the table which defines the computation of upper case
values. Rewriting this difference as -0x75fc leads to the same result
in an upper case table truncated to two bytes, and this avoid the
compiler warning.
On Solaris/OpenIndiana, use the macroes makedev(), major() and minor()
from <sys/mkdev.h>. Those from <sys/sysmacros.h> are inappropriate for
current builds.
In the mailing list discussion we came to the conclusion that there
doesn't seem to be any reason to keep these declarations separate since
they address the same issue, namely libntfs-3g's tolerance for bad
Unicode data in filenames and other UTF-16 strings in the file system,
so merge the two defines into the new define ALLOW_BROKEN_UNICODE.
Windows filenames may contain invalid UTF-16 sequences (specifically
broken surrogate pairs), which cannot be converted to UTF-8 if we do
strict conversion.
This patch enables encoding broken UTF-16 into similarly broken UTF-8 by
encoding any surrogate character that don't have a match into a separate
3-byte UTF-8 sequence.
This is "sort of" valid UTF-8, but not valid Unicode since the code
points used for surrogate pair encoding are not supposed to occur in a
valid Unicode string... but on the other hand the source UTF-16 data is
also broken, so we aren't really making things any worse.
This format is sometimes referred to as WTF-8 (Wobbly Translation
Format, 8-bit encoding) and is a common solution to represent broken
UTF-16 as UTF-8.
It is a lossless round-trip conversion, i.e converting from broken
UTF-16 to "WTF-8" and back to UTF-16 yields the same broken UTF-16
sequence. Because of this property it enables accessing these files
by filename through ntfs-3g and the ntfsprogs (e.g. ls -la works as
expected).
To disable this behaviour you can pass the preprocessor/compiler flag
'-DALLOW_BROKEN_SURROGATES=0' when building ntfs-3g.
Prepare merging ntfsrecover.h into logfile.h by adding a usn field to
RESTART_PAGE_HEADER. As this changes the record size, ignore the new
field in existing code.
User extended attributes should only be set on files and directories,
not on symlinks, sockets, devices, etc. For safety they are also
forbidden on metadata files, but should be allowed on the root
directory. For files based on reparse points, requests are made
to the plugin to determine the type.
When creating a new MFT record, do not issue a warning if the current
record has bad fixups. These warnings are meaningless, difficult to
interpret and cause unneeded worries.
The new "system compression" files used by Windows 10 make use of reparse
points to record the compression parameters, and a specific named data
stream is used to store the compressed data. With this patch, processing
of reparse points can be done by an external plugin only loaded as needed.
Junctions and symlinks, which are also based on reparse points, are now
processed by "internal plugins".
For 64-bit (e.g. x86_64) Linux the 64-bit wide types resolve to long,
not long long as is the case in 32-bit (e.g. i386) Linux. So we need an
explicit cast to long long for 64-bit types since the format string must
specify the 'll' modifier in order to print 64-bit values.
These variable are only ever assigned to/from s64 values, so their type
should be s64, not u64. This fixes a compiler warning about
signed/unsigned comparison.
When looking up the lowercase equivalent of a Unicode character in
ntfs_fix_file_name, no byte swapping was performed on the ntfschar used
as index into the 'locase' array. This would lead to very strange
results on big-endian systems.
This commit addresses issues where little-endian variables are emitted
raw to a log or output stream which is to be interpreted by the user.
Outputting data in non-native endianness can cause confusion for anybody
attempting to debug issues with a file system.
When the unreadable directory has an ATTRIBUTE_LIST attribute and an
INDEX_ALLOCATION attribute occupying split over several extents, the first
of which defines a single cluster, the first INDEX_ALLOCATION extent has
lowest_vcn=0 and highest_vcn=0, and the second one has lowest_vcn=1.
This unusual case, which can be created by the combination of a small
volume and near-full MFT records, triggers some special-case behavior in
ntfs_mapping_pairs_decompress_i(). That behavior is incorrect if the
attribute's first extent only contains a single cluster, since in that case
highest_vcn=0 as well.
This configuration has been tested on Windows and it *is* able to
successfully read the directory. This supports the hypothesis that the
volume is valid and NTFS-3g has a bug on the read side.
This bug could, in theory, occur with any non-resident attribute, not just
INDEX_ALLOCATION attributes.
(Contributed by Eric Biggers)
Some constraints put on reparse points of unknown type (e.g. they cannot
be deleted) are not acceptable to archivers. This patch removes some
constraints.
Windows requires non-Microsoft reparse points (identified by having bit
31 of the reparse tag clear) to have a 16-byte GUID following the regular
reparse point header. This GUID is not, and cannot, be included in the
"reparse data length" field.
(Contributed by Eric Biggers)
Under some rare condition there is no space in an MFT entry to make
an index non-resident, and the index root has to be moved to an extent.
This fix cares for the situation when the attribute list was inserted
beforehand.
When writing to compressed data, the function ntfs_attr_pwrite()
cannot cross a compression block border. This is a problem for archivers
which rely on libntfs-3g, so the function is now wrapped in another one
which restarts the writing as needed.
ntfs_valid_sid() required that the subauthority count be between 1 and 8
inclusively. However, Windows permits more than 8 subauthorities as well
as 0 subauthorities:
- The install.wim file for the latest Windows 10 build contains a file
whose DACL contains a SID with 10 subauthorities.
ntfs_set_ntfs_acl() was failing on this file.
- The IsValidSid() function on Windows returns true for subauthority
less than or equal to 15, including 0.
There was actually already a another SID validation function that had the
Windows-compatible behavior, so I merged the two together.
Contributed by Eric Biggers
Compressed records may be written as full clusters even though cluster
tails are meaningless. This is to avoid the lower levels doing a read-
modify-write cycle. Be sure to zero the meaningless bytes to avoid
leaking information.
Contributed by Eric Biggers
When the owner and the group of a file have the same SID, and permissions
for the group is the same as permissions for other, no ACE is needed for
the group.
Newer versions of Windows use more recent definitions of upper-case
table defined by the Unicode consortium. Now using the same table as
Windows 7, windows 8 and Windows 10. This only has an effect on file
systems newly created by mkntfs.
An unused MFT record may show a bad length, leading to fetch fixups from
unallocated memory when allocating the record to a new file. So check
the length before applying the fixups. Such records have been found after
the MFT has been reallocated by a defragmenter, and they are not cleaned
by chkdsk.
When chmod'ing a file, no new ACL has to be created if the one needed
is already present in the cache. However the read-only flag may have
to be updated, so that it is kept as the opposite of S_IWUSR.
When the security attribute is present, chkdsk may set a null security id
in the standard attributes, and this should not be considered as an error.
(this partially reverts commit [70e5b1])
This patch changes the algorithm to use hash chains instead of binary
trees, with much stronger hashing. It also introduces useful (for
performance) parameters, such as the "nice match length" and "maximum
search depth", that are similar to those used in other commonly used
compression algorithms such as zlib's DEFLATE implementation.
The speed improvement is very significant, with some loss of compression
rate. The compression rate is still better than then Windows one.
Contributed by Eric Biggers
The new way goes via /sys/dev/block/MAJOR:MINOR to map partitions to
devices and get discard parameters of the parent device. It also ensures
that the partition is aligned to the discard block size.
Contributed by Richard W.M. Jones
fstrim(8) discards unused blocks on a mounted filesystem. It is useful for
solid-state drives (SSDs) and thinly-provisioned storage.
Only trimming the full device (with no option) is supported.
Contributed by Richard W.M. Jones
When Posix ACLs are used, the umask is ignored and the initial permissions
of created files are taken for the parent directory. However the umask
should still be used when the Posix ACLs are not enabled in the mount
options.
Ownership of files should always represent the creator of files.
This fixes a situation, currently disabled, where there is no user
mapping and the owner of the parent directory is used as the owner
of the created file.
When using Windows inheritance, the cacheing of ACLs for files created
within a directory only depended on the directory. Actually it also
depends on the user who creates the file. With the patch, only the ACLs
created by the owner of the directory are cached.
When grouping of users are determined from /etc/group (a compile-time
option not currently used), the groups examined for checking access rights
to a file were wrongly derived from the uid of the file instead of the
uid of the current process.
Since Vista, the standard directory /Users/Public which should be accessed
by any user is actually restricted to a few group of users, among them
the interactive ones. To make this directory accessible without using
the Posix ACLs, all Linux users are considered as interactive.
However, when Posix ACLs are used, users supposed to be interactive have
to be put into a secondary group mapped to the equivalent Windows group.
When using the Windows permission inheritance mode and the current user
has not been mapped, try to derive a reasonable user from the parent
directory.
The Windows-type inheritance of an ACE may imply creating two ACE's : one
for access and one for further inheritance. The conditions for doing so,
and the flags set on created ACE were sometimes wrong.
Note : the rules have been derived from testing multiple situations, but
there still are some gray cases.
Since Vista, Windows defines a /Users/Public directory supposed to be
public, but actually only allowed to a few user categories (interactive,
batch, etc.) This patch makes possible to create equivalent Unix groups
and group users the same way as in Windows. Posix ACLs have to be enabled
for access to /Users/Public to be allowed to several groups.
chkdsk deletes the ACLs when they are bad or when they are not used any
more. This fixes inserting a new ACL after the previously last ACL (or
even all of them) was deleted.
A bug was introduced by commit d2c7d40a2b :
when the beginning of a file was a hole and the runlist span over several
MFT extents, the runlist was not mapped on filling the initial hole.
This lead to a crash when using torrent to download big files.
Windows applies legacy restrictions to file names, so when the option
windows_names is applied, reject the same reserved names, which are
CON, PRN, AUX, NUL, COM1..COM9, and LPT1..LPT9
When the runlist of the data attribute of MFT has to be split across
several extents, the location of each extent has to be known from the
runlist present in previous extents. So, force the first extent into
record 15 to avoid a bad layout.
When a hole in a sparse file was filled, the runlist was fully recomputed.
When a sparse file spans over several MFT extents, this patch leads to
only recompute the runlist from the modified extent to the end.
Jean-Pierre André
Jean-Pierre André
Erik Larsson