In converting some of our helpers to take new args I would miss some
locations because we don't stop on any warning, and I would miss the
warning in the scrollback.
Note: the Centos7 build is not yet warning-free so we can't enable
-Werror by default. It can be used as:
make EXTRA_CFLAGS=-Werror
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
[ add note ]
Signed-off-by: David Sterba <dsterba@suse.com>
When attempting an encoded write, if it fails for some specific reason
like -EINVAL (when an offset is not sector size aligned) or -ENOSPC, we
then fallback into decompressing the data and writing it using regular
buffered IO. This logic however is not correct, one of the reasons is
that it assumes the encoded offset is smaller than the unencoded file
length and that they can be compared, but one is an offset and the other
is a length, not an end offset, so they can't be compared to get correct
results. This bad logic will often result in not copying all data, or even
no data at all, resulting in a silent data loss. This is easily seen in
with the following reproducer:
$ cat test.sh
#!/bin/bash
DEV=/dev/sdj
MNT=/mnt/sdj
umount $DEV &> /dev/null
mkfs.btrfs -f $DEV > /dev/null
mount -o compress $DEV $MNT
# File foo has a size of 33K, not aligned to the sector size.
xfs_io -f -c "pwrite -S 0xab 0 33K" $MNT/foo
xfs_io -f -c "pwrite -S 0xcd 0 64K" $MNT/bar
# Now clone the first 32K of file bar into foo at offset 0.
xfs_io -c "reflink $MNT/bar 0 0 32K" $MNT/foo
# Snapshot the default subvolume and create a full send stream (v2).
btrfs subvolume snapshot -r $MNT $MNT/snap
btrfs send --compressed-data -f /tmp/test.send $MNT/snap
echo -e "\nFile bar in the original filesystem:"
od -A d -t x1 $MNT/snap/bar
umount $MNT
mkfs.btrfs -f $DEV > /dev/null
mount $DEV $MNT
echo -e "\nReceiving stream in a new filesystem..."
btrfs receive -f /tmp/test.send $MNT
echo -e "\nFile bar in the new filesystem:"
od -A d -t x1 $MNT/snap/bar
umount $MNT
Running the test without this patch:
$ ./test.sh
(...)
File bar in the original filesystem:
0000000 cd cd cd cd cd cd cd cd cd cd cd cd cd cd cd cd
*
0065536
Receiving stream in a new filesystem...
At subvol snap
File bar in the new filesystem:
0000000 cd cd cd cd cd cd cd cd cd cd cd cd cd cd cd cd
*
0033792
We end up with file bar having less data, and a smaller size, than in the
original filesystem.
This happens because when processing file bar, send issues the following
commands:
clone bar - source=foo source offset=0 offset=0 length=32768
write bar - offset=32768 length=1024
encoded_write bar - offset=33792, len=4096, unencoded_offset=33792, unencoded_file_len=31744, unencoded_len=65536, compression=1, encryption=0
The first 32K are cloned from file foo, as that file ranged is shared
between the files.
Then there's a regular write operation for the file range [32K, 33K),
since file foo has different data from bar for that file range.
Finally for the remainder of file bar, the send side issues an encoded
write since the extent is compressed in the source filesystem, for the
file offset 33792 (33K), remaining 31K of data. The receiver will try the
encoded write, but that fails with -EINVAL since the offset 33K is not
sector size aligned, so it will fallback to decompressing the data and
writing it using regular buffered writes. However that results in doing
no writes at decompress_and_write() because 'pos' is initialized to the
value of 33K (unencoded_offset) and unencoded_file_len is 31K, so the
while loop has no iterations.
Another case where we can fallback to decompression plus regular buffered
writes is when the destination filesystem has a sector size larger then
the sector size of the source filesystem (for example when the source
filesystem is on x86_64 with a 4K sector size and the destination
filesystem is PowerPC with a 64K sector size). In that scenario encoded
write attempts will fail with -EINVAL due to offsets not being aligned
with the sector size of the destination filesystem, and the receive will
attempt the fallback of decompressing the buffer and writing the
decompressed using regular buffered IO.
Fix this by tracking the number of written bytes instead, and increment
it, and the unencoded offset, after each write.
Fixes: d20e759fc9 ("btrfs-progs: receive: encoded_write fallback to explicit decode and write")
Reviewed-by: Boris Burkov <boris@bur.io>
Signed-off-by: Filipe Manana <fdmanana@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Unlike for commands from the v1 stream, we have no debug messages logged
when processing fallocate commands, which makes it harder to debug issues.
So add log messages, when the log verbosity level is >= 3, for fallocate
commands, mentioning the value of all fields.
Reviewed-by: Boris Burkov <boris@bur.io>
Signed-off-by: Filipe Manana <fdmanana@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Unlike for commands from the v1 stream, we have no debug messages logged
when processing encoded write commands, which makes it harder to debug
issues.
So add log messages, when the log verbosity level is >= 3, for encoded
write commands, mentioning the value of all fields and also log when we
fallback from an encoded write to the decompress and write approach.
The log messages look like this:
encoded_write f3 - offset=33792, len=4096, unencoded_offset=33792, unencoded_file_len=31744, unencoded_len=65536, compression=1, encryption=0
encoded_write f3 - falling back to decompress and write due to errno 22 ("Invalid argument")
Reviewed-by: Boris Burkov <boris@bur.io>
Signed-off-by: Filipe Manana <fdmanana@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Defrag should not print the filenames by default, this got accidentally
changed in v6.0. Do a workaround that restores the original behaviour,
ie. no filenames and print them with -v, either as global or local
option. Proper fix is not to initialize with BTRFS_BCONF_UNSET and only
adjust the levels by -v/-q options.
Issue: #540
Signed-off-by: David Sterba <dsterba@suse.com>
The new test case will make sure btrfs check is fine checking a degraded
raid5 filesystem.
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
[BUG]
For a degraded RAID5, btrfs check will fail to even read the chunk root:
# mkfs.btrfs -f -m raid5 -d raid5 $dev1 $dev2 $dev3
# wipefs -fa $dev1
# btrfs check $dev2
Opening filesystem to check...
warning, device 1 is missing
bad tree block 22036480, bytenr mismatch, want=22036480, have=0
ERROR: cannot read chunk root
ERROR: cannot open file system
[CAUSE]
Although read_tree_block() function from btrfs-progs is properly
iterating the mirrors (mirror 1 is reading from the disk directly,
mirror 2 will be rebuild from parity), the raid56 recovery path is not
handling the read error correctly.
The existing code will try to read the full stripe, but any read failure
(including missing device) will immediately cause an error:
for (i = 0; i < num_stripes; i++) {
ret = btrfs_pread(multi->stripes[i].dev->fd, pointers[i],
BTRFS_STRIPE_LEN, multi->stripes[i].physical,
fs_info->zoned);
if (ret < BTRFS_STRIPE_LEN) {
ret = -EIO;
goto out;
}
}
[FIX]
To make failed_a/failed_b calculation much easier, and properly handle
too many missing devices, here this patch will introduce a new bitmap
based solution.
The new @failed_stripe_bitmap will represent all the failed stripes.
So the initial read will mark all the missing devices in the
@failed_stripe_bitmap, and later operations will all operate on that
bitmap.
Only before we call raid56_recov(), we convert the bitmap to the old
failed_a/failed_b interface and continue.
Now btrfs check can handle above case properly:
# btrfs check $dev2
Opening filesystem to check...
warning, device 1 is missing
Checking filesystem on /dev/test/scratch2
UUID: 8b2e1cb4-f35b-4856-9b11-262d39d8458b
[1/7] checking root items
[2/7] checking extents
[3/7] checking free space tree
[4/7] checking fs roots
[5/7] checking only csums items (without verifying data)
[6/7] checking root refs
[7/7] checking quota groups skipped (not enabled on this FS)
found 147456 bytes used, no error found
total csum bytes: 0
total tree bytes: 147456
total fs tree bytes: 32768
total extent tree bytes: 16384
btree space waste bytes: 139871
file data blocks allocated: 0
referenced 0
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Currently fileattr commands, introduced in the send stream v2, always
fail, since we have commented the FS_IOC_SETFLAGS ioctl() call and set
'ret' to -EOPNOTSUPP, which is then overwritten to -errno, which may
have a random value since it was not initialized before. This results
in a failure like this:
ERROR: fileattr: set file attributes on p0/f1 failed: Invalid argument
The error reason may be something else, since errno is undefined at
this point.
Unfortunately we don't have a way yet to apply attributes, since the
attributes value we get from the kernel is what we store in flags field
of the inode item. This means that for example we can not just call
FS_IOC_SETFLAGS with the values we got, since they need to be converted
from BTRFS_INODE_* flags to FS_* flags
Besides that we'll have to reorder how we apply certain attributes like
FS_NOCOW_FL for example, which must happen always on an empty file and
right now we run write commands before attempting to change attributes,
as that's the order the kernel sends the operations.
So for now comment all the code, so that anyone using the v2 stream will
not have a receive failure but will get a behaviour like the v1 stream:
file attributes are ignored. This will have to be fixed later, but right
now people can't use a send stream v2 for the purpose of getting better
performance by avoid decompressing extents at the source and compression
of the data at the destination.
Link: https://lore.kernel.org/linux-btrfs/6cb11fa5-c60d-e65b-0295-301a694e66ad@inbox.ru/
Fixes: 8356c423e6 ("btrfs-progs: receive: implement FILEATTR command")
Signed-off-by: Filipe Manana <fdmanana@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
We're trying to get a U32 for the attributes, but the kernel sends a U64
(which is correct as we store attributes in a u64 flags field of the
inode). This makes anyone trying to receive a v2 send stream to fail with:
ERROR: invalid size for attribute, expected = 4, got = 8
We actually recently got such a report of someone using send stream v2 and
getting such failure. See the Link tag below.
Link: https://lore.kernel.org/linux-btrfs/6cb11fa5-c60d-e65b-0295-301a694e66ad@inbox.ru/
Fixes: 7a6fb356dc ("btrfs-progs: receive: process setflags ioctl commands")
Signed-off-by: Filipe Manana <fdmanana@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
This reverts commit 55438f3930.
The patch breaks resize cancel.
Reproducer:
#!/bin/bash
fallocate -l 7g /var/tmp/7g1 && fallocate -l 7g /var/tmp/7g2
thing1=$(sudo losetup --show -f /var/tmp/7g1)
thing2=$(sudo losetup --show -f /var/tmp/7g2)
echo Make the fs
mkfs.btrfs -L test539 $thing1
mkdir test539
mount -L test539 test539
echo Get rid of devid:1 by adding a new device and removing the original
btrfs dev add $thing2 test539
btrfs dev del $thing1 test539
echo Creating wiggleroom
fallocate -l 3g test539/3g1 && fallocate -l 3g test539/3g2
rm test539/3g1
echo Start a resize operation and wait 3s to run a cancel
echo Under 6.0 cancel, under 6.0.1 no cancel and runs out of space
btrfs fi re 2:-4g test539 &
sleep 3s && btrfs fi re cancel test539
wait
echo Cleanup
umount test539
losetup -d $thing1 && losetup -d $thing2
rm /var/tmp/7g{1,2}
rmdir test539
Issue: #539
Signed-off-by: David Sterba <dsterba@suse.com>
Add a command to create a new swapfile. The same can be achieved by
seandalone tools but they're just wrappers around the syscalls. The swap
format is simple enough to be created directly without mkswap command so
the swapfile can be created in one go.
The file must not exist before, this is to avoid problems with file
attributes or any other effects of existing extents. This also means the
command can't be used on block devices.
Default size is 2G, minimum size is 40KiB.
Signed-off-by: David Sterba <dsterba@suse.com>
The send stream v2 is supported if the file exists and does not contain
"1". The previous fix reversed the condition but this does not work on
kernel with v1 only support.
Signed-off-by: David Sterba <dsterba@suse.com>
We want to notrun if this test fails, not if it succeeds. Additionally
we want -s, as -q will still print an error if it gets ENOENT from the
file we're trying to grep.
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Kernel function name is btrfs_qgroup_subvolid so rename it in progs. The
libbtrfs can't API be changed without versioning so at least add the new
helper.
Signed-off-by: David Sterba <dsterba@suse.com>
A stale qgroup is level 0 and without a corresponding subvolume. There's
no convenient command for removing them and kernel does not remove them
automatically. Add a command so users don't have to parse and script the
output and/or delete them manually.
Signed-off-by: David Sterba <dsterba@suse.com>
The fact that the +C attribute excludes compression is mentioned in
<https://btrfs.readthedocs.io/en/latest/ch-compression.html#compatibility>.
Also mention it at the swapfile for clarity.
Pull-request: #530
Author: Nikolaos Chatzikonstantinou <nchatz314@gmail.com>
Reviewed-by: David Sterba <dsterba@suse.com>
Use more human readable column description and adjust the width. Use a
single "-" for an empty value as is done elsewhere too.
Sample output:
Qgroupid Referenced Exclusive Path
-------- ---------- --------- ----
0/5 16.00KiB 16.00KiB <toplevel>
0/256 16.00KiB 16.00KiB subv1
0/257 16.00KiB 16.00KiB <stale>
0/258 16.00KiB 16.00KiB dir1/subv3
0/259 16.00KiB 16.00KiB snap1
1/1 16.00KiB 16.00KiB <0 member qgroups>
Signed-off-by: David Sterba <dsterba@suse.com>
There are two column name definitions, one for sorting and one for more
human readable format but it was not used for some reason.
Signed-off-by: David Sterba <dsterba@suse.com>
Convert fputs and printf to message helpers that respect the verbosity
levels.
- print <stale> instead of <missing> for qgroups without a corresponding
subvolume after it was deleted
- print <toplevel> for toplevel
- for higher level qgroups print the number of member groups, 0 if empty
and not a special string
- drop the <FS_ROOT>
- print paths relative to toplevel path, like subvolume list does by
default
Signed-off-by: David Sterba <dsterba@suse.com>
Previous patch optionally printed the path but it would be better to
print it by default, so drop the option and verbosity. This is a
separate change as the original change was from an old pull request and
it was ported without significant changes first.
Signed-off-by: David Sterba <dsterba@suse.com>
The 'btrfs qgroup show' command currently only prints qgroup IDs,
forcing the user to resolve which subvolume each corresponds to.
Adds subvolume path resolution to 'qgroup show' so that when
the -P option is used, the last column contains the pathname of
the root of the subvolume it describes. In the case of nested
qgroups, it will show the number of member qgroups or the paths
of the members if the -v option is used.
Path can also be used as a sort parameter.
Sample output:
qgroupid rfer excl path
-------- ---- ---- ----
0/5 16.00KiB 16.00KiB <FS_ROOT>
0/256 16.00KiB 16.00KiB <FS_ROOT>/subv1
0/257 16.00KiB 16.00KiB <missing>
0/258 16.00KiB 16.00KiB <FS_ROOT>/subv3
0/259 16.00KiB 16.00KiB <FS_ROOT>/snap1
Pull-request: #139
Reviewed-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: Jeff Mahoney <jeffm@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Adds a new options -W and --wait-norescan to wait for a rescan without
starting a new operation. This is useful for things like fstests where
we want do to do a "btrfs quota enable" and not continue until the
subsequent rescan has finished.
In addition to documenting the new option in the man page, clean up the
rescan entry to document the -w option a bit better.
Pull-request: #139
Reviewed-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: Jeff Mahoney <jeffm@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
The message could be confusing in case there's no send in progress and
the real reason is lack of permissions when deleting a subvolume.
Mention the permissions as first reason. Also update documentation.
Signed-off-by: David Sterba <dsterba@suse.com>
Print warning when one of the following is requested by some command
line option:
- btrfstune -b: conversion to block-group-tree
- mkfs.btrfs --num-global-roots: extent-tree-v2
- btrfs-image -d: dump image with data
Issue: #523
Signed-off-by: David Sterba <dsterba@suse.com>
We should warn that there's an experimental feature used. Add a helper
with optional description. Should be used only if such feature is used
and not always.
Signed-off-by: David Sterba <dsterba@suse.com>
Currently we put EXTENT_TREE_V2 incompat flag entry under EXPERIMENTAL
features, thus at compile time, incompat_flags_array[] is determined at
compile time.
But the truth is, we have @supported_flags for __print_readable_flag(),
which is already defined based on EXPERIMENTAL flag.
Thus for __print_readable_flag(), we can always include the entry for
EXTENT_TREE_V2, and only print the flag if it's in the @supported_flags
By this, we can remove one EXPERIMENTAL ifdef.
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
check_resize_args() function checks user argument amount but does not
return the correct value in case it's not valid.
Signed-off-by: Sidong Yang <realwakka@gmail.com>
Signed-off-by: David Sterba <dsterba@suse.com>
When 'btrfs send --proto 2', the max buffer in kernel is changed from
BTRFS_SEND_BUF_SIZE_V1(SZ_64K) to (SZ_16K + BTRFS_MAX_COMPRESSED).
The performance is improved when we use the same buffer size in
btrfs-progs:
without this patch: 57.96s
with this patch: 48.44s
Bigger buffer size 512K was tested too, but it did not improve protocol
2 over 1 significantly.
Signed-off-by: Wang Yugui <wangyugui@e16-tech.com>
Signed-off-by: David Sterba <dsterba@suse.com>
The -O and -R help texts say that compatible version for
block-group-tree is 6.0 but it's in fact 6.1.
Issue: #523
Signed-off-by: David Sterba <dsterba@suse.com>
The option is listed among normal options but getopt does not recognize
it outside of experimental build, so make it consistent. Also mention
the correct version and need of the special build in documentation.
Signed-off-by: David Sterba <dsterba@suse.com>
As reported the documentation stated the send protocol v2 support was
supported since 5.18, but that's probably remnants of past revisions of
the patches introducing the support. Correct version is 6.0
Issue: #529
Signed-off-by: David Sterba <dsterba@suse.com>
The extent-tree-v2 is still experimental but it should be printed among
the other incompat flags if enabled by the build.
Signed-off-by: Anand Jain <anand.jain@oracle.com>
Signed-off-by: David Sterba <dsterba@suse.com>
[BUG]
Even with chunk_objectid bug fixed, mkfs.btrfs can still caused stack
overflow when enabling extent-tree-v2 feature (need experimental
features enabled):
# ./mkfs.btrfs -f -O extent-tree-v2 ~/test.img
btrfs-progs v5.19.1
See http://btrfs.wiki.kernel.org for more information.
ERROR: superblock magic doesn't match
NOTE: several default settings have changed in version 5.15, please make sure
this does not affect your deployments:
- DUP for metadata (-m dup)
- enabled no-holes (-O no-holes)
- enabled free-space-tree (-R free-space-tree)
Label: (null)
UUID: 205c61e7-f58e-4e8f-9dc2-38724f5c554b
Node size: 16384
Sector size: 4096
Filesystem size: 512.00MiB
Block group profiles:
Data: single 8.00MiB
Metadata: DUP 32.00MiB
System: DUP 8.00MiB
SSD detected: no
Zoned device: no
=================================================================
[... Skip full ASAN output ...]
==65655==ABORTING
[CAUSE]
For experimental build, we have unified feature output, but the old
buffer size is only 64 bytes, which is too small to cover the new full
feature string:
extref, skinny-metadata, no-holes, free-space-tree, block-group-tree, extent-tree-v2
Above feature string is already 84 bytes, over the 64 on-stack memory
size.
This can also be proved by the ASAN output:
==65655==ERROR: AddressSanitizer: stack-buffer-overflow on address 0x7ffc4e03b1d0 at pc 0x7ff0fc05fafe bp 0x7ffc4e03ac60 sp 0x7ffc4e03a408
WRITE of size 17 at 0x7ffc4e03b1d0 thread T0
#0 0x7ff0fc05fafd in __interceptor_strcat /usr/src/debug/gcc/libsanitizer/asan/asan_interceptors.cpp:377
#1 0x55cdb7b06ca5 in parse_features_to_string common/fsfeatures.c:316
#2 0x55cdb7b06ce1 in btrfs_parse_fs_features_to_string common/fsfeatures.c:324
#3 0x55cdb7a37226 in main mkfs/main.c:1783
#4 0x7ff0fbe3c28f (/usr/lib/libc.so.6+0x2328f)
#5 0x7ff0fbe3c349 in __libc_start_main (/usr/lib/libc.so.6+0x23349)
#6 0x55cdb7a2cb34 in _start ../sysdeps/x86_64/start.S:115
[FIX]
Introduce a new macro, BTRFS_FEATURE_STRING_BUF_SIZE, along with a new
sanity check helper, btrfs_assert_feature_buf_size().
The problem is I can not find a build time method to verify
BTRFS_FEATURE_STRING_BUF_SIZE is large enough to contain all feature
names, thus have to go the runtime function to do the BUG_ON() to verify
the macro size.
Now the minimal buffer size for experimental build is 138 bytes, just
bump it to 160 for future expansion.
And if further features go beyond that number, mkfs.btrfs/btrfs-convert
will immediately crash at that BUG_ON(), so we can definitely detect it.
Reviewed-by: Anand Jain <anand.jain@oracle.com>
Tested-by: Anand Jain <anand.jain@oracle.com>
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
[BUG]
When enabling extent-tree-v2 feature at mkfs time (need to enable
experimental features), mkfs.btrfs will crash:
# ./mkfs.btrfs -f -O extent-tree-v2 ~/test.img
btrfs-progs v5.19.1
See http://btrfs.wiki.kernel.org for more information.
ERROR: superblock magic doesn't match
NOTE: several default settings have changed in version 5.15, please make sure
this does not affect your deployments:
- DUP for metadata (-m dup)
- enabled no-holes (-O no-holes)
- enabled free-space-tree (-R free-space-tree)
Segmentation fault (core dumped)
[CAUSE]
The block group tree looks like this after make_btrfs() call:
(gdb) call btrfs_print_tree(root->fs_info->block_group_root->node, 0)
leaf 1163264 items 1 free space 16234 generation 1 owner BLOCK_GROUP_TREE
leaf 1163264 flags 0x0() backref revision 1
checksum stored f137c1ac
checksum calced f137c1ac
fs uuid 450d4b15-4954-4574-9801-8c6d248aaec6
chunk uuid 4c4cc54d-f240-4aa4-b88b-bd487db43444
item 0 key (1048576 BLOCK_GROUP_ITEM 4194304) itemoff 16259 itemsize 24
block group used 131072 chunk_objectid 256 flags SYSTEM|single
^^^
This looks completely sane, but notice that chunk_objectid 256.
That 256 value is the expected one for regular non-extent-tree-v2 btrfs,
but for extent-tree-v2, chunk_objectid is reused as the global id of
extent tree where the block group belongs to.
With the old 256 value as chunk_objectid, btrfs will not find an extent
tree root for the block group, and return NULL for btrfs_extent_root()
call, and trigger segfault.
This is a regression caused by commit 1430b41427 ("btrfs-progs:
separate block group tree from extent tree v2"), which doesn't take
extent-tree-v2 on-disk format into consideration.
[FIX]
For the initial btrfs created by make_btrfs(), all block group items
will be in extent-tree global id 0, thus we can reset chunk_objectid to
0, if and only if extent-tree-v2 is enabled.
Reviewed-by: Anand Jain <anand.jain@oracle.com>
Tested-by: Anand Jain <anand.jain@oracle.com>
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
