The new test case will make sure btrfs check is fine checking a degraded
raid5 filesystem.
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
[BUG]
For a degraded RAID5, btrfs check will fail to even read the chunk root:
# mkfs.btrfs -f -m raid5 -d raid5 $dev1 $dev2 $dev3
# wipefs -fa $dev1
# btrfs check $dev2
Opening filesystem to check...
warning, device 1 is missing
bad tree block 22036480, bytenr mismatch, want=22036480, have=0
ERROR: cannot read chunk root
ERROR: cannot open file system
[CAUSE]
Although read_tree_block() function from btrfs-progs is properly
iterating the mirrors (mirror 1 is reading from the disk directly,
mirror 2 will be rebuild from parity), the raid56 recovery path is not
handling the read error correctly.
The existing code will try to read the full stripe, but any read failure
(including missing device) will immediately cause an error:
for (i = 0; i < num_stripes; i++) {
ret = btrfs_pread(multi->stripes[i].dev->fd, pointers[i],
BTRFS_STRIPE_LEN, multi->stripes[i].physical,
fs_info->zoned);
if (ret < BTRFS_STRIPE_LEN) {
ret = -EIO;
goto out;
}
}
[FIX]
To make failed_a/failed_b calculation much easier, and properly handle
too many missing devices, here this patch will introduce a new bitmap
based solution.
The new @failed_stripe_bitmap will represent all the failed stripes.
So the initial read will mark all the missing devices in the
@failed_stripe_bitmap, and later operations will all operate on that
bitmap.
Only before we call raid56_recov(), we convert the bitmap to the old
failed_a/failed_b interface and continue.
Now btrfs check can handle above case properly:
# btrfs check $dev2
Opening filesystem to check...
warning, device 1 is missing
Checking filesystem on /dev/test/scratch2
UUID: 8b2e1cb4-f35b-4856-9b11-262d39d8458b
[1/7] checking root items
[2/7] checking extents
[3/7] checking free space tree
[4/7] checking fs roots
[5/7] checking only csums items (without verifying data)
[6/7] checking root refs
[7/7] checking quota groups skipped (not enabled on this FS)
found 147456 bytes used, no error found
total csum bytes: 0
total tree bytes: 147456
total fs tree bytes: 32768
total extent tree bytes: 16384
btree space waste bytes: 139871
file data blocks allocated: 0
referenced 0
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Currently fileattr commands, introduced in the send stream v2, always
fail, since we have commented the FS_IOC_SETFLAGS ioctl() call and set
'ret' to -EOPNOTSUPP, which is then overwritten to -errno, which may
have a random value since it was not initialized before. This results
in a failure like this:
ERROR: fileattr: set file attributes on p0/f1 failed: Invalid argument
The error reason may be something else, since errno is undefined at
this point.
Unfortunately we don't have a way yet to apply attributes, since the
attributes value we get from the kernel is what we store in flags field
of the inode item. This means that for example we can not just call
FS_IOC_SETFLAGS with the values we got, since they need to be converted
from BTRFS_INODE_* flags to FS_* flags
Besides that we'll have to reorder how we apply certain attributes like
FS_NOCOW_FL for example, which must happen always on an empty file and
right now we run write commands before attempting to change attributes,
as that's the order the kernel sends the operations.
So for now comment all the code, so that anyone using the v2 stream will
not have a receive failure but will get a behaviour like the v1 stream:
file attributes are ignored. This will have to be fixed later, but right
now people can't use a send stream v2 for the purpose of getting better
performance by avoid decompressing extents at the source and compression
of the data at the destination.
Link: https://lore.kernel.org/linux-btrfs/6cb11fa5-c60d-e65b-0295-301a694e66ad@inbox.ru/
Fixes: 8356c423e6 ("btrfs-progs: receive: implement FILEATTR command")
Signed-off-by: Filipe Manana <fdmanana@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
We're trying to get a U32 for the attributes, but the kernel sends a U64
(which is correct as we store attributes in a u64 flags field of the
inode). This makes anyone trying to receive a v2 send stream to fail with:
ERROR: invalid size for attribute, expected = 4, got = 8
We actually recently got such a report of someone using send stream v2 and
getting such failure. See the Link tag below.
Link: https://lore.kernel.org/linux-btrfs/6cb11fa5-c60d-e65b-0295-301a694e66ad@inbox.ru/
Fixes: 7a6fb356dc ("btrfs-progs: receive: process setflags ioctl commands")
Signed-off-by: Filipe Manana <fdmanana@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
This reverts commit 55438f3930.
The patch breaks resize cancel.
Reproducer:
#!/bin/bash
fallocate -l 7g /var/tmp/7g1 && fallocate -l 7g /var/tmp/7g2
thing1=$(sudo losetup --show -f /var/tmp/7g1)
thing2=$(sudo losetup --show -f /var/tmp/7g2)
echo Make the fs
mkfs.btrfs -L test539 $thing1
mkdir test539
mount -L test539 test539
echo Get rid of devid:1 by adding a new device and removing the original
btrfs dev add $thing2 test539
btrfs dev del $thing1 test539
echo Creating wiggleroom
fallocate -l 3g test539/3g1 && fallocate -l 3g test539/3g2
rm test539/3g1
echo Start a resize operation and wait 3s to run a cancel
echo Under 6.0 cancel, under 6.0.1 no cancel and runs out of space
btrfs fi re 2:-4g test539 &
sleep 3s && btrfs fi re cancel test539
wait
echo Cleanup
umount test539
losetup -d $thing1 && losetup -d $thing2
rm /var/tmp/7g{1,2}
rmdir test539
Issue: #539
Signed-off-by: David Sterba <dsterba@suse.com>
Add a command to create a new swapfile. The same can be achieved by
seandalone tools but they're just wrappers around the syscalls. The swap
format is simple enough to be created directly without mkswap command so
the swapfile can be created in one go.
The file must not exist before, this is to avoid problems with file
attributes or any other effects of existing extents. This also means the
command can't be used on block devices.
Default size is 2G, minimum size is 40KiB.
Signed-off-by: David Sterba <dsterba@suse.com>
The send stream v2 is supported if the file exists and does not contain
"1". The previous fix reversed the condition but this does not work on
kernel with v1 only support.
Signed-off-by: David Sterba <dsterba@suse.com>
We want to notrun if this test fails, not if it succeeds. Additionally
we want -s, as -q will still print an error if it gets ENOENT from the
file we're trying to grep.
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Kernel function name is btrfs_qgroup_subvolid so rename it in progs. The
libbtrfs can't API be changed without versioning so at least add the new
helper.
Signed-off-by: David Sterba <dsterba@suse.com>
A stale qgroup is level 0 and without a corresponding subvolume. There's
no convenient command for removing them and kernel does not remove them
automatically. Add a command so users don't have to parse and script the
output and/or delete them manually.
Signed-off-by: David Sterba <dsterba@suse.com>
The fact that the +C attribute excludes compression is mentioned in
<https://btrfs.readthedocs.io/en/latest/ch-compression.html#compatibility>.
Also mention it at the swapfile for clarity.
Pull-request: #530
Author: Nikolaos Chatzikonstantinou <nchatz314@gmail.com>
Reviewed-by: David Sterba <dsterba@suse.com>
Use more human readable column description and adjust the width. Use a
single "-" for an empty value as is done elsewhere too.
Sample output:
Qgroupid Referenced Exclusive Path
-------- ---------- --------- ----
0/5 16.00KiB 16.00KiB <toplevel>
0/256 16.00KiB 16.00KiB subv1
0/257 16.00KiB 16.00KiB <stale>
0/258 16.00KiB 16.00KiB dir1/subv3
0/259 16.00KiB 16.00KiB snap1
1/1 16.00KiB 16.00KiB <0 member qgroups>
Signed-off-by: David Sterba <dsterba@suse.com>
There are two column name definitions, one for sorting and one for more
human readable format but it was not used for some reason.
Signed-off-by: David Sterba <dsterba@suse.com>
Convert fputs and printf to message helpers that respect the verbosity
levels.
- print <stale> instead of <missing> for qgroups without a corresponding
subvolume after it was deleted
- print <toplevel> for toplevel
- for higher level qgroups print the number of member groups, 0 if empty
and not a special string
- drop the <FS_ROOT>
- print paths relative to toplevel path, like subvolume list does by
default
Signed-off-by: David Sterba <dsterba@suse.com>
Previous patch optionally printed the path but it would be better to
print it by default, so drop the option and verbosity. This is a
separate change as the original change was from an old pull request and
it was ported without significant changes first.
Signed-off-by: David Sterba <dsterba@suse.com>
The 'btrfs qgroup show' command currently only prints qgroup IDs,
forcing the user to resolve which subvolume each corresponds to.
Adds subvolume path resolution to 'qgroup show' so that when
the -P option is used, the last column contains the pathname of
the root of the subvolume it describes. In the case of nested
qgroups, it will show the number of member qgroups or the paths
of the members if the -v option is used.
Path can also be used as a sort parameter.
Sample output:
qgroupid rfer excl path
-------- ---- ---- ----
0/5 16.00KiB 16.00KiB <FS_ROOT>
0/256 16.00KiB 16.00KiB <FS_ROOT>/subv1
0/257 16.00KiB 16.00KiB <missing>
0/258 16.00KiB 16.00KiB <FS_ROOT>/subv3
0/259 16.00KiB 16.00KiB <FS_ROOT>/snap1
Pull-request: #139
Reviewed-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: Jeff Mahoney <jeffm@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Adds a new options -W and --wait-norescan to wait for a rescan without
starting a new operation. This is useful for things like fstests where
we want do to do a "btrfs quota enable" and not continue until the
subsequent rescan has finished.
In addition to documenting the new option in the man page, clean up the
rescan entry to document the -w option a bit better.
Pull-request: #139
Reviewed-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: Jeff Mahoney <jeffm@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
The message could be confusing in case there's no send in progress and
the real reason is lack of permissions when deleting a subvolume.
Mention the permissions as first reason. Also update documentation.
Signed-off-by: David Sterba <dsterba@suse.com>
Print warning when one of the following is requested by some command
line option:
- btrfstune -b: conversion to block-group-tree
- mkfs.btrfs --num-global-roots: extent-tree-v2
- btrfs-image -d: dump image with data
Issue: #523
Signed-off-by: David Sterba <dsterba@suse.com>
We should warn that there's an experimental feature used. Add a helper
with optional description. Should be used only if such feature is used
and not always.
Signed-off-by: David Sterba <dsterba@suse.com>
Currently we put EXTENT_TREE_V2 incompat flag entry under EXPERIMENTAL
features, thus at compile time, incompat_flags_array[] is determined at
compile time.
But the truth is, we have @supported_flags for __print_readable_flag(),
which is already defined based on EXPERIMENTAL flag.
Thus for __print_readable_flag(), we can always include the entry for
EXTENT_TREE_V2, and only print the flag if it's in the @supported_flags
By this, we can remove one EXPERIMENTAL ifdef.
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
check_resize_args() function checks user argument amount but does not
return the correct value in case it's not valid.
Signed-off-by: Sidong Yang <realwakka@gmail.com>
Signed-off-by: David Sterba <dsterba@suse.com>
When 'btrfs send --proto 2', the max buffer in kernel is changed from
BTRFS_SEND_BUF_SIZE_V1(SZ_64K) to (SZ_16K + BTRFS_MAX_COMPRESSED).
The performance is improved when we use the same buffer size in
btrfs-progs:
without this patch: 57.96s
with this patch: 48.44s
Bigger buffer size 512K was tested too, but it did not improve protocol
2 over 1 significantly.
Signed-off-by: Wang Yugui <wangyugui@e16-tech.com>
Signed-off-by: David Sterba <dsterba@suse.com>
The -O and -R help texts say that compatible version for
block-group-tree is 6.0 but it's in fact 6.1.
Issue: #523
Signed-off-by: David Sterba <dsterba@suse.com>
The option is listed among normal options but getopt does not recognize
it outside of experimental build, so make it consistent. Also mention
the correct version and need of the special build in documentation.
Signed-off-by: David Sterba <dsterba@suse.com>
As reported the documentation stated the send protocol v2 support was
supported since 5.18, but that's probably remnants of past revisions of
the patches introducing the support. Correct version is 6.0
Issue: #529
Signed-off-by: David Sterba <dsterba@suse.com>
The extent-tree-v2 is still experimental but it should be printed among
the other incompat flags if enabled by the build.
Signed-off-by: Anand Jain <anand.jain@oracle.com>
Signed-off-by: David Sterba <dsterba@suse.com>
[BUG]
Even with chunk_objectid bug fixed, mkfs.btrfs can still caused stack
overflow when enabling extent-tree-v2 feature (need experimental
features enabled):
# ./mkfs.btrfs -f -O extent-tree-v2 ~/test.img
btrfs-progs v5.19.1
See http://btrfs.wiki.kernel.org for more information.
ERROR: superblock magic doesn't match
NOTE: several default settings have changed in version 5.15, please make sure
this does not affect your deployments:
- DUP for metadata (-m dup)
- enabled no-holes (-O no-holes)
- enabled free-space-tree (-R free-space-tree)
Label: (null)
UUID: 205c61e7-f58e-4e8f-9dc2-38724f5c554b
Node size: 16384
Sector size: 4096
Filesystem size: 512.00MiB
Block group profiles:
Data: single 8.00MiB
Metadata: DUP 32.00MiB
System: DUP 8.00MiB
SSD detected: no
Zoned device: no
=================================================================
[... Skip full ASAN output ...]
==65655==ABORTING
[CAUSE]
For experimental build, we have unified feature output, but the old
buffer size is only 64 bytes, which is too small to cover the new full
feature string:
extref, skinny-metadata, no-holes, free-space-tree, block-group-tree, extent-tree-v2
Above feature string is already 84 bytes, over the 64 on-stack memory
size.
This can also be proved by the ASAN output:
==65655==ERROR: AddressSanitizer: stack-buffer-overflow on address 0x7ffc4e03b1d0 at pc 0x7ff0fc05fafe bp 0x7ffc4e03ac60 sp 0x7ffc4e03a408
WRITE of size 17 at 0x7ffc4e03b1d0 thread T0
#0 0x7ff0fc05fafd in __interceptor_strcat /usr/src/debug/gcc/libsanitizer/asan/asan_interceptors.cpp:377
#1 0x55cdb7b06ca5 in parse_features_to_string common/fsfeatures.c:316
#2 0x55cdb7b06ce1 in btrfs_parse_fs_features_to_string common/fsfeatures.c:324
#3 0x55cdb7a37226 in main mkfs/main.c:1783
#4 0x7ff0fbe3c28f (/usr/lib/libc.so.6+0x2328f)
#5 0x7ff0fbe3c349 in __libc_start_main (/usr/lib/libc.so.6+0x23349)
#6 0x55cdb7a2cb34 in _start ../sysdeps/x86_64/start.S:115
[FIX]
Introduce a new macro, BTRFS_FEATURE_STRING_BUF_SIZE, along with a new
sanity check helper, btrfs_assert_feature_buf_size().
The problem is I can not find a build time method to verify
BTRFS_FEATURE_STRING_BUF_SIZE is large enough to contain all feature
names, thus have to go the runtime function to do the BUG_ON() to verify
the macro size.
Now the minimal buffer size for experimental build is 138 bytes, just
bump it to 160 for future expansion.
And if further features go beyond that number, mkfs.btrfs/btrfs-convert
will immediately crash at that BUG_ON(), so we can definitely detect it.
Reviewed-by: Anand Jain <anand.jain@oracle.com>
Tested-by: Anand Jain <anand.jain@oracle.com>
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
[BUG]
When enabling extent-tree-v2 feature at mkfs time (need to enable
experimental features), mkfs.btrfs will crash:
# ./mkfs.btrfs -f -O extent-tree-v2 ~/test.img
btrfs-progs v5.19.1
See http://btrfs.wiki.kernel.org for more information.
ERROR: superblock magic doesn't match
NOTE: several default settings have changed in version 5.15, please make sure
this does not affect your deployments:
- DUP for metadata (-m dup)
- enabled no-holes (-O no-holes)
- enabled free-space-tree (-R free-space-tree)
Segmentation fault (core dumped)
[CAUSE]
The block group tree looks like this after make_btrfs() call:
(gdb) call btrfs_print_tree(root->fs_info->block_group_root->node, 0)
leaf 1163264 items 1 free space 16234 generation 1 owner BLOCK_GROUP_TREE
leaf 1163264 flags 0x0() backref revision 1
checksum stored f137c1ac
checksum calced f137c1ac
fs uuid 450d4b15-4954-4574-9801-8c6d248aaec6
chunk uuid 4c4cc54d-f240-4aa4-b88b-bd487db43444
item 0 key (1048576 BLOCK_GROUP_ITEM 4194304) itemoff 16259 itemsize 24
block group used 131072 chunk_objectid 256 flags SYSTEM|single
^^^
This looks completely sane, but notice that chunk_objectid 256.
That 256 value is the expected one for regular non-extent-tree-v2 btrfs,
but for extent-tree-v2, chunk_objectid is reused as the global id of
extent tree where the block group belongs to.
With the old 256 value as chunk_objectid, btrfs will not find an extent
tree root for the block group, and return NULL for btrfs_extent_root()
call, and trigger segfault.
This is a regression caused by commit 1430b41427 ("btrfs-progs:
separate block group tree from extent tree v2"), which doesn't take
extent-tree-v2 on-disk format into consideration.
[FIX]
For the initial btrfs created by make_btrfs(), all block group items
will be in extent-tree global id 0, thus we can reset chunk_objectid to
0, if and only if extent-tree-v2 is enabled.
Reviewed-by: Anand Jain <anand.jain@oracle.com>
Tested-by: Anand Jain <anand.jain@oracle.com>
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
[BUG]
The test case convert/022 will fail if the system doesn't have
reiserfs support nor reiserfs user space tools:
# make TEST=022\* test-convert
[TEST] convert-tests.sh
WARNING: reiserfs filesystem not listed in /proc/filesystems, some tests might be skipped
[TEST/conv] 022-reiserfs-parent-ref
Failed system wide prerequisities: mkreiserfs
test failed for case 022-reiserfs-parent-ref
make: *** [Makefile:443: test-convert] Error 1
[CAUSE]
Unlike other test cases, convert/022 doesn't even check if we have
kernel support for it.
[FIX]
Add the proper check before doing system wide prerequisities checks.
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
[BUG]
After upgrading to kernel v6.0-rc, btrfs-progs selftest mkfs/001 no
longer checks single device RAID0 and other new features introduced in
v5.13:
# make TEST=001\* test-mkfs
[TEST] mkfs-tests.sh
[TEST/mkfs] 001-basic-profiles
$ grep -IR "RAID0\/1" tests/mkfs-tests-results.txt
^^^ No output
[CAUSE]
The existing check_min_kernel_version() is doing an incorrect check.
The old check looks like this:
[ "$unamemajor" -lt "$argmajor" ] || return 1
[ "$unameminor" -lt "$argminor" ] || return 1
return 0
For 6.0-rc kernels, we have the following values for mkfs/001
$unamemajor = 6
$unameminor = 0
$argmajor = 5
$argminor = 12
The first check doesn't exit immediately, as 6 > 5.
Then we check the minor, which is already incorrect.
If our major is larger than target major, we should exit immediate with
0.
[FIX]
Fix the check and add extra comment.
Personally speaking I'm not a fan or short compare and return, thus all
the checks will explicit "if []; then fi" checks.
Now mkfs/001 works as expected:
# make TEST=001\* test-mkfs
[TEST] mkfs-tests.sh
[TEST/mkfs] 001-basic-profiles
$ grep -IR "RAID0\/1" tests/mkfs-tests-results.txt
Data,RAID0/1: 204.75MiB
Metadata,RAID0/1: 204.75MiB
System,RAID0/1: 8.00MiB
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
[BUG]
Commit "btrfs-progs: prepare merging compat feature lists" tries to
merged "-O" and "-R" options, as they don't correctly represents
btrfs features.
But that commit caused the following bug during mkfs for experimental
build:
$ mkfs.btrfs -f -O block-group-tree /dev/nvme0n1
btrfs-progs v5.19.1
See http://btrfs.wiki.kernel.org for more information.
ERROR: superblock magic doesn't match
ERROR: illegal nodesize 16384 (not equal to 4096 for mixed block group)
[CAUSE]
Currently btrfs_parse_fs_features() will return a u64, and reuse the
same u64 for both incompat and compat RO flags for experimental branch.
This can easily leads to conflicts, as
BTRFS_FEATURE_INCOMPAT_MIXED_BLOCK_GROUP and
BTRFS_FEATURE_COMPAT_RO_BLOCK_GROUP_TREE both share the same bit
(1 << 2).
Thus for above case, mkfs.btrfs believe it has set MIXED_BLOCK_GROUP
feature, but what we really want is BLOCK_GROUP_TREE.
[FIX]
Instead of incorrectly re-using the same bits in btrfs_feature, split
the old flags into 3 flags:
- incompat_flag
- compat_ro_flag
- runtime_flag
The first two flags are easy to understand, the corresponding flag of
each feature.
The last runtime_flag is to compensate features which doesn't have any
on-disk flag set, like QUOTA and LIST_ALL.
And since we're no longer using a single u64 as features, we have to
introduce a new structure, btrfs_mkfs_features, to contain above 3
flags.
This also mean, things like default mkfs features must be converted to
use the new structure, thus those old macros are all converted to
const static structures:
- BTRFS_MKFS_DEFAULT_FEATURES + BTRFS_MKFS_DEFAULT_RUNTIME_FEATURES
-> btrfs_mkfs_default_features
- BTRFS_CONVERT_ALLOWED_FEATURES -> btrfs_convert_allowed_features
And since we're using a structure, it's not longer as easy to implement
a disallowed mask.
Thus functions with @mask_disallowed are all changed to using
an @allowed structure pointer (which can be NULL).
Finally if we have experimental features enabled, all features can be
specified by -O options, and we can output a unified feature list,
instead of the old split ones.
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Replace fprintf(stderr, ...) by the level-aware helper instead of the
explicit verbosity level checks. No change for commands that don't have
the global -q/-v options, otherwise the output can be quieted.
Signed-off-by: David Sterba <dsterba@suse.com>
Replace fprintf(stderr, ...) by the level-aware helper. No change for
commands that don't have the global -q/-v options, otherwise the output
can be quieted.
Signed-off-by: David Sterba <dsterba@suse.com>
