Now that all callers are using the _nr variations we can simply rename
these helpers to btrfs_item_##member/btrfs_set_item_##member and change
the actual item SETGET funcs to raw_item_##member/set_raw_item_##member
and then change all callers to drop the _nr part.
Reviewed-by: Nikolay Borisov <nborisov@suse.com>
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: David Sterba <dsterba@suse.com>
All callers use the btrfs_item_end_nr() variation, simply drop
btrfs_item_end() and make btrfs_item_end_nr() use the _nr() variations
of the item get helpers.
Reviewed-by: Nikolay Borisov <nborisov@suse.com>
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: David Sterba <dsterba@suse.com>
This matches how the kernel does it, simply pass in the slot and fix up
btrfs_file_extent_inline_item_len to use the btrfs_item_nr() helper and
the correct define. Fixup all the callers to use the slot now instead
of passing in the btrfs_item.
Reviewed-by: Nikolay Borisov <nborisov@suse.com>
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: David Sterba <dsterba@suse.com>
We have a lot of the following patterns
item = btrfs_item_nr(nr);
btrfs_set_item_*(eb, item, val);
btrfs_set_item_*(eb, btrfs_item_nr(nr), val);
in a lot of places in our code. Instead add _nr variations of these
helpers and convert all of the users to this new helper.
Reviewed-by: Nikolay Borisov <nborisov@suse.com>
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: David Sterba <dsterba@suse.com>
We have this pattern in a lot of places
item = btrfs_item_nr(slot);
btrfs_item_size(leaf, item);
btrfs_item_offset(leaf, item);
when we could simply use
btrfs_item_size_nr(leaf, slot);
btrfs_item_offset_nr(leaf, slot);
Fix all callers of btrfs_item_size() and btrfs_item_offset() to use the
_nr variation of the helpers.
Reviewed-by: Nikolay Borisov <nborisov@suse.com>
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: David Sterba <dsterba@suse.com>
This helper only takes the nodesize, but in the future it'll take a bool
to indicate if we're extent tree v2. The remaining users are all where
we only have extent_buffer, but we should always have a valid
eb->fs_info in these cases, so add BUG_ON()'s for the !eb->fs_info case
and then convert these callers to use BTRFS_LEAF_DATA_SIZE which takes
the fs_info.
Reviewed-by: Nikolay Borisov <nborisov@suse.com>
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: David Sterba <dsterba@suse.com>
The mkfs_config can hold the BTRFS_LEAF_DATA_SIZE, so calculate this at
config creation time and then use that value throughout convert instead
of calling __BTRFS_LEAF_DATA_SIZE.
Reviewed-by: Nikolay Borisov <nborisov@suse.com>
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: David Sterba <dsterba@suse.com>
This is going to be a different value based on the incompat settings of
the file system, just store this in the fs_info instead of calculating
it every time.
Reviewed-by: Nikolay Borisov <nborisov@suse.com>
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: David Sterba <dsterba@suse.com>
We use __BTRFS_LEAF_DATA_SIZE() in a few places for mkfs. With extent
tree v2 we'll be increasing the size of btrfs_header, so it'll be kind
of annoying to add flags to all callers of __BTRFS_LEAF_DATA_SIZE, so
simply calculate it once and put it in the mkfs_config and use that.
Reviewed-by: Nikolay Borisov <nborisov@suse.com>
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: David Sterba <dsterba@suse.com>
When adding the GC support I noticed we were failing fsck when we had a
directory that hadn't been cleaned up yet because of rm -rf. However
this isn't limited to extent-tree-v2, we'll actually fail in the same
way if we were unable to do the evict portion of the deletion and left
the orphan items around for everybody.
This is a valid file system, it'll be cleaned up properly at mount time,
so fsck shouldn't fail in this case.
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: David Sterba <dsterba@suse.com>
When implementing the GC tree I started getting btrfsck errors when a
test rm -rf <directory> with files inside of it and immediately unmount,
leaving behind orphaned directory items that have GC items for them.
This made me realize that we don't actually handle this case currently
for our normal orphan path. If we fail to clean everything up and leave
behind the orphan items we'll fail fsck.
Fix this by not processing any backrefs we find if we found an inode
item and its nlink is 0. This allows us to pass the test case I've
provided to validate this patch.
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: David Sterba <dsterba@suse.com>
I started hitting a segfault on fuzz test 006 because we couldn't find
the extent root. This is because the global root search stuff expects
the actual key to be setup properly, not just an objectid. Fix this by
initializing the key properly so we can find the extent root and other
trees properly.
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: David Sterba <dsterba@suse.com>
The chunk recover code passes in a buffer it allocates with metadata but
no fs_info, causing fuzz-test 008 to segfault. Fix this test to only
check the flag if we have buf->fs_info set.
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: David Sterba <dsterba@suse.com>
With my global roots prep patches I regressed us on handling the case
where we didn't find a root at all. In this case we need to return an
error (prior we returned -ENOENT) or we need to populate a dummy tree if
we have OPEN_CTREE_PARTIAL set. This fixes a segfault of fuzz test 006.
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: David Sterba <dsterba@suse.com>
We were previously getting away with this because the
load_global_roots() treated ENOENT like everything was a-ok. However
that was a bug and fixing that bug uncovered a problem where we were
unconditionally trying to load the free space tree. Fix that by
skipping the load if we do not have the compat bit set.
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: David Sterba <dsterba@suse.com>
The fuzz-test/003 was infinite looping when I reworked the code to
re-calculate the used bytes for the superblock. This is because fsck
wasn't properly fixing the bad extent before my change, it just happened
to error out nicely, whereas my change made it so we go the wrong bytes
used count and just infinite looped trying to fix the problem.
Fix this by sanity checking the extent when we try to re-calculate the
bytes_used. This makes us no longer infinite loop so we can get through
the fuzz tests.
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: David Sterba <dsterba@suse.com>
003-multi-check-unmounted was taking a long time, this is because it was
doing the 10 second countdown for each iteration of --init-csum-tree.
Fix this by using --force to bypass the countdown.
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: David Sterba <dsterba@suse.com>
While running make test on other patches I noticed we are now
segfaulting on the fuzz tests. This is because when I converted us to a
rb tree for the global roots we lost the ability to catch that there's
no extent root at all. Before we'd populate a dummy
fs_info->extent_root with a not uptodate node, but now you simply don't
get an extent root in the rb_tree. Fix the check_global_roots_uptodate
helper to count how many roots we find and make sure it matches the
number we expect. If it doesn't then we can return -EIO.
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: David Sterba <dsterba@suse.com>
'make clean' can fail in case the _build directory is not present, can
be observed with the build-tests.sh. Use plain 'rm -fd'.
Signed-off-by: David Sterba <dsterba@suse.com>
The fsstress tool is a useful file generator, pull it from fstests as
it's not packaged as a standalone tool anywhere and the LTP version is
out of date.
The file has been modified to build, some xfs-specific ioctls are not
supported.
Signed-off-by: David Sterba <dsterba@suse.com>
This is still work in progress but can survive some stress testing.
There are still some sanity checks missing, do not user this on valuable
data. To enables this, configure must be run with the experimental
features enabled.
$ mkfs.btrfs --csum crc32c /dev/sdx
$ <mount, fill with data, unmount>
$ btrfstune --csum sha256
Will change the checksum to sha256.
Implementation:
- set bit on superblock when the checksums are being changed (similar to
the uuid rewrite)
- metadata checksums are overwritten in place
- data checksums:
- the checksum tree is completely deleted and no checksums are
verified
- data blocks are enumerated and all checksums generated (same as
check --init-csum-tree)
To make it usable, it should be restartable and track the current
progress somehow. Also the previous data checksums should be verified
any time they're available.
Signed-off-by: David Sterba <dsterba@suse.com>
There was a workaround for asciidoc/xmlto build because page btrfs.5 and
btrfs.8 used the same intermediate file. To avoid clash for the section
5 page it is part of the name, this is still needed, but for sphinx we
can use the final name as it will get the right suffix. This affects
only the generated manual pages.
Signed-off-by: David Sterba <dsterba@suse.com>
Update sphinx build so that by default it now builds the manual pages,
in quiet mode. 'make html' builds the html, other sphinx targets are
available, see 'make help'. Installation now works as well.
Note: sphinx is still conditional and must be selected by
ASCIIDOOC_TOOL=sphinx ./configure
Signed-off-by: David Sterba <dsterba@suse.com>
Building 32bit binaries on 64bit hosts is tedious due to all the
required libraries with the right target, also the static versions.
Limit the build only to the native arch.
Signed-off-by: David Sterba <dsterba@suse.com>
There are some configure options not covered by build tests, eg. the
--disable-zoned build broke recently.
Signed-off-by: David Sterba <dsterba@suse.com>
A snapshot could be created in an existing directory, explain the
difference in the command line help options.
Pull-request: #117
Author: Howard <hwj@BridgeportContractor.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Update the remaining erroneous entry of -s in the 'exit status' section
to -c in accordance with the changes made in:
"btrfs-progs: dev stats: update option name for checking non-zero status"
Pull-request: #146
Signed-off-by: Philip Guyton <philip@yewtreeapps.com>
Signed-off-by: David Sterba <dsterba@suse.com>
The zoned support requires a header file and some structures for full
support. There are distros that have only part of that and the
autodetection at configure time does not handle that properly, assuming
the user requested the support.
Check if there was any of the --*able-zoned options and if not detect
the support level and fallback to no zoned eventually, not requiring
users to specify --disable-zoned as before.
Issue: #425
Signed-off-by: David Sterba <dsterba@suse.com>
Since 5.15 enables a runtime feature, this statement is incorrect.
Pull-request: #437
Author: dathide <47128084+dathide@users.noreply.github.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Question forwarded from wiki:
https://btrfs.wiki.kernel.org/index.php?title=Talk:Compression&diff=0&oldid=33545
(info few months old) higher levels: upstream and library has levels 1..22.
mapping to the upstream levels 1 to 15
As implemented in kernel zstd.c, max level is 15 and the level selection
is direct without any translation.
Pull-request: #442
Author: jtagcat <git-12dbd862@jtag.cat>
[ update changelog ]
Signed-off-by: David Sterba <dsterba@suse.com>
In commit 88895a920f ("btrfs-progs: use profile_supported in mkfs as
well") there's a wrapper but not available on non-zoned builds. Add it.
Issue: #445
Signed-off-by: David Sterba <dsterba@suse.com>
Pass BTRFS_BLOCK_GROUP_DATA and BTRFS_BLOCK_GROUP_METADATA to
zoned_profile_supported(), so we can actually distinguish if it is a data
or a meta-data block group.
Fixes: 8f914d518a46 ("btrfs-progs: zoned support DUP on metadata block groups")
Signed-off-by: Johannes Thumshirn <johannes.thumshirn@wdc.com>
Signed-off-by: David Sterba <dsterba@suse.com>
For !extent tree v2 we should validate the key.offset == 0, and for
extent tree v2 we should validate that key.offset < nr_global_roots. If
this fails we need to fail to load the global root so that the
appropriate action is taken.
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: David Sterba <dsterba@suse.com>
The kernel uses 'unsigned long' for u64 specifically for ppc64 and
mips64.
Remove asm/types.h include as it will get included properly later.
Fixe -Wformat warnings.
Signed-off-by: Rosen Penev <rosenp@gmail.com>
Signed-off-by: David Sterba <dsterba@suse.com>
The image has a key in extent tree, (30457856 METADATA_ITEM 256), which
has invalid level (256 > BTRFS_MAX_LEVEL).
Make sure check can at least detect such problem.
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
[BUG]
There is a bug report that kernel tree-checker rejected an invalid
metadata item:
corrupt leaf: block=934474399744 slot=68 extent bytenr=425173254144 len=16384 invalid tree level, have 33554432 expect [0, 7]
But original mode btrfs-check reports nothing wrong.
(lowmem mode will just crash, and fixed in previous patch).
[CAUSE]
For original mode it doesn't really check tree level, thus didn't find
the problem.
[FIX]
I don't have a good idea to completely make original mode to verify the
level in backref and in the tree block (while lowmem does that).
But at least we can detect obviously corrupted level just like kernel.
Now original mode will detect such problem:
...
[2/7] checking extents
ERROR: tree block 30457856 has bad backref level, has 256 expect [0, 7]
ref mismatch on [30457856 16384] extent item 0, found 1
tree backref 30457856 root 5 not found in extent tree
backpointer mismatch on [30457856 16384]
ERROR: errors found in extent allocation tree or chunk allocation
[3/7] checking free space tree
...
Reported-by: Stickstoff <stickstoff@posteo.de>
Link: https://lore.kernel.org/linux-btrfs/6ed4cd5a-7430-f326-4056-25ae7eb44416@posteo.de/
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
[BUG]
When running lowmem mode with METADATA_ITEM which has invalid level, it
will crash with the following backtrace:
(gdb) bt
#0 0x0000555555616b0b in btrfs_header_bytenr (eb=0x4)
at ./kernel-shared/ctree.h:2134
#1 0x0000555555620c78 in check_tree_block_backref (root_id=5,
bytenr=30457856, level=256) at check/mode-lowmem.c:3818
#2 0x0000555555621f6c in check_extent_item (path=0x7fffffffd9c0)
at check/mode-lowmem.c:4334
#3 0x00005555556235a5 in check_leaf_items (root=0x555555688e10,
path=0x7fffffffd9c0, nrefs=0x7fffffffda30, account_bytes=1)
at check/mode-lowmem.c:4835
#4 0x0000555555623c6d in walk_down_tree (root=0x555555688e10,
path=0x7fffffffd9c0, level=0x7fffffffd984, nrefs=0x7fffffffda30,
check_all=1) at check/mode-lowmem.c:4967
#5 0x000055555562494f in check_btrfs_root (root=0x555555688e10, check_all=1)
at check/mode-lowmem.c:5266
#6 0x00005555556254ee in check_chunks_and_extents_lowmem ()
at check/mode-lowmem.c:5556
#7 0x00005555555f0b82 in do_check_chunks_and_extents () at check/main.c:9114
#8 0x00005555555f50ea in cmd_check (cmd=0x55555567c640 <cmd_struct_check>,
argc=3, argv=0x7fffffffdec0) at check/main.c:10892
#9 0x000055555556b2b1 in cmd_execute (argv=0x7fffffffdec0, argc=3,
cmd=0x55555567c640 <cmd_struct_check>) at cmds/commands.h:125
[CAUSE]
For function check_extent_item() it will go through inline extent items
and then check their backrefs.
But for METADATA_ITEM, it doesn't really validate key.offset, which is
u64 and can contain value way larger than BTRFS_MAX_LEVEL (mostly caused
by bit flip).
In that case, if we have a larger value like 256 in key.offset, then
later check_tree_block_backref() will use 256 as level, and overflow
path->nodes[level] and crash.
[FIX]
Just verify the level, no matter if it's from btrfs_tree_block_level()
(which is just u8), or it's from key.offset (which is u64).
To do the check properly and detect higher bits corruption, also change
the type of @level from u8 to u64.
Now lowmem mode can detect the problem properly:
...
[2/7] checking extents
ERROR: tree block 30457856 has bad backref level, has 256 expect [0, 7]
ERROR: extent[30457856 16384] level mismatch, wanted: 0, have: 256
ERROR: errors found in extent allocation tree or chunk allocation
[3/7] checking free space tree
...
Reviewed-by: Su Yue <l@damenly.su>
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
The search for default subvolume could fail for two reasons, the lack of
CAP_SYS_ADMIN for TREE_SEARCH ioctl is one but the default subvolume
could be unset as well, thus no restrictions for deletion.
Signed-off-by: David Sterba <dsterba@suse.com>
Checking the default subvolume uses TREE_SEARCH which is a CAP_SYS_ADMIN
only operation, and thus will fail when unprivileged, even if we have
permissions to actually delete the subvolume.
This produces a warning even if all is ok. Let's hide it if we're not
root (root but !CAP is odd enough to warn).
Fixes 87804a3f06 ("btrfs-progs: subvolume: check deleting default subvolume")
Link: https://bugs.debian.org/998840
Signed-off-by: Adam Borowski <kilobyte@angband.pl>
Signed-off-by: David Sterba <dsterba@suse.com>
