Use the new docref and duplabel syntax to fix build warnings and fix the
link targets in the pages.
[ci skip]
Signed-off-by: David Sterba <dsterba@suse.com>
Sphinx/RST does not(?) have native support for cross references to
labels in specific documents (doc, ref but not both at the same time),
also allowing duplicate labels. We have web and manual pages and to
share the same text there are common files included from each, defining
labels. This leads to warnings and clicking the links ends up in the
included document (like ch-volume-management-intro.rst) instead of the
parent.
As a last resort, implement own role that does the doc and ref in one
go. A special directive is used to define a label that is expected
to be duplicate (but otherwise it's an ordinary label) and this gets rid
of the warning too.
Signed-off-by: David Sterba <dsterba@suse.com>
Subpage promoted to 'OK', no serious problems since 6.0. Otherwise
references, adjustments, wording updates.
[ci skip]
Signed-off-by: David Sterba <dsterba@suse.com>
We've deprecated the -R option and runtime feature distinction, there's
only one option -O recommended so let it also print on the same line.
Incompat/compat/runtime status of a feature shall be consulted with the
documentation.
Signed-off-by: David Sterba <dsterba@suse.com>
Be verbose about the potential compatibility problems with the
sectorsize and page size. Also print the page size on the overview.
Signed-off-by: David Sterba <dsterba@suse.com>
There are new backends and builtin accelerated implementations.
Recalculate the table results, a different host than before, with SHANI
extension.
[ci skip]
Signed-off-by: David Sterba <dsterba@suse.com>
[BUG]
Although commit b2a1be83b8 ("btrfs-progs: mkfs: keep file descriptors
open during whole time") is making sure we're only closing the writeable
fds after the fs is properly created, there is still a missing fd not
following the requirement.
And this explains the issue why sometimes after mkfs.btrfs, lsblk still
doesn't give a valid uuid.
Shown by the strace output (the command is "mkfs.btrfs -f
/dev/test/scratch1"):
openat(AT_FDCWD, "/dev/test/scratch1", O_RDWR) = 5 <<< Writeable open
fadvise64(5, 0, 0, POSIX_FADV_DONTNEED) = 0
sysinfo({uptime=2529, loads=[8704, 6272, 2496], totalram=4104548352, freeram=3376611328, sharedram=9211904, bufferram=43016192, totalswap=3221221376, freeswap=3221221376, procs=190, totalhigh=0, freehigh=0, mem_unit=1}) = 0
lseek(5, 0, SEEK_END) = 10737418240
lseek(5, 0, SEEK_SET) = 0
......
close(5) = 0 <<< Closed now
pwrite64(6, "O\250\22\261\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 16384, 1163264) = 16384
pwrite64(6, "\201\316\272\342\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 16384, 1179648) = 16384
pwrite64(6, "K}S\t\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 16384, 1196032) = 16384
pwrite64(6, "\207j$\265\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 16384, 1212416) = 16384
pwrite64(6, "q\267;\336\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 16384, 5242880) = 16384
fsync(6) <<< But we're still writing into the disk.
[CAUSE]
After more digging, it turns out we have a very obvious escape in
open_ctree_fs_info():
open_ctree_fs_info()
|- fp = open(oca->filename, flags);
|- info = __open_ctree_fd();
|- close(fp);
As later we only do IO using the device fd, this close() seems fine.
But the truth is, for mkfs usage, this fs_info is a temporary one, with
a special magic number for the disk. And since mkfs is doing writeable
operations, this close() would immediately trigger udev scan.
And since at this stage, the fs is not yet fully created, udev can race
with mkfs, and may get the invalid temporary superblock.
[FIX]
Introduce a new btrfs_fs_info member, initial_fd, for
open_ctree_fs_info() to record the fd.
And on close_ctree(), if we find fs_info::initial_fd is a valid fd, then
close it.
By this, we make sure all writeable fds are only closed after we have
written valid super blocks into the disk.
Issue: #734
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
[BUG]
In "btrfs tune" subcommand, we're using open_ctree_fd(), which
requires passing a valid fd, and the caller is responsible to properly
handling the lifespan of the fd.
But we just call close_ctree() and btrfs_close_all_devices(), without
closing the fd.
[FIX]
Just manually close the fd.
Reviewed-by: Anand Jain <anand.jain@oracle.com>
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
[BUG]
In cmd_rescue_clear_ino_cache(), we opened the fs, but without
closing it using close_ctree().
[CAUSE]
This was introduced in 42404a4e44 ("btrfs-progs: move inode cache
removal to rescue group"), the original code inside btrfs check
had a "goto out_close;" to properly close the fs.
[FIX]
Manually call close_ctree() on the fs_info->tree_root.
Reviewed-by: Anand Jain <anand.jain@oracle.com>
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
The symbol BTRFS_UPDATE_KERNEL seems to be unused since 2f55fd7019
("btrfs-progs: optimize btrfs_scan_lblkid() for multiple calls"), remove
it.
Signed-off-by: David Sterba <dsterba@suse.com>
Wherever it makes sense, save the logs as artifacts when something
fails (most likely the main step with real tests).
Signed-off-by: David Sterba <dsterba@suse.com>
There's a report that passing raw device mapper path and -d don't work
together:
yyy@xxx ~ $ sudo btrfs filesystem show /dev/dm-0
Label: none uuid: a7fbb8d6-ec5d-4e88-bd8b-c686553e0dc7
Total devices 1 FS bytes used 144.00KiB
devid 1 size 256.00MiB used 88.00MiB path /dev/mapper/da0972636816-LogVol00
With --all-devices
yyy@xxx ~ $ sudo btrfs filesystem show --all-devices /dev/dm-0
ERROR: not a valid btrfs filesystem: /dev/dm-0
Where dm-0 corresponds to the LogVol00 device from above.
Passing the option -d skips some steps but still uses the real path of
the device that is required for scanning and identification, while
blkid uses the canonicalized path.
The combination of raw device name and -d was not handled as the raw
path is not in cache and thus not recognized. Canonicalization fixes
that although this changes the device name in the output.
Issue: #732
Signed-off-by: David Sterba <dsterba@suse.com>
There are now two tests with 060, we'd like to be that a unique number,
rename the one that was added more recently.
Signed-off-by: David Sterba <dsterba@suse.com>
The raid-stripe-tree can be enabled for convert, though it's still
considered incomplete and slightly experimental. Due to that the tests
need to be adjusted to check for support and skip mount eventually.
Possible remaining options to add: quota, squota
Issue: #694
Signed-off-by: David Sterba <dsterba@suse.com>
Now docker hub images can be pulled for build tests (sources are
downloaded) and this is faster than rebuilding them each time so this
can be enabled for all ci/* and devel branches.
Signed-off-by: David Sterba <dsterba@suse.com>
The runner scripts ci/ci-build... pass build options that work for each
distro, however this was not matching some Dockerfiles. Pulling such
image would then fail due to missing dependencies (namely libudev).
Signed-off-by: David Sterba <dsterba@suse.com>
There was a bug when a branch contained a slash then the file with
downloaded sources was not found. Update all, all images have to be
rebuilt and pushed to docker hub so the changes are applied inside
github actions.
Signed-off-by: David Sterba <dsterba@suse.com>
Add a possibility to test branches independent of devel, the pattern is
prefix "ci/" or "CI/".
[ci skip]
Signed-off-by: David Sterba <dsterba@suse.com>
We have had working subpage support in Btrfs for many cycles now.
Generally, we do not want people creating filesystems by default
with non-4k sectorsizes since it creates portability problems.
As the subpage has stabilized it seems to be safe to do the switch.
This may still affect users that relying on the previous behaviour.
Issue: #604
Reviewed-by: Anand Jain <anand.jain@oracle.com>
Reviewed-by: Qu Wenruo <wqu@suse.com>
Reviewed-by: Josef Bacik <josef@toxicpanda.com>
Reviewed-by: Eric Curtin <ecurtin@redhat.com>
Signed-off-by: Neal Gompa <neal@gompa.dev>
Signed-off-by: David Sterba <dsterba@suse.com>
In case of a raid5/6 filesystem 'btrfs fi us' returns wrong values
without the root capabilities:
$ sudo btrfs fi us /tmp/raid5fs # as root
Overall:
Device size: 3.00GiB
Device allocated: 1.51GiB <--- OK
Device unallocated: 1.49GiB <--- OK
Device missing: 0.00B
Device slack: 0.00B
Used: 769.03MiB <--- OK
Free (estimated): 1.32GiB (min: 1.32GiB) <-OK
Free (statfs, df): 1.32GiB
Data ratio: 1.50 <--- OK
Metadata ratio: 1.50 <--- OK
Global reserve: 5.50MiB (used: 0.00B)
Multiple profiles: no
[...]
$ btrfs fi us /tmp/raid5fs # as user
WARNING: cannot read detailed chunk info, per-device usage will not be shown, run as root
Overall:
Device size: 3.00GiB
Device allocated: 0.00B <--- WRONG
Device unallocated: 3.00GiB <--- WRONG
Device missing: 0.00B
Device slack: 0.00B
Used: 0.00B <--- WRONG
Free (estimated): 0.00B (min: 8.00EiB) <- WRONG
Free (statfs, df): 1.32GiB
Data ratio: 0.00 <--- WRONG
Metadata ratio: 0.00 <--- WRONG
Global reserve: 5.50MiB (used: 0.00B)
Multiple profiles: no
[...]
The reason is that the BTRFS_IOC_SPACE_INFO ioctl doesn't return enough
information. To bypass it a scan of the chunks is required when a
raid5/6 profile is present.
To avoid providing wrong information, in case of a raid5/6 filesystem
without the root capabilities the "btrfs fi us" is not executed at all
and a warning with a suggestion to run it as root is printed.
$ ./btrfs fi us /tmp/t/
WARNING: cannot read detailed chunk info, per-device usage will not be shown, run as root
WARNING: due to the presence of a raid5/raid6 profile, we cannots compute some values;
WARNING: run as root instead.
Signed-off-by: Goffredo Baroncelli <kreijack@libero.it>
Signed-off-by: David Sterba <dsterba@suse.com>
If "btrfs dev us" is invoked by a not root user, it is impossible to
collect the chunk info data (not enough privileges). This causes
"btrfs dev us" to print as "Unallocated" value the size of the disk.
This patch handles the case where print_device_chunks() is invoked
without the chunk info data, printing "Unallocated N/A":
Before the patch:
$ btrfs dev us t/
WARNING: cannot read detailed chunk info, per-device usage will not be shown, run as root
/dev/loop0, ID: 1
Device size: 5.00GiB
Device slack: 0.00B
Unallocated: 5.00GiB <-- Wrong
$ sudo btrfs dev us t/
[sudo] password for ghigo:
/dev/loop0, ID: 1
Device size: 5.00GiB
Device slack: 0.00B
Data,single: 8.00MiB
Metadata,DUP: 512.00MiB
System,DUP: 16.00MiB
Unallocated: 4.48GiB <-- Correct
After the patch:
$ ./btrfs dev us /tmp/t/
WARNING: cannot read detailed chunk info, per-device usage will not be shown, run as root
/dev/loop0, ID: 1
Device size: 5.00GiB
Device slack: 0.00B
Unallocated: N/A
$ sudo ./btrfs dev us /tmp/t/
[sudo] password for ghigo:
/dev/loop0, ID: 1
Device size: 5.00GiB
Device slack: 0.00B
Data,single: 8.00MiB
Metadata,DUP: 512.00MiB
System,DUP: 16.00MiB
Unallocated: 4.48GiB
Reviewed-by: Anand Jain <anand.jain@oracle.com>
Signed-off-by: Goffredo Baroncelli <kreijack@libero.it>
Signed-off-by: David Sterba <dsterba@suse.com>
This patch introduces a new parser helper, parse_u64_with_suffix(),
which has a better error handling, following all the parse_*()
helpers to return non-zero value for errors.
This new helper is going to replace parse_size_from_string(), which
would directly call exit(1) to stop the whole program.
Furthermore most callers of parse_size_from_string() are expecting
exit(1) for error, so that they can skip the error handling.
For those call sites, introduce a wrapper, arg_strtou64_with_suffix(),
to do that. The only disadvantage is a little less detailed error
report for why the parse failed, but for most cases the generic error
string should be enough.
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Both functions are just doing the same thing, the only difference is
only in error handling, as parse_u64() requires callers to handle it,
meanwhile arg_strtou64() would call exit(1).
This patch would convert arg_strtou64() to utilize parse_u64(), and use
the return value to output different error messages.
This also means the return value of parse_u64() would be more than just
0 or 1, but -EINVAL for invalid string (including no numeric string at
all, has any tailing characters, or minus value), and -ERANGE for
overflow.
The existing callers are only checking if the return value is 0, thus
not really affected.
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
The CI test uses openssl 3.2 but it still hasn't made it to Tumbleweed,
it's still waiting in the queue. It'll be enabled again.
Signed-off-by: David Sterba <dsterba@suse.com>
Enhance the conversion macro to recognize the filesystem type in case
the ntfs-specific converter utility has to be used. Its current feature
set does not match btrfs-convert.
Signed-off-by: David Sterba <dsterba@suse.com>
The convert tests weren't enabled in the CI due to some problems that
seem to be fixed now. Add it to the default and coverage workflows, the
run time is about 2 minutes which is acceptable for coverage and for
devel it's running in parallel.
Signed-off-by: David Sterba <dsterba@suse.com>
The test does not pass in environments where /bin/sh is not bash (like
github CI). Unify the initial setup like the other tests do: source,
compatibility, prerequisities, the rest.
Signed-off-by: David Sterba <dsterba@suse.com>
[BUG]
There is a report about failed btrfs-convert, which shows the following
error:
Create btrfs metadata
corrupt leaf: root=5 block=5001931145216 slot=1 ino=89911763, invalid previous key objectid, have 89911762 expect 89911763
leaf 5001931145216 items 336 free space 7 generation 90 owner FS_TREE
leaf 5001931145216 flags 0x1(WRITTEN) backref revision 1
fs uuid 8b69f018-37c3-4b30-b859-42ccfcbe2449
chunk uuid 448ce78c-ea41-49f6-99dc-46ad80b93da9
item 0 key (89911762 INODE_REF 3858733) itemoff 16222 itemsize 61
index 171 namelen 51 name: [FILENAME1]
item 1 key (89911763 INODE_REF 3858733) itemoff 16161 itemsize 61
index 103 namelen 51 name: [FILENAME2]
[CAUSE]
When iterating a directory, btrfs-convert would insert the DIR_ITEMs,
along with the INODE_REF of that inode.
This leads to above stray INODE_REFs, and trigger the tree-checker.
This can only happen for large fs, as for most cases we have all these
modified tree blocks cached, thus tree-checker won't be triggered.
But when the tree block cache is not hit, and we have to read from disk,
then such behavior can lead to above tree-checker error.
[FIX]
Insert a dummy INODE_ITEM for the INODE_REF first, the inode items would
be updated when iterating the child inode of the directory.
Issue: #731
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
The strict check is enabled for both check_image() and
convert_test_do_convert() to detect chunk alignment related problems.
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Recently we had a scrub use-after-free caused by unaligned chunk
length, although the fix was submitted, we may want to do extra checks
for a chunk's alignment.
This patch adds such check for the starting bytenr and length of a
chunk, to make sure they are properly aligned to 64K stripe boundary.
By default, the check only leads to a warning but is not treated as an
error, as we expect kernel to handle such unalignment without any
problem.
But if the new debug environmental variable,
BTRFS_PROGS_DEBUG_STRICT_CHUNK_ALIGNMENT, is specified, then we will
treat it as an error. So that we can detect unexpected chunks from
btrfs-progs, and fix them before reaching the end users.
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Although kernel scrub code has been updated to handle the unaligned
chunk length, there is also no harm if we can allocate data chunk with
both start and length aligned.
This patch handles this by rounding up the end bytenr when allocating
data chunks for the conversion.
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
On multi-device filesystems, a scrub limit may be applied to any of the
devices. Ensure that any limit found is not disregarded.
Since it's more intuitive, keep the lowest non-zero limit found, even
though at the present we don't actually use the exact value.
Pull-request: #733
Issue: #727
Fixes: 7e4a235df1 ("btrfs-progs: scrub status: print device speed limit in status if set")
Signed-off-by: Jonas Malaco <jonas@protocubo.io>
Signed-off-by: David Sterba <dsterba@suse.com>
On multi-device filesystems, a scrub limit may be applied to any of the
devices. Ensure that any limit found is not disregarded.
Since it's more intuitive, keep the lowest non-zero limit found, even
though at the present we don't actually use the exact value.
Pull-request: #733
Issue: #727
Fixes: 7e4a235df1 ("btrfs-progs: scrub status: print device speed limit in status if set")
Signed-off-by: Jonas Malaco <jonas@protocubo.io>
Signed-off-by: David Sterba <dsterba@suse.com>
On multi-device filesystems, scrub status should report "some limits
set" if at least one device has a scrub limit set.
However, with btrfs-progs 6.6.3, this was being reported regardless of
whether any limit actually being set:
# sudo btrfs scrub limit /more/butter
UUID: 989129d9-c96f-4d52-9d68-cbb6d9b2c499
Id Limit Path
-- ----- ---------
1 - /dev/sdc1
2 - /dev/sdd1
# sudo btrfs scrub status /more/butter/
UUID: 989129d9-c96f-4d52-9d68-cbb6d9b2c499
Scrub started: Mon Jan 15 02:00:30 2024
Status: running
Duration: 6:23:19
Time left: 0:49:08
ETA: Mon Jan 15 09:12:57 2024
Total to scrub: 9.83TiB
Bytes scrubbed: 8.72TiB (88.64%)
Rate: 397.47MiB/s (some device limits set)
Error summary: no errors found
Fix it by only setting `limit` to the special marker value 1 if at least
one actual limit is found.
Pull-request: #733
Issue: #727
Fixes: 7e4a235df1 ("btrfs-progs: scrub status: print device speed limit in status if set")
Signed-off-by: Jonas Malaco <jonas@protocubo.io>
Signed-off-by: David Sterba <dsterba@suse.com>
[BUG]
There is a report about failed btrfs-convert, which shows the following
error:
corrupt leaf: root=5 block=5001928998912 slot=1 ino=89911763, invalid previous key objectid, have 89911762 expect 89911763
ERROR: failed to copy ext2 inode 89911320: -5
ERROR: error during copy_inodes -5
WARNING: error during conversion, the original filesystem is not modified
[CAUSE]
Above error is triggered when checking the following items inside a
subvolume:
- inode ref
- dir item/index
- file extent
- xattr
This is to make sure these items have correct previous key.
However btrfs-convert is not following this requirement, it always
inserts those items first, then creates a btrfs_inode for it.
Thus it can lead to the error.
This can only happen for large fs, as for most cases we have all these
modified tree blocks cached, thus tree-checker won't be triggered.
But when the tree block cache is not hit, and we have to read from disk,
then such behavior can lead to above tree-checker error.
[FIX]
Make sure we insert the inode item first, then the file extents/dir
items/xattrs. And after the file extents/dir items/xattrs inserted, we
update the existing inode (to update its size and bytes).
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Unlike kernel where tree-checker would provide enough info so later we
can use "btrfs inspect dump-tree" to catch the offending tree block, in
progs we may not even have a btrfs to start "btrfs inspect dump-tree".
E.g during btrfs-convert.
To make later debuging easier, let's call btrfs_print_tree() for every
error we hit inside tree-checker.
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
