Commit graph

6735 commits

Author SHA1 Message Date
Josef Bacik ba01aadff1 btrfs-progs: take a ref in the root locking code
This code in the kernel not only returns the locked root, but also takes
a reference on the node.  This is important for when we sync ctree.c
into btrfs-progs, it expects that references are held on the root node
after calling these helpers.

Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2023-08-28 17:24:25 +02:00
David Sterba ce93d20fd5 btrfs-progs: docs: update kernel changes
Signed-off-by: David Sterba <dsterba@suse.com>
2023-08-28 17:24:25 +02:00
David Sterba 8001e37409 btrfs-progs: docs: updates
- reformatting
- new documents
- enhancements
- status updates

Signed-off-by: David Sterba <dsterba@suse.com>
2023-08-28 17:24:25 +02:00
David Sterba a81a3d771b btrfs-progs: image: factor out the restore part from main.c
The remaining part of restore functionality starting from
restore_metadump() has been factored out. Same incremental set of
changes so the diff is not clean.

Signed-off-by: David Sterba <dsterba@suse.com>
2023-08-28 17:24:25 +02:00
David Sterba add18714b8 btrfs-progs: image: factor out the create part from main.c
The functionality of create and restore is all in main.c, split the
create functionality first. This is not the cleaniest diff to do it, the
functions are entangled and the final result is from several compile and
edit cycles.

Signed-off-by: David Sterba <dsterba@suse.com>
2023-08-28 17:24:24 +02:00
David Sterba 00df7245f4 btrfs-progs: image: move defintions from main.c to header
Signed-off-by: David Sterba <dsterba@suse.com>
2023-08-28 17:24:24 +02:00
David Sterba 55bed0a8db btrfs-progs: ci: add cleanup script for coverage tests
Signed-off-by: David Sterba <dsterba@suse.com>
2023-08-28 17:24:24 +02:00
David Sterba 5b9206727b btrfs-progs: rename and move get_device_info
The helper belongs to the device-utils, move it there and use the common
prefix.

Signed-off-by: David Sterba <dsterba@suse.com>
2023-08-28 17:24:24 +02:00
David Sterba 588f82f333 btrfs-progs: hash-vectest: make test_hash static to fix build warning
The warning:

crypto/hash-vectest.c:559:5: warning: no previous prototype for ‘test_hash’ [-Wmissing-prototypes]
  559 | int test_hash(const struct hash_testspec *sp

Signed-off-by: David Sterba <dsterba@suse.com>
2023-08-28 17:24:24 +02:00
David Sterba 83ac6e0a72 btrfs-progs: crypto: make the PCL implementation default for crc32c
Drop the old native intel implementation and use the PCL one. Remove the
artifical CPU flags.

Signed-off-by: David Sterba <dsterba@suse.com>
2023-08-28 17:24:24 +02:00
David Sterba 992be8b50a btrfs-progs: crypto: add PCL based implementation for crc32c
Copy faster implementation of crc32c from linux kernel as of 6.5-rc7
(x86_64, arch/x86/crypto/crc32c-pcl-intel-asm_64.S). This needs
assembler build support, so detect target architecture so
cross-compilation still works.

Add a special CPU flag so the old and new implementations can be
benchmarked and verified separately.

Sample benchmark:

CPU flags: 0x1ff
CPU features: SSE2 SSSE3 SSE41 SSE42 SHA AVX AVX2 CRC32C_PCL
Block size:     4096
Iterations:     1000000
Implementation: builtin
Units:          CPU cycles

      NULL-NOP: cycles:     77177218, cycles/i       77
   NULL-MEMCPY: cycles:    226313072, cycles/i      226,    62133.395 MiB/s
    CRC32C-ref: cycles:  24418596066, cycles/i    24418,      575.859 MiB/s
     CRC32C-NI: cycles:   1188335920, cycles/i     1188,    11833.073 MiB/s
    CRC32C-PCL: cycles:    463193456, cycles/i      463,    30358.037 MiB/s
        XXHASH: cycles:    851606646, cycles/i      851,    16511.916 MiB/s
    SHA256-ref: cycles:  74476234956, cycles/i    74476,      188.808 MiB/s
     SHA256-NI: cycles:  34198637428, cycles/i    34198,      411.177 MiB/s
    BLAKE2-ref: cycles:  14761411664, cycles/i    14761,      952.597 MiB/s
   BLAKE2-SSE2: cycles:  18101896796, cycles/i    18101,      776.807 MiB/s
  BLAKE2-SSE41: cycles:  12599091062, cycles/i    12599,     1116.087 MiB/s
   BLAKE2-AVX2: cycles:   9668247506, cycles/i     9668,     1454.418 MiB/s

The new implementation is about 2.5x faster.

Note: there new version does not work on musl because of linkage
problems (relocations in .rodata), so it's still using the old
implementation.

Signed-off-by: David Sterba <dsterba@suse.com>
2023-08-28 17:24:24 +02:00
David Sterba 8461513b9a btrfs-progs: rename prefixcmp to string_has_prefix
We already have the string_* namespace for such helpers.

Signed-off-by: David Sterba <dsterba@suse.com>
2023-08-28 17:24:24 +02:00
David Sterba 5aa2ea7db0 btrfs-progs: add more helpers to read values from sysfs
In some places we want to read a single u64 value from a sysfs path, or
from fsid directory. Add helpers that do that in one go.

Signed-off-by: David Sterba <dsterba@suse.com>
2023-08-28 17:24:24 +02:00
David Sterba 4a96a935ad btrfs-progs: move sysfs related helpers to own file
The sysfs could use more convenience helpers so move the current code to
own file before adding more helpers.

Signed-off-by: David Sterba <dsterba@suse.com>
2023-08-28 17:24:24 +02:00
Qu Wenruo 6abbad4d1b btrfs-progs: docs: enhance the error handling guidelines
Currently we only have a very brief explanation on the unexpected error
handling (only ASSERT()/WARN_ON()/BUG_ON()), and no further
recommendation on the proper usage of them.

This patch would improve the guideline by:

- Add btrfs_abort_transaction() usage
  Which is the recommended way when possible.

- More detailed explanation on the usage of ASSERT()
  Which is only a fail-fast option mostly designed for developers, thus
  is only recommended to rule out some invalid function usage.

- More detailed explanation on the usage of WARN_ON()
  Mostly for call sites which need a call trace strongly, and is not
  applicable for a btrfs_abort_transaction() call.

- Completely discourage the usage of BUG_ON()

Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2023-08-28 17:24:24 +02:00
David Sterba 85c841f961 btrfs-progs: docs: add developer docs for internal APIs
Signed-off-by: David Sterba <dsterba@suse.com>
2023-08-28 17:24:24 +02:00
David Sterba 00e401752f btrfs-progs: list-chunks: port to sorting API
Use the sorting API. This is 1:1 transformation of previous single key
sorting and needs to be updated so there are multiple accepted instead.

Signed-off-by: David Sterba <dsterba@suse.com>
2023-08-28 17:24:24 +02:00
David Sterba 7f9bcab309 btrfs-progs: add sorting API
Signed-off-by: David Sterba <dsterba@suse.com>
2023-08-28 17:24:24 +02:00
David Sterba a5d95a73f9 btrfs-progs: add an extensible pointer array struct
API for extensible array of pointers for covenience. A simple wrapper
around a (void *) array with length.

Signed-off-by: David Sterba <dsterba@suse.com>
2023-08-28 17:24:24 +02:00
David Sterba 2b35741a39 btrfs-progs: fix more typos found by codespell
Signed-off-by: David Sterba <dsterba@suse.com>
2023-08-28 17:24:24 +02:00
Francesco Yoshi Gobbo 240f518441 btrfs-progs: balance start: fix typo in help text
Pull-request: #660
Author: Francesco Yoshi Gobbo <yoshi@fgobbo.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2023-08-28 17:24:24 +02:00
David Sterba 7ed0172bf6 btrfs-progs: docs: add dev docs about json
Programmer documentation, to be continued.

Signed-off-by: David Sterba <dsterba@suse.com>
2023-08-28 17:24:24 +02:00
David Sterba ebd046eb8d btrfs-progs: tests: renumber json test cases so they match the parameter
The test utility takes a numeric parameter from 1 to max tests but this
is off by one to the test case function names. Unify that so it's clear
which test fails.

Signed-off-by: David Sterba <dsterba@suse.com>
2023-08-28 17:24:24 +02:00
David Sterba ede538f98d btrfs-progs: print null value for empty uuid in json
This is a potentially breaking change to json output. An all zeros uuid
was printed as "-" but we can utilize native json type null for that.
Note the va_copy must be used as va_arg advances the pointer.

{
	"nulluuid": null
}

Signed-off-by: David Sterba <dsterba@suse.com>
2023-08-28 17:24:23 +02:00
David Sterba 7ed79f2b3b btrfs-progs: use escaped format for subvolume path strings in json
Signed-off-by: David Sterba <dsterba@suse.com>
2023-08-28 17:24:23 +02:00
David Sterba 70ae158ec8 btrfs-progs: rename time-long format name to date-time
Make the timestamp format more descriptive what is actually printed. We
may need separate date or time in the future.

Signed-off-by: David Sterba <dsterba@suse.com>
2023-08-28 17:24:23 +02:00
David Sterba 386765be15 btrfs-progs: tests: fix build warnings in json tests (-Wmissing-prototypes)
The test cases should be static, otherwise it leads to a warning like

    [LD]     json-formatter-test
tests/json-formatter-test.c:40:6: warning: no previous prototype for ‘test_simple_empty’ [-Wmissing-prototypes]
   40 | void test_simple_empty()
      |      ^~~~~~~~~~~~~~~~~

Signed-off-by: David Sterba <dsterba@suse.com>
2023-08-28 17:24:23 +02:00
David Sterba 25fc470252 btrfs-progs: print numeric values unquoted in json
The json spec allows numeric values and it's recommended to use them
instead of the stringified numbers. This is a potentially breaking change
if some tools relied on the string value.

As most formats we now have are '%llu' and it's convenient to just pass
it to vprintf, don't add a special type for ints. Any new int type must
be added to the list.

{
	"number": 1234
}

Signed-off-by: David Sterba <dsterba@suse.com>
2023-08-28 17:24:23 +02:00
David Sterba 155eb56417 btrfs-progs: document json format str
The 'str' type was added in ecbb6a7fcd ("btrfs-progs: add json
formatter for escaped string") but not documented. It should be used
e.g. for paths or strings from unknown origin.

Signed-off-by: David Sterba <dsterba@suse.com>
2023-08-28 17:24:23 +02:00
David Sterba be5cb7dfe9 btrfs-progs: add bool type to json
Signed-off-by: David Sterba <dsterba@suse.com>
2023-08-28 17:24:23 +02:00
David Sterba b890183072 btrfs-progs: support unquoted values in json
For null or boolean values the "..." quoting must not be done, add
support for that. This is detected internally for each printed value.

Signed-off-by: David Sterba <dsterba@suse.com>
2023-08-28 17:24:23 +02:00
David Sterba 29060ec176 btrfs-progs: subvolume show: print all items unconditionally in json
The json output could be easily filtered on the user side so we'll print
everything unconditionally.

Signed-off-by: David Sterba <dsterba@suse.com>
2023-08-28 17:24:23 +02:00
David Sterba 42d0861e31 btrfs-progs: subvolume: rename some json key names
Don't abbreviate generation and use qgroup where it's related to the
qgroup itself and not quotas in general.

Signed-off-by: David Sterba <dsterba@suse.com>
2023-08-28 17:24:23 +02:00
David Sterba 6120fdadc2 btrfs-progs: subvolume: move json format output under experimental
The format of the json data needs some time to be finalized so it'll
be under experimental.

Signed-off-by: David Sterba <dsterba@suse.com>
2023-08-28 17:24:23 +02:00
Christoph Heiss 328333f9cf btrfs-progs: subvol show: implement json format output
Implements JSON-formatted output for the `subvolume list` command using
the `--format json` global option, much like it is implemented for other
commands.

Signed-off-by: Christoph Heiss <christoph@c8h4.io>
Signed-off-by: David Sterba <dsterba@suse.com>
2023-08-28 17:24:23 +02:00
Christoph Heiss f134c86411 btrfs-progs: subvol get-default: implement json format output
Implements JSON-formatted output for the `subvolume get-default` command
using the `--format json` global option, much like it is implemented for
other commands.

Signed-off-by: Christoph Heiss <christoph@c8h4.io>
Signed-off-by: David Sterba <dsterba@suse.com>
2023-08-28 17:24:23 +02:00
Christoph Heiss ee233db769 btrfs-progs: subvol list: implement json format output
Implements JSON-formatted output for the `subvolume list` command using
the `--format json` global option, much like it is implemented for other
commands.

Re-uses the `btrfs_list_layout` infrastructure to nicely fit it into the
existing formatting code.

A notable difference to the normal, text-based output is that in the
JSON output, timestamps include the timezone offset as well.

Signed-off-by: Christoph Heiss <christoph@c8h4.io>
Signed-off-by: David Sterba <dsterba@suse.com>
2023-08-28 17:24:23 +02:00
Christoph Heiss da440e66fb btrfs-progs: subvol: introduce rowspec definition for json output
List and export all fields that may be needed for any subvolume related
json output.

Signed-off-by: Christoph Heiss <christoph@c8h4.io>
Signed-off-by: David Sterba <dsterba@suse.com>
2023-08-28 17:24:23 +02:00
Christoph Heiss 7c983d2352 btrfs-progs: subvol show: factor out text printing to own function
Prepare for switching the plain and json output. The format is slightly
different so we can't utilize the unified fmt_* helpers and two separate
printer functions make more sense.

Signed-off-by: Christoph Heiss <christoph@c8h4.io>
Signed-off-by: David Sterba <dsterba@suse.com>
2023-08-28 17:24:23 +02:00
Christoph Heiss 8b73f22484 btrfs-progs: subvol show: remove duplicated quotas error check
The exact same check is repeated here, with the second being dead code.
Keep the second instance, as that informs the user what is happening.

Signed-off-by: Christoph Heiss <christoph@c8h4.io>
Signed-off-by: David Sterba <dsterba@suse.com>
2023-08-28 17:24:23 +02:00
Christoph Heiss 27a5722fe5 btrfs-progs: common: document time-long output format
Seems this was forgotten; rectify that.

Signed-off-by: Christoph Heiss <christoph@c8h4.io>
Signed-off-by: David Sterba <dsterba@suse.com>
2023-08-28 17:24:23 +02:00
Anand Jain 59fe1f46fe btrfs-progs: btrfstune: consolidate error handling in main()
The upcoming "--device" option requires memory to parse devices, which
should be freed before returning from the main() function. As a
preparation for adding the "--device" option to the "btrfstune" command,
provide a consolidated error return exit from the main function with a
"goto" labeled "free_out". The label "free_out" may not make sense
currently, but it will when the "--device" option is added.

There are several return statements within the main function, and
changing all of them in the main "--device" feature patch would deviate
from the actual for the feature changes. Hence, it made sense to create
a preparatory patch.

The return value for any failure remains the same as in the original code.

Signed-off-by: Anand Jain <anand.jain@oracle.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2023-08-28 17:24:23 +02:00
Anand Jain 9ce233965f btrfs-progs: track active metadata_uuid per fs_devices
When the fsid does not match the metadata_uuid, the METADATA_UUID flag
is set in the superblock.

Changing the fsid using the btrfstune -U|-u option is not possible on a
filesystem with the METADATA_UUID flag set. But we are checking the
METADATA_UUID only from the super_copy, and not from the other scanned
device.

To fix this bug, track the metadata_uuid at the fs_devices level instead
of checking it only on the specified device in the argument, and use it.

Signed-off-by: Anand Jain <anand.jain@oracle.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2023-08-28 17:24:22 +02:00
Anand Jain 26f25402ff btrfs-progs: kernel-shared: track total_devs in fs devices
Similar to the kernel, introduce the btrfs_fs_devices::total_devs
attribute to know the overall count of devices that are expected to be
present per filesystem.

Signed-off-by: Anand Jain <anand.jain@oracle.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2023-08-23 19:36:31 +02:00
Anand Jain cb53874f6d btrfs-progs: track changing_fsid flag in fs_devices
To prepare for reuniting separated devices due to an incomplete fsid
change task, consolidate and monitor the per device's changing_fsid flag
in the struct btrfs_fs_devices.

Signed-off-by: Anand Jain <anand.jain@oracle.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2023-08-23 19:36:31 +02:00
Anand Jain 19c337ad9a btrfs-progs: btrfstune: check for missing devices
If btrfstune is executed on a filesystem that contains a missing device,
the command will now fail.

It is ok to fail when any of the options supported by btrfstune are
used, the filesystem devices should be all available.

Signed-off-by: Anand Jain <anand.jain@oracle.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2023-08-23 19:36:31 +02:00
Anand Jain 4673ec037e btrfs-progs: kernel-shared: track missing devices number
Maintain the btrfs_fs_devices::missing counter to track the number of
missing devices, similar to what kernel does.

Signed-off-by: Anand Jain <anand.jain@oracle.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2023-08-23 19:36:31 +02:00
Anand Jain b8b0f1ec41 btrfs-progs: fix duplicate missing device
btrfs_read_sys_array() adds a missing device with its devid only.  So,
any subsequent btrfs_find_device(..devid, uuid) call in the
open_ctree_fd() still fails resulting in addition of duplicate struct
btrfs_device to the list, as shown below.

  open_ctree_fd()
  ::
   btrfs_setup_chunk_tree_and_device_map()
    btrfs_read_sys_array()
     read_one_chunk()
      btrfs_find_device()
      fill_missing_device() <--- dev_uuid wasn't updated
      list_add()
    btrfs_read_chunk_tree
      read_one_dev()
	 btrfs_find_device(..,devid, dev_uuid,..); <-- fails
	 list_add
	 fill_device_from_item(leaf, dev_item, device);

This ends up having two btrfs_device for the same missing devid.

For example:

(btrfstune is compiled with a boilerplate code to dump the device list,
as in the mailing list).

Before:

There are two device with devid 1.

$ btrfstune -m ./disk4.raw
warning, device 1 is missing
[fsid: 95bbc163-671a-4a0a-bd34-03a65e4b338c]
	size:			120
	metadata_uuid:		95bbc163-671a-4a0a-bd34-03a65e4b338c
	fs_devs_addr:		0xdb64e0
	total_rw_bytes:		1048576000
	[[UUID: 703a4cac-bca0-47e4-98f6-55e530800172]]
		sb_flags: 0x0
		sb_incompact_flags: 0x0
		dev_addr:	0xdb69a0
		device:		(null)
		devid:		1
		generation:	0
		total_bytes:	524288000
		bytes_used:	127926272
		type:		0
		io_align:	4096
		io_width:	4096
		sector_size:	4096
	[[UUID: 00000000-0000-0000-0000-000000000000]]
		sb_flags: 0x0
		sb_incompact_flags: 0x0
		dev_addr:	0xdb3060
		device:		(null)
		devid:		1
		generation:	0
		total_bytes:	0
		bytes_used:	0
		type:		0
		io_align:	0
		io_width:	0
		sector_size:	0
	[[UUID: 1db7564f-e53b-46ff-8a33-a8b2d00d86d1]]
		sb_flags: 0x1000000001
		sb_incompact_flags: 0x141
		dev_addr:	0xdb6e90
		device:		/tdev/disk4.raw
		devid:		2
		generation:	6
		total_bytes:	524288000
		bytes_used:	127926272
		type:		0
		io_align:	4096
		io_width:	4096
		sector_size:	4096

Fix this issue by adding the UUID to the missing device created in
fill_missing_device().

After:

$ btrfstune -m /tdev/disk4.raw
warning, device 1 is missing
[fsid: 95bbc163-671a-4a0a-bd34-03a65e4b338c]
        size:                   120
        metadata_uuid:          95bbc163-671a-4a0a-bd34-03a65e4b338c
        fs_devs_addr:           0x161f380
        total_rw_bytes:         1048576000
        [[UUID: 703a4cac-bca0-47e4-98f6-55e530800172]]
                sb_flags: 0x0
                sb_incompact_flags: 0x0
                dev_addr:       0x161c060
                device:         (null)
                devid:          1
                generation:     0
                total_bytes:    524288000
                bytes_used:     127926272
                type:           0
                io_align:       4096
                io_width:       4096
                sector_size:    4096
        [[UUID: 1db7564f-e53b-46ff-8a33-a8b2d00d86d1]]
                sb_flags: 0x1000000001
                sb_incompact_flags: 0x141
                dev_addr:       0x161fe90
                device:         /tdev/disk4.raw
                devid:          2
                generation:     6
                total_bytes:    524288000
                bytes_used:     127926272
                type:           0
                io_align:       4096
                io_width:       4096
                sector_size:    4096

Signed-off-by: Anand Jain <anand.jain@oracle.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2023-08-23 19:36:31 +02:00
Anand Jain 57cd29c895 btrfs-progs: tests: return metadata_uuid or fsid as per METADATA_UUID flag
Previous commit "btrfs-progs: dump-super: print actual metadata_uuid
value" changed the value of the super_block::metadata_uuid to be printed
as it is, without tweaking it depending on the METADATA_UUID flag.

Apply similar tweak in the common helper functions used to read the
metadata_uuid so that test-cases still be successful.

Signed-off-by: Anand Jain <anand.jain@oracle.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2023-08-23 19:36:31 +02:00
Anand Jain 1d988f3638 btrfs-progs: dump-super: print actual metadata_uuid value
The function btrfs_print_superblock() prints all members of the
superblock as they are, except for the superblock::metadata_uuid.
If the METADATA_UUID flag is unset, it prints the fsid instead of
zero as in the superblock::metadata_uuid.

Perhaps this was done because to match with the kernel
btrfs_fs_devices::metadata_uuid value as it also sets fsid if
METADATA_UUID flag is unset.

However, the actual superblock::metadata_uuid is always zero if the
METADATA_UUID flag is unset. Just to mention the kernel does not alter
the superblock::metadata_uuid value any time.

The dump-super printing fsid instead of zero, is confusing because we
generally expect dump_super to print the superblock value in the raw
format without modification.

Fix this by printing the actual metadata_uuid value instead of fsid.

Signed-off-by: Anand Jain <anand.jain@oracle.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2023-08-23 19:36:31 +02:00