This patch adds the function to check correspondence between block group,
chunk and device extent.
Original-signed-off-by: Cheng Yang <chenyang.fnst@cn.fujitsu.com>
Signed-off-by: Miao Xie <miaox@cn.fujitsu.com>
Signed-off-by: Chris Mason <chris.mason@fusionio.com>
As we know, btrfs can manage several devices in the same fs, so [offset, size]
is not sufficient for unique identification of an device extent, we need the
device id to identify the device extents which have the same offset and size,
but are not in the same device. So, we added a member variant named objectid
into the extent cache, and introduced some functions to make the extent cache
be suitable to manage the device extent.
Signed-off-by: Miao Xie <miaox@cn.fujitsu.com>
Signed-off-by: Chris Mason <chris.mason@fusionio.com>
Because the fs/file roots are not extents, so it is better to use rb-tree
to manage them. Fix it.
Signed-off-by: Miao Xie <miaox@cn.fujitsu.com>
Signed-off-by: Chris Mason <chris.mason@fusionio.com>
In fact, the code of many rb-tree insert/search/delete functions is similar,
so we can abstract them, and implement common functions for rb-tree, and then
simplify them.
Signed-off-by: Miao Xie <miaox@cn.fujitsu.com>
Signed-off-by: Chris Mason <chris.mason@fusionio.com>
Some commands(such as btrfs-convert) access the devices again after we close
the ctree, so it is better that we don't free the devices objects when the ctree
is closed, or we need re-allocate the memory for the devices. We needn't worry
the memory leak problem, because all the memory will be freed after the taskes
die.
Signed-off-by: Miao Xie <miaox@cn.fujitsu.com>
Signed-off-by: Chris Mason <chris.mason@fusionio.com>
As we know, the file descriptor 0 is a special number, so we shouldn't
use it to initialize the file descriptor of the devices, or we might
close this special file descriptor by mistake when we close the devices.
"-1" is a better choice.
Signed-off-by: Miao Xie <miaox@cn.fujitsu.com>
Signed-off-by: Chris Mason <chris.mason@fusionio.com>
When making btrfs filesystem. we firstly write root leaf to
specified filed, and then we recow the root. If we don't recow,
some trees are not in the correct block group.
Steps to reproduce:
dd if=/dev/zero of=test.img bs=1M count=100
mkfs.btrfs -f test.img
btrfs-debug-tree test.img
extent tree key (EXTENT_TREE ROOT_ITEM 0)
leaf 4210688 items 10 free space 3349 generation 4 owner 2
fs uuid 2e08fd93-f24d-4f44-a226-e2116fcd544f
chunk uuid dc482988-6246-46ce-9329-68bcf6d3683c
item 0 key (0 BLOCK_GROUP_ITEM 4194304) itemoff 3971 itemsize 24
block group used 12288 chunk_objectid 256 flags 2
[..snip..]
item 3 key (1138688 EXTENT_ITEM 4096) itemoff 3827 itemsize 42
extent refs 1 gen 1 flags 2
tree block key (0 UNKNOWN.0 0) level 0
item 4 key (1138688 TREE_BLOCK_REF 7) itemoff 3827 itemsize 0
tree block backref
[..snip..]
checksum tree key (CSUM_TREE ROOT_ITEM 0)
leaf 1138688 items 0 free space 3995 generation 1 owner 7
fs uuid 2e08fd93-f24d-4f44-a226-e2116fcd544f
chunk uuid dc482988-6246-46ce-9329-68bcf6d3683c
For the above example, csum root leaf comes into system block group which
is wrong,csum root leaf should be in metadata block group.
Signed-off-by: Wang Shilong <wangsl-fnst@cn.fujitsu.com>
Reviewed-by: Miao Xie <miaox@cn.fujitsu.com>
Signed-off-by: Chris Mason <chris.mason@fusionio.com>
I noticed that I was getting these errors on a bigger file system with more
snapshots that had been removed. This check is bogus since we won't inc
rec->found_ref if we don't find a REF_KEY _and_ a DIR_ITEM, so we only have to
worry about there being no references to a root if it actually has a root item.
If it doesn't then it's just referenced by things that will go no where anyway.
With this patch fsck no longer incorrectly complains about this file system
image I have. Thanks,
Signed-off-by: Josef Bacik <jbacik@fusionio.com>
Signed-off-by: Chris Mason <chris.mason@fusionio.com>
A user reported that fsck was complaining about unresolved refs for some
snapshots. You can reproduce this by doing
mkfs.btrfs /dev/sdb
mount /dev/sdb /mnt
btrfs subvol snap /mnt/ /mnt/a
btrfs subvol snap /mnt/ /mnt/b
btrfs subvol del /mnt/a
umount /mnt
btrfsck /dev/sdb
and you'd get this
unresolved ref root 258 dir 256 index 2 namelen 1 name a error 600
because snapshot b has a dir item that points to a. Except we encode in our
root ref the dirid of the ref holder, and if it doesn't match we just give it
back a empty directory since we can't hardlink directories. This makes the
check in btrfsck bogus, when we delete a we remove the ref key for it so any
lookups into /mnt/b/a will just give a blank directory as it's supposed to. Fix
this by only saying the backref is reachable if there is both a DIR_ITEM and a
REF_KEY for the given root. With this patch I no longer see errors when running
this reproducer. Thanks,
Signed-off-by: Josef Bacik <jbacik@fusionio.com>
Signed-off-by: Chris Mason <chris.mason@fusionio.com>
There is a problem where if we find a backref extent record first that doesn't
match a extent item we will delete some of the duplicates but not others. In
order to deal with this we need to make sure we only pay attention to duplicates
that actually have duplicate extent items. If a extent_rec has a duplicate but
the record itself doesn't have an associated extent item we promote the
duplicate to the extent record and just discard the original extent_rec since it
was just added by the backref. We copy the backref onto the promoted extent
record and then continue processing. This allowed me to fix a file system that
previously was not able to be fixed by fsck. Thanks,
Signed-off-by: Josef Bacik <jbacik@fusionio.com>
Signed-off-by: Chris Mason <chris.mason@fusionio.com>
The allocator looks for these hints when moving on to another block group which
will make it reset which block group it looks at, when we've already searched
that block group and didn't find any space to allocate, we need to fix this by
just letting the allocator make the determination if the block group is good
enough. This also fixes a problem where if we couldn't find space in the block
group we were given we'd just error out instead of moving on to the next block
group. Previously I couldn't fix some file systems that were relatively full,
but with this patch I can now run fsck on them with no allocation errors.
Thanks,
Signed-off-by: Josef Bacik <jbacik@fusionio.com>
Signed-off-by: Chris Mason <chris.mason@fusionio.com>
This fixes two bugs with the free space cache checker. First is we apparently
always use root->sectorsize for our unit in the kernel so we have to do that in
progs otherwise bitmaps turn out to not look right if we have leafsize !=
sectorsize. The second is a small issue if we had skinny metadata extents set,
we wouldn't advance last properly because we unconditionally use key.offset
instead of root->leafsize. Thanks,
Signed-off-by: Josef Bacik <jbacik@fusionio.com>
Signed-off-by: Chris Mason <chris.mason@fusionio.com>
In some cases the extent tree can just be so gone there is no point in trying to
figure out how to put it back together. So add a --init-extent-tree mode which
will zero out the extent tree and then re-add extents for all of the blocks we
find. This will also undo any balance that was going on at the time of the
crash, this is needed because the reloc tree seems to confuse fsck at the
moment. With this patch I can put back together a users file system that was
completely gone. Thanks,
Signed-off-by: Josef Bacik <jbacik@fusionio.com>
Signed-off-by: Chris Mason <chris.mason@fusionio.com>
Left out a newline in the generation check printf.
Signed-off-by: Josef Bacik <jbacik@fusionio.com>
Signed-off-by: Chris Mason <chris.mason@fusionio.com>
I noticed a slight problem with btrfs-image, since it was building a chunk tree
by setting the physical offset of the stripes to the same as the logical offset
it created this problem where the super block was now mapped into the file
system differently than it was before. This isn't a huge deal except that we
also carry along the free space cache with us, which is setup with the idea that
super at physical X is at logical Y. So this would make the free space checker
in fsck freak out because it would see that the cache says that the super block
is free space, and that the area where it thought the super block was located is
in fact used. In the mount case we'd end up overwriting real metadata with
backup super blocks. So we need to maintain the physical offsets in our
stripes. This is a huge pain because we store the logical bytenrs of all of our
metadata. This patch scans the entire image looking for chunk tree blocks and
builds an in memory chunk tree so we can write logical blocks to their physical
offsets. With this patch we no longer have the problems I described above.
Thanks,
Signed-off-by: Josef Bacik <jbacik@fusionio.com>
Signed-off-by: Chris Mason <chris.mason@fusionio.com>
We have a BUG_ON() in read_fs_root that expects key->offset == (u64)-1. Restore
will just pass in the location it reads out of the inode if it has a subvol
reference and pass that in, which doesn't have offset == (u64)-1, so it causes
restore blow up. This was reported in bugzilla and this patch fixed the
problem. Thanks,
Signed-off-by: Josef Bacik <jbacik@fusionio.com>
Signed-off-by: Chris Mason <chris.mason@fusionio.com>
The tree log bug I introduced could create inconsistent file extent entries in
the file system tree and in some worst cases even create multiple extent entries
for the same entry. To fix this we need to do a few things
1) Keep track of extent items that overlap and then pick the one that covers the
largest area and delete the rest of the items.
2) Keep track of file extent items that land in extent items but don't match
disk_bytenr/disk_num_bytes exactly. Once we find these we need to figure out
who is the right ref and then fix all of the other refs to agree.
Each of these cases require a complete rescan of all of the extents, so
unfortunately if you hit this particular problem the fsck is going to take quite
a while since it will likely rescan all the trees 2 or 3 times. With this patch
the broken file system a user sent me is fixed and a broken file system that was
created by my reproducer is also fixed. Thanks,
Signed-off-by: Josef Bacik <jbacik@fusionio.com>
Signed-off-by: Chris Mason <chris.mason@fusionio.com>
We've had a few users who wouldn't (or couldn't) provide us btrfs-images because
we maintain the file names when making an image. So introduce a sanitize
option. There are two uses, one that is fast and the other that is dog slow.
The fast way just generates garbage that's equal in length to the original name.
The slow way will try and find a crc32c collision for the file name that is also
the same length. Finding a crc32c collision for the file name "btrfs-progs" on
my box without CPU crc32c support takes a little more than 3 minutes, and a
little less than 2 minutes for my box that has CPU crc32c support, so it's a
lengthy and CPU intensive process.
The idea is that we use -s for most cases, and then only use -ss when we need
the file system tree to be somewhat sane. I could probably do a better job
about finding collisions, but I'll have to revist that later. Thanks,
Signed-off-by: Josef Bacik <jbacik@fusionio.com>
Signed-off-by: Chris Mason <chris.mason@fusionio.com>
An unfortunate side effect to my fsync bug means that anybody who didn't hit the
BUG_ON() during tree log replay would have ended up with a corrupted file
system. Currently our fsck does not catch this because it just looks for
bytenrs for backrefs, it doesn't look at the num_bytes at all. So this patch
makes us keep track of how big the backrefs are, since their disk_num_bytes
_have_ to match the number of bytes for the actual extent item. With this patch
fsck now finds problems with a file system it previously thought was ok.
Thanks,
Signed-off-by: Josef Bacik <jbacik@fusionio.com>
I hit this while working on fsck, I got some weird corruption where the number
of items was way higher than what would fit in a leaf, which would make things
blow up. This fixes the problem by catching it and returning an error so we
gracefully exit instead of segfaulting. Thanks,
Signed-off-by: Josef Bacik <jbacik@fusionio.com>
Signed-off-by: Chris Mason <chris.mason@fusionio.com>
See the warnings below:
[CC] btrfs-list.o
btrfs-list.c: In function 'filter_by_parent':
btrfs-list.c:1183:34: warning: cast to pointer from integer of different size
[-Wint-to-pointer-cast]
[CC] cmds-subvolume.o
cmds-subvolume.c: In function 'cmd_subvol_show':
cmds-subvolume.c:917:5: warning: cast from pointer to integer of different size
[-Wpointer-to-int-cast]
[CC] cmds-restore.o
cmds-restore.c: In function 'decompress_lzo':
cmds-restore.c:116:10: warning: passing argument 4 of 'lzo1x_decompress_safe'
from incompatible pointer type [enabled by default]
In file included from cmds-restore.c:31:0:
/usr/include/lzo/lzo1x.h:77:1: note: expected 'lzo_uint *' but argument is of
type 'size_t *'
Reported-by: Russell Coker <russell@coker.com.au>
Signed-off-by: Wang Shilong <wangsl-fnst@cn.fujitsu.com>
Signed-off-by: Chris Mason <chris.mason@fusionio.com>
Only the first byte of the wanted csum is printed:
checksum verify failed on 65536 found DA97CF61 wanted 6B
checksum verify failed on 65536 found DA97CF61 wanted 6BC3870D
Also add leading zeros to the format.
Signed-off-by: David Sterba <dsterba@suse.cz>
Signed-off-by: Chris Mason <chris.mason@fusionio.com>
The patch set previously sent was sent together with the kernel part, but
was not updated as I added some reserved bytes to the ioctl struct for
future compatibility. This fixes struct btrfs_ioctl_quota_rescan_args.
Signed-off-by: Jan Schmidt <list.btrfs@jan-o-sch.net>
Signed-off-by: Chris Mason <chris.mason@fusionio.com>
btrfsck was blowing up when checking the free space cache when we ran xfstests
with -l 64k. That is because I was init'ing the free space ctl to whatever the
leafsize was, which isn't right for data block groups. With this patch btrfsck
no longer complains. This also fixes a tiny little typo in free-space-cache.c I
noticed while figuring this problem out. Thanks,
Signed-off-by: Josef Bacik <jbacik@fusionio.com>
Signed-off-by: Chris Mason <chris.mason@fusionio.com>
btrfs-image is assuming that every metadata block is the same size. The
super is a special snowflake though. It's 4K even on 64K filesystems.
Signed-off-by: Chris Mason <chris.mason@fusionio.com>
With commit
87c09f7 Btrfs-progs: fix memory leaks on cleanup
mkfs on multiple dev is ending with segfault at
close_all_devices() during kfree(device->name)
because mkfs calls btrfs_add_to_fsid, which does not initialize
name when dev is added to the list.
Signed-off-by: Anand Jain <anand.jain@oracle.com>
Signed-off-by: Chris Mason <chris.mason@fusionio.com>
Add a default rule for any btrfs-$FOO or btrfs-$FOO.static
target, allowing it to be built from btrfs-$FOO.c along with
all the normal userspace objects.
This gets rid of a lot of the cut and pasted rules for
each individual command, and as an added bonus makes it
easy to build any btrfs-$FOO statically as well, i.e.
# make btrfs-convert.static
Signed-off-by: Eric Sandeen <sandeen@redhat.com>
For any btrfs-$FOO executable, rename the main source file from
$FOO.c to to btrfs-$FOO.c
This makes it slightly more obvious what's building what,
and allows us to write a default rule in the Makefile for
these tools.
(also add btrfs-calc-size to the list of objects to remove
on make clean)
Signed-off-by: Eric Sandeen <sandeen@redhat.com>
Several tools like btrfs-send and btrfs-receive need to map a
subvolume ID to a filesystem path. The so far existing methods
in btrfs-list.c cause a horrible effort when performing this
operation (and the effort is dependent on the number of
existing subvolumes with quadratic effort). This commit adds a
function that is able to map a subvolume ID to a filesystem path
with an effort that is independent of the number of existing
subvolumes.
In addition to this function, a command line frontend is added as well:
btrfs inspect-internal subvolid-resolve <subvolid> <path>
Signed-off-by: Stefan Behrens <sbehrens@giantdisaster.de>
For the objectid and offset field of a key, print -1 instead of the
decimal representation of 0xffffffffffffffff. At least for me it is
more readable like this.
Signed-off-by: Stefan Behrens <sbehrens@giantdisaster.de>
Not important at all since exit() is called afterwards and this is
not part of the library. It just makes valgrind happy and thus
allows to search for real flaws.
Signed-off-by: Stefan Behrens <sbehrens@giantdisaster.de>
This addresses the same issue as did:
2bd1169 btrfs-progs: root_item generation_v2 is out of sync after btrfsck
but rather than optionally updating generation_v2 based
on the size of the existing item, increase the size of the
item as needed, and unconditionally set generation_v2.
This matches the kernel code, and keeping things in sync is a
Good Thing.
Signed-off-by: Eric Sandeen <sandeen@redhat.com>
With this integration branch commit in place:
2bd1169 btrfs-progs: root_item generation_v2 is out of sync after btrfsck
I started seeing generation mismatch messages from the kernel
at mount time, after a fresh mkfs(!):
btrfs: mismatching generation and generation_v2 found in root item...
This is because the code which emits the warning does not do so if
there is a mismatch but generation_v2 is 0; the above commit began
setting generation_v2 to something non-zero, so the warning was emitted.
The reason there is a mismatch at all is because mkfs.btrfs calls
create_data_reloc_tree(), which copies a root, and then calls
btrfs_set_root_generation(), bumping the original copied generation.
But nothing updated generation_v2 to match on the way to disk.
Fix this by updating generation_v2 in btrfs_insert_root(),
as is done in the kernel.
This is safe because it's a new root created by userspace, so
the btrfs_root_item is guaranteed to be big enough to contain
generation_v2.
Signed-off-by: Eric Sandeen <sandeen@redhat.com>
btrfs_read_and_process_send_stream() recently changed its prototype;
this gets it building again. 0 means "do not honor the <end cmd>"
which was the previous default, before
2bfac02 Btrfs-progs: btrfs-receive optionally honors the end-cmd
Signed-off-by: Eric Sandeen <sandeen@redhat.com>
This adds the quota rescan command to be used if qgroup tracking should get
out of sync. Can also be used to query the status of a running rescan
operation.
Signed-off-by: Jan Schmidt <list.btrfs@jan-o-sch.net>
All we need for restore to work is the chunk root, the tree root and the fs root
we want to restore from. So to do this we need to make a few adjustments
1) Make open_ctree_fs_info fail completely if it can't read the chunk tree.
There is no sense in continuing if we can't read the chunk tree since we won't
be able to translate logical to physical blocks.
2) Use open_ctree_fs_info in restore, and if we didn't load a tree root or
fs root go ahead and try to set those up manually ourselves.
This is related to work I did last year on restore, but it uses the
open_ctree_fs_info instead of my open coded open_ctree. Thanks,
Signed-off-by: Josef Bacik <jbacik@fusionio.com>
I was running fsync() tests and noticed that occasionally I was getting a bunch
of errors from fsck complaining about csums not having corresponding extents.
Thankfully after a few days of debugging this it turned out to be a bug with
fsck. The csums were for an extent that started at the same offset as a block
group, and were offset within the extent. So the search put us out at the block
group item and we just walked forward from there, never finding the actual
extent. This is because the block group item key is higher than the extent item
key, so it comes first. In order to fix this we need to check and see if we
landed on a block group item and take another step backwards to make sure we end
up at the extent item. With this patch my reproducer no longer finds csums that
don't have matching extent records. Thanks,
Signed-off-by: Josef Bacik <jbacik@fusionio.com>
Looking at a recent user problem I noticed there are weird cases we could
possibly be leaving csums in place for an extent we've free'd. I don't think
this can happen unless the extent tree is also corrupt, but just in case I'm
adding sanity checks to btrfsck. This way we will catch this if it happens
normally since xfstests runs btrfsck between each run. Thanks,
Signed-off-by: Josef Bacik <jbacik@fusionio.com>
I made open_ctree fail if the chunk tree couldn't be open, which means that fsck
now segfaults if it can't open the chunk tree. So fix fsck to check the fs_info
we get back from open_ctree_fsinfo to make sure it's valid and exit if it's not
instead of segfaulting. Thanks,
Signed-off-by: Josef Bacik <jbacik@fusionio.com>
When working with a user with a broken file system I noticed I wasn't able to
read some of the blocks properly from the restored image. This is because his
extent tree was corrupt and was missing references to some of the blocks, which
means they weren't copied into the image when he generated it. So add a -w
option which will walk all of the trees manually and copy them into the image.
This way we can run fsck against a complete file system image and fix any bugs
in fsck. Thanks,
Signed-off-by: Josef Bacik <jbacik@fusionio.com>
is_ssd() uses nondescript variable names; path - to what?
disk - it's a dev_t not a disk name, unlike dev, which is
a name not a dev_t!
Rename some vars to make things hopefully clearer:
wholedisk - the name of the node for the entire disk
devno - the dev_t of the device we're mkfs'ing
sysfs_path - the path in sysfs we ultimately check
Signed-off-by: Eric Sandeen <sandeen@redhat.com>
blkid_probe_get_wholedisk_devno() isn't available in some older
versions of libblkid. It was used to work around an old
bug in blkid_devno_to_wholedisk(), but that has been fixed since
5cd0823 libblkid: fix blkid_devno_to_wholedisk(), present in
util-linux 2.17 and beyond.
If we happen to be missing that fix, the worst that happens is
that we'd fail to detect that a device is an ssd; the upside is
that this code compiles on older systems.
Signed-off-by: Eric Sandeen <sandeen@redhat.com>
The receive code was not distinguishing properly between the mount root
and the directory to create the received subvolume in.
Also make sure the find_mount_root reports an error if it cannot find
a match at all.
Reported-by: Robert Buhren <robert@robertbuhren.de>
Reported-by: Rory Campbell-Lange <rory@campbell-lange.net>
Reported-by: Stefan Priebe - Profihost AG <s.priebe@profihost.ag>
Signed-off-by: Alex Lyakas <alex.btrfs@zadarastorage.com>
Signed-off-by: Stefan Behrens <sbehrens@giantdisaster.de>
