I have a 2 OSDs ceph cluster. The initial size of backing volumes was 16GB. Then I shutdown OSDs, did a lvextend on both, and turn OSDs on again. Now ceph osd df shows:

ceph osd df

But ceph -s show it's stucked at active+remapped+backfill_toofull for 50 pgs: ceph -s

I tried to understand the mechanism by reading CRUSH algorithm but seems a lot of effort and knowledge is required. It would be very appreciated if anyone could describe the behaviour (why stuck in toofull despite the free space is increased significantly.) and help me to resolve this state.

8

1 Answer

Your RAW USE is times larger than DATA. Note: solution not tried, still this is what I've found.

Re: Raw use 10 times higher than data use

Probably the first thing to check is if you have objects that are under the min_alloc size. Those objects will result in wasted space as they will use the full min_alloc size.

Similar advice:

This is related to bluestore_min_alloc_size_hdd=64K (default on Octopus).

If using Erasure Coding, data is broken up into smaller chunks, which each take 64K on disk.

Another way to fix the problem one may try to follow instructions on your second screen:

add storage if this doesn't resolve itself

Your Answer

Sign up or log in

Sign up using Google Sign up using Facebook Sign up using Email and Password

Post as a guest

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy