Correct weight recalculation of space-based metaslabs

[pcd1193182]

Sponsored by: [Wasabi Technology, Inc; Klara, Inc]

Motivation and Context

Currently, after a failed allocation, the metaslab code recalculates the weight for a metaslab. However, for space-based metaslabs, it uses the maximum free segment size instead of the normal weighting algorithm. This is presumably because the normal metaslab weight is (roughly) intended to estimate the size of the largest free segment, but it doesn't do that reliably at most fragmentation levels¹. This means that recalculated metaslabs are forced to a weight that isn't really using the same units as the rest of them, resulting in undesirable behaviors (mostly metaslabs never being selected again due to an artificially low weight).

As far as I can tell this code dates back to 2010, long before we had metaslab_should_allocate, segment-based metaslabs, or any of the modern features of the allocation code.

Description

We switch this to use the normal space-weighting function for this recalculation.

How Has This Been Tested?

Tested for correctness with the ZFS test suite. For performance effects, ran extensive performance testing on a highly fragmented pool. The change resulted in a 71% reduction in stddev of TXG sync times, and a 56% reduction in 99th percentile sync times. It also reduced the number of loads of metaslabs that did not result in eventual allocations by approximately 50%.

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Performance enhancement (non-breaking change which improves efficiency)
• Code cleanup (non-breaking change which makes code smaller or more readable)
• Quality assurance (non-breaking change which makes the code more robust against bugs)
• Breaking change (fix or feature that would cause existing functionality to change)
• Library ABI change (libzfs, libzfs_core, libnvpair, libuutil and libzfsbootenv)
• Documentation (a change to man pages or other documentation)

Checklist:

• My code follows the OpenZFS code style requirements.
• I have updated the documentation accordingly.
• I have read the contributing document.
• I have added tests to cover my changes.
• I have run the ZFS Test Suite with this change applied.
• All commit messages are properly formatted and contain Signed-off-by.

Footnotes:
1. The weight works by multiplying the total free space by 100 - fragmentation, and then dividing by 100. As one simple example of how this doesn't do a good job of approximating the largest free segment, if you have free chunks of size ~2MiB, this results in a division by roughly 1.5-2 (It maps to 36 on the frag table, plus presumably many smaller free segments, which increase the fragmentation). However, if you have a 50 of these chunks, this results in a weight that is ~25 times higher than the largest free segment. ⤴