Add some helpers to report potential problem on scheduler side

[imran-kn]

No description provided.

[brenns10]

I really like this module, it's going to be really useful. The code looks great as well, but I have a suggestion. The docstrings for check_runq_wakelist and check_idle_runq have really useful context, and it's a bit of a shame for that context to be hidden away in the code.

Do you think it would be possible to make each explanation a global variable, and then have the run_queue_check() function print each explanation at the end, only if necessary? Something like this:

EXPLAIN_WAKELIST = """
[TASKS ON WAKE LIST]
    A runq's wakelist temporarily holds tasks that are about to be woken
    up on that CPU. If this list has multiple tasks, it usually means that
    this CPU has missed multiple scheduler IPIs. This can imply issues
    like IRQs being disabled for too long, IRQ delivery issues between hypervisor
    and VM, or some other issue.
"""
EXPLAIN_IDLE_RUNQ = """
[IDLE RUN QUEUE]
    ...
"""

Then, each function that performs checks could take a parameter, explanations: Set[str], and you could do:

explanations.add(EXPLAIN_WAKELIST)
print(f"cpu: {cpu} has following tasks in its runq wake_list:")
...
print("See TASKS ON WAKE LIST below")

And then run_queue_check() could handle printing out explanations:

explanations = set()
for cpu in for_each_online_cpu(prog):
    check_runq_wakelist(prog, cpu, explanations) 
    ...
if not explanations:
    return
print("Note: found some possible run queue issues. Explanations below:")
for explanation in explanations:
    print(explanation)

I think this would be really helpful even for support & sustaining because we don't all have the context for why a detected issue may be a problem, and what next steps we should look into.

[imran-kn]

I really like this module, it's going to be really useful. The code looks great as well, but I have a suggestion. The docstrings for check_runq_wakelist and check_idle_runq have really useful context, and it's a bit of a shame for that context to be hidden away in the code.

Do you think it would be possible to make each explanation a global variable, and then have the run_queue_check() function print each explanation at the end, only if necessary? Something like this:

EXPLAIN_WAKELIST = """
[TASKS ON WAKE LIST]
    A runq's wakelist temporarily holds tasks that are about to be woken
    up on that CPU. If this list has multiple tasks, it usually means that
    this CPU has missed multiple scheduler IPIs. This can imply issues
    like IRQs being disabled for too long, IRQ delivery issues between hypervisor
    and VM, or some other issue.
"""
EXPLAIN_IDLE_RUNQ = """
[IDLE RUN QUEUE]
    ...
"""

Then, each function that performs checks could take a parameter, explanations: Set[str], and you could do:

explanations.add(EXPLAIN_WAKELIST)
print(f"cpu: {cpu} has following tasks in its runq wake_list:")
...
print("See TASKS ON WAKE LIST below")

And then run_queue_check() could handle printing out explanations:

explanations = set()
for cpu in for_each_online_cpu(prog):
    check_runq_wakelist(prog, cpu, explanations) 
    ...
if not explanations:
    return
print("Note: found some possible run queue issues. Explanations below:")
for explanation in explanations:
    print(explanation)

I think this would be really helpful even for support & sustaining because we don't all have the context for why a detected issue may be a problem, and what next steps we should look into.

Yes its definitely possible and I agree that this help string can help in getting a better context of this data. I have made the suggested change

[imran-kn]

@brenns10 thanks a lot for taking a look and sharing your feedback. I have addressed the review comments but at the moment I see some jobs failing. This looks like a setup issue.