Guest os not started and the Virtual Machine Service for vfkit keeps using 400% CPU

[hgl]

I use the following command to start my VM:

vfkit \
  --cpus 8 \
  --memory 6144 \
  --bootloader efi,variable-store=/var/lib/virby/efistore.nvram,create \
  --device virtio-blk,path=/var/lib/virby/diff.img \
  --device virtio-fs,sharedDir=/var/lib/virby/vm_sshd_keys,mountTag=sshd-keys \
  --device virtio-net,nat,mac=02:94:26:8b:f0:13 \
  --restful-uri tcp://localhost:31223 \
  --device virtio-rng \
  --device virtio-balloon

And from the Activity Monitor, a Virtual Machine Service for vfkit process will start and keep using 400% CPU. /var/db/dhcpd_leases doesn't get updated.

If I add an additional --device virtio-serial,stdioor --device virtio-serial,logFilePath=/tmp/vm.out, I can see the os correctly boots and /var/db/dhcpd_leases gets updated.

If I use --device virtio-serial,pty, however, the vm hangs again.

vfkit is built from the main branch.

Any idea what could have caused that?

My machine is macOS 15.6, M2 Pro

[hgl]

If I use --device virtio-serial,pty, wait a long time to confirm /var/db/dhcpd_leases not gets updated and then use screen /dev/ttys009, I can see the systemd loading screen and then /var/db/dhcpd_leases is updated.

I think I'm experiencing Schrödinger's vfkit.

[cfergeau]

We haven’t been able to get --device virtio-serial,pty to work as we’d like, as you’ve noticed, in some cases, when pty is specified, the VM won’t start unless there’s a connection to the PTY it allocated.

At this point, it’s unclear if it’s a limitation of Apple’s virtualization framework or of vfkit’s code.

[hgl]

Thanks for the quick reply.

In my case if I omit --device virtio-serial completely, the vm also hangs. I guess that's probably not related to pty? I tried 0.6.1 as well, same result.

[hgl]

Running vfkit without any --device virtio-serial

sudo /nix/store/piwmiszrbqdyr294j1x6vfc0ygpwspn8-vfkit-0.6.1/bin/vfkit --cpus 8 --memory 6144 --bootloader efi,variable-store=/var/lib/virby/efistore.nvram,create --device virtio-blk,path=/var/lib/virby/diff.img --device virtio-fs,sharedDir=/var/lib/virby/vm_sshd_keys,mountTag=sshd-keys --device virtio-net,nat,mac=02:94:d2:96:ba:9a --restful-uri tcp://localhost:31223 --device virtio-rng --device virtio-balloon

The CPU usage stays 800% forever, and my MacBook gradually becomes hot.

[evidolob]

@hgl Could you share diff.img that you using? I trying to reproduce your problem but I could't.

[hgl]

Thanks. Not sure if it's possible to post a file sharing link here. it's too big to directly upload to GitHub.

https://mega.nz/file/aNkShaaI#z8jgCsTrgaUCPrLqK_zFEiE_sfysTYy_mJTClnBc9Zs

[evidolob]

@hgl I got kernel panic with your image:

<<< NixOS Stage 1 >>>

loading module dm_mod...
running udev...
Starting systemd-udevd version 257.6
kbd_mode: KDSKBMODE: Inappropriate ioctl for device
starting device mapper and LVM...
waiting for device /dev/disk/by-label/nixos to appear.......................
Timed out waiting for device /dev/disk/by-label/nixos, trying to mount anyway.
mounting /dev/disk/by-label/nixos on /...
[   22.224815] /dev/disk/by-label/nixos: Can't lookup blockdev
mount: mounting /dev/disk/by-label/nixos on /mnt-root/ failed: No such file or directory

An error occurred in stage 1 of the boot process, which must mount the
root filesystem on `/mnt-root' and then start stage 2.  Press one
of the following keys:

  r) to reboot immediately
  *) to ignore the error and continue
*Continuing...
stage 2 init script (/mnt-root//nix/store/bcsbq79b5adilj67yw8jzxj867805b2d-nixos-system-virby-vm-efi-raw-25.11.20250809.85dbfc7/init) not found

An error occurred in stage 1 of the boot process, which must mount the
root filesystem on `/mnt-root' and then start stage 2.  Press one
of the following keys:

  r) to reboot immediately
  *) to ignore the error and continue
*Continuing...
BusyBox v1.36.1 () multi-call binary.

Usage: switch_root [-c CONSOLE_DEV] NEW_ROOT NEW_INIT [ARGS]

Free initramfs and switch to another root fs:
chroot to NEW_ROOT, delete all in /, move NEW_ROOT to /,
execute NEW_INIT. PID must be 1. NEW_ROOT must be a mountpoint.

        -c DEV  Reopen stdio to DEV after switch
[   65.469496] Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000100
[   65.469546] CPU: 4 UID: 0 PID: 1 Comm: switch_root Not tainted 6.12.41 #1-NixOS
[   65.469590] Hardware name: Apple Inc. Apple Virtualization Generic Platform, BIOS 2075.140.4.0.0 07/11/2025
[   65.469639] Call trace:
[   65.469657]  dump_backtrace+0xf8/0x150
[   65.469689]  show_stack+0x20/0x50
[   65.469734]  dump_stack_lvl+0x60/0x98
[   65.469767]  dump_stack+0x18/0x30
[   65.469805]  panic+0x164/0x378
[   65.469836]  do_exit+0x9f4/0xa00
[   65.469858]  do_group_exit+0x3c/0xa0
[   65.469889]  __arm64_sys_exit_group+0x20/0x28
[   65.469950]  invoke_syscall+0x50/0x160
[   65.469976]  el0_svc_common.constprop.0+0x48/0x130
[   65.470016]  do_el0_svc+0x24/0x50
[   65.470041]  el0_svc+0x38/0x178
[   65.470067]  el0t_64_sync_handler+0x140/0x150
[   65.470105]  el0t_64_sync+0x190/0x198
[   65.470132] SMP: stopping secondary CPUs
[   65.470462] Kernel Offset: disabled
[   65.470664] CPU features: 0x00,00000003,00222928,66267723
[   65.470763] Memory Limit: none
[   65.573447] ---[ end Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000100 ]---

Is that expected?
You could try to use --device virtio-serial,stdio to see boot log immediately

[hgl]

@evidolob yes. I can reproduce the kernel panic, but I can also reproduce the high CPU usage. Let me explain and sorry I was not clear.

The issue current for me is that, when I run the vfkit command without any --device virtio-serial, the CPU usage is kept at 800% and never drops, this is same with either the current image that panics (I probably removed too much stuff, while trying to keep the size small), or the image (much bigger) that I used in OP that doesn't panic

If I run the vfkit command with --device virtio-serial,stdio, even though the boot fails, the Virtual Machine Service for vfkit process never uses more than 20% CPU.

So I can already reproduce the issue with this image.

As I mentioned in my OP, once I use --device virtio-serial,stdio or logFilePath, the issue goes away.

And I strongly suspect that the OS never boots for the high CPU usage case, with the bigger image I used in the OP. Because when I use --device virtio-serial,stdio, /var/db/dhcpd_leases gets updated, when I don't, it's never updated. (I make it empty before running any vfkit command.)

I wonder if you can reproduce the high CPU usage by omitting --device virtio-serial?

I'm not sure it will be helpful to send the bigger image that boots, but if you insist, I will.

[hgl]

@evidolob are you able to reproduce? is there any more info I can provide?

[evidolob]

Yes, I was able to see 800% cpu usage on my machine, but only after vm kernel get panic. So I unsure that it is bug in our code.
And what we could do it that case.

[cfergeau]

I agree there is not much we can do after a kernel panic. It could even be the kernel which is using all this CPU ? Or just a bug in the virtualization framework