Multicore LIBC prototype working but issues with SystemLog::write

[torgeiru]

I have a working prototype of multicore LIBC and thread local storage.

It works by patching musl pthread_create so that it only initializes the thread area (mmap'd), copy TLS image and returning thread pointer. The thread pointer can be directly inserted into core FS_BASE msr.

There is an issue SystemLog::write. SystemLog::write runs smoothly on CPU#0, but not CPU#N where N>0. The code accesses some memory that is not mapped into the page tables of the aux CPUs. By removing SystemLog I can successfully print text concurrently (patched musl __lockfile with spinlock and away with the futex calls).

IncludeOS TLS fork: https://github.com/torgeiru/IncludeOS/tree/TLS_implementation
Musl patch: https://github.com/torgeiru/musl-includeos (1.2.5 commit pinned + patches)

[alfreb]

Oh cool! Is there an integration test that demonstrates this working? Would be happy to give it a spin!

[fwsGonzo]

There is an issue SystemLog::write. SystemLog::write runs smoothly on CPU#0, but not CPU#N where N>0. The code accesses some memory that is not mapped into the page tables of the aux CPUs. By removing SystemLog I can successfully print text concurrently (patched musl __lockfile with spinlock and away with the futex calls).

Yeah, we never really got far enough to make everything nice with SMP. The systemlog was mostly there to have a fixed arena of bytes that was inspectable after/during a crash. It wouldn't necessarily make sense to expand it to all CPUs. But, if you want to you can just make the other CPUs send a log message back to CPU 0 which then appends to the log.

[torgeiru]

There is an issue SystemLog::write. SystemLog::write runs smoothly on CPU#0, but not CPU#N where N>0. The code accesses some memory that is not mapped into the page tables of the aux CPUs. By removing SystemLog I can successfully print text concurrently (patched musl __lockfile with spinlock and away with the futex calls).

Yeah, we never really got far enough to make everything nice with SMP. The systemlog was mostly there to have a fixed arena of bytes that was inspectable after/during a crash. It wouldn't necessarily make sense to expand it to all CPUs. But, if you want to you can just make the other CPUs send a log message back to CPU 0 which then appends to the log.

I may do something like that. I still think there is an issue because systemlog CPU#0 stuff maps to weird addrs atm (if I remember correctly), may corrupt stuff when writing to the log buffer. Will look more into this.

I think it would be useful to store logs in non-volatile storage e.g. firmware storage. UEFI has uefi variable interface provided by runtime services. I don't know the support for this in QEMU with direct boot (default boot).

[torgeiru]

Oh cool! Is there an integration test that demonstrates this working? Would be happy to give it a spin!

I will verify, fix, write tests today. I did manual test before taking a longer summer break

I tested TBSS and TDATA worked correctly. Writing to TLS storage only modified storage for the core in question.

I tested concurrent writing to screen and it worked. Will need to stress test malloc

I will stress test malloc today.

Watch out for tomorrow

[torgeiru]

Ok, made some cute tests. They are located in the same place (kernel, may move to better place): libc_malloc and tls.

The branch is my own thus the MUSL point to my musl and commented out the patching.

I think it is nice that we have this MUSL/libc compatibility layer but it is hacky to use patch-files. Maybe we should have our own fork of MUSL excl for IncludeOS (stored under IncludeOS org). The threads part of MUSL is not needed and we could use these nice locks made by Magnus.

[alfreb-scalemem]

Cool! I'm a bit bogged down at work these days, but looking forward to giving it a spin! I'm open to maintaining a fork, if it's kept super clean. One reason for having patch files is to enforce just that by making it harder to change musl 😁 I think the approach should be that we try to make patches that we think could be considered for upstreaming, if the musl maintainer decided to support a subset of the linux system calls and not all of them.

[torgeiru]

The main reason why I suggest a fork is this: The MUSL code assumes threading, but IncludeOS is not threaded. This results in need for patches multiple places to work e.g. locks. I think it may be more readable to have our own fork with the entire code in a repo. I assume that the repo is kept very clean (with clear separation of duties) and we are very pedantic about this.

What patches could be considered for upstreaming? Why would the maintainer want to support our subset?

[alfreb-scalemem]

A fork requires maintenance, and protecting libc against vulnerabilities is not something to be taken lightly.
https://www.cve.org/CVERecord/SearchResults?query=musl

[torgeiru]

A fork requires maintenance, and protecting libc against vulnerabilities is not something to be taken lightly.
https://www.cve.org/CVERecord/SearchResults?query=musl

Makes sense.

I reckon we could actually implement a working futex 😁 and we don't need as many patches.
I can do that. Still, we need to update the commit from time to time.

Regarding security and general stability: I think implementing some sort of sanitizing during testing phase may unmask potential bugs e.g. overflows (heap, stack), double frees, uninitialized vars.

I think we should have a discord.

[alfreb-scalemem]

I think mocking a futex is a decent way to go about it. We should error out at the soonest possible point if people are trying to use it in a way we don't support, but that's the same with all other syscalls. Maybe even a compile-time switch so we don't fool people into thinking it's something it's not. But it's better to stub what we can than to maintain a diverging fork. (Officially that is - unofficial forks for proving a point is perfectly fine of course)

And wrt. discord, let's talk about it! I'm bogged down most of next week with a couple of deadlines, but will be back after that.

[alfreb-scalemem]

Wrt. a fully working futex - I think we can do that too with a stack switch. I'd be happy to review a PR for that, but it could be worth discussing the approach beforehand. We generally want to avoid stack switches, but there are some good reasons for it.

[torgeiru]

We can wait, discuss and implement afterwards.