Twincloud's Youtube Subtitle Extractor

kimylamp

280.6K

10.2K

2024-04-11

A quick introduction to one of the interprocess communication mechanisms in linux. Demonstration purpose only, don't use in the real code: - signal handler in Linux has a bit different semantics - while executing signal handler code, another signal might come. don't use printf and other reentrant-unsafe functions inside signal handlers - implement global variables concurrency more careful 00:00 Main idea. We want to execute a function when external signal arrives 00:50 While loop. We can't simp...

At some point we want our programs to communicate with other programs. What if we have this function, and when another process sends us a signal, we want this function to be

executed! It's a very natural idea. If we get such a signal, we can, for instance, change some variable and get out of the loop. Let's try to design such a mechanism. First, we

need to receive a signal. It is not so clear what it means, but soon we will understand. And we want to execute the foo() function, which eventually means that we need to make the program counter to

point to its first instruction. A normal way to do it is to use "branch" instruction. But this time it doesn't really help. To understand why, let's take a closer look at our while loop: we load

"signal_arrived" variable, compare it to zero and "branch" back. The variable is always zero - so the CPU runs the loop for infinity. So again, when the signal arrives, we want to force the CPU to

execute foo() function. Unfortunately, the CPU doesn't have any mechanism to break that natural instruction flow and suddenly jump somewhere else. But some similar mechanism does actually

exist. Let's try to recall what interrupts are. If the keyboard button is pressed, the operating system wants to know about it. That's why the CPU has electrical pins to be connected to external

hardware now the keyboard can raise the voltage of that pin and the CPU will immediately detect it. But what happens to our program? The CPU was executing our program. The interrupt arrives. At

this moment the CPU immediately forces the program counter to point to the particular kernel address. Which is the interrupt entry. Look at that! The natural instruction flow is

broken and we jumped to another code! It is not the foo function yet, but still, very promising! The kernel code is not so different from user code. Yes, it has more privileges and so on,

but it is not so important right now. What important, is that the kernel code consists mostly of the same kind of CPU instructions as any other user program. And instructions use the same CPU

registers. So when the program counter suddenly jumps to the interrupt handler, all the other registers still keep values of user process. The kernel's first job is not to lose a single value.

That's why the first thing the kernel code does is to save all the register values to the memory. if we take a look at the interrupt entry code that is running right now on my computer, we can see

that it actually saves all the register values one by one to the memory address next to the stack pointer. The exception here is the program counter. When the interrupt arrives the execution

immediately jumps to the kernel code. Which means that the program counter is already pointing to that kernel code. Didn't we lose the userspace program counter? Yes :( The hardware knows about

that problem. And during the jump to the kernel the CPU automatically saves the userspace program counter to a separate register. so we don't lose program counter. The kernel entry code

very carefully saves all the registers into the memory. When the interrupt related job is done, the kernel wants to resume our process execution. Which just means that the kernel needs to load all

the saved register values back from memory to the registers. Including the saved program counter. At that exact moment when the program counter is restored, the CPU continues the user process.

Wait a minute... Isn't it a perfect moment to replace the real program counter with the foo function address? Yes, it is. If we do that the CPU will continue the execution not from the

interrupted instruction, but it will jump back, to the foo function. Exactly what we wanted! Now it becomes clear, what does it mean to receive a signal. Let's take a look at the big picture. Our

program informs the kernel that it is ready to get the SIGUSR1 signal. And when the SIGUSR1 arrives, we want the foo function to be executed. We can register such a handler with a signal system call.

Now another process wants to send us a signal. Hey kernel, I want to send SIGUSR1 signal to that process. This could be done with a kill system call. Don't be confused, kill just sends a signal

and doesn't actually kill anything. The kernel knows that our process has already registered a signal handler for SIGUSR1. "Aha!", thinks the kernel. Now I will mark the busyloop process

with a signal pending flag. And the next time this process jumps to the kernel code, I will replace its program counter right before resuming to make it point to the signal handler. So,

we can't really predict, when the signal handler will be executed. Even though another process has already sent a signal, the handler will not be executed until the next interrupt, exception or

a system call. Now to the fun part. Let's trace the whole mechanism in the kernel code for arm64. We will try to find the implementation of that part. When the interrupt arrives, how does the

CPU know where to jump? It works like this the CPU has a special register. It says "Dear operating system. Put the address of your interrupt handling code into that register; and when the interrupt

arrives, I will jump directly there." In arm64 such register is called "vbar_el1". "Holds the vector base address for any exception that is taken to el1." El1 means kernel space. Let's

try to find it in the kernel code. Looking for vbar_el1 in the kernel source tree.. Hmm.. This line looks promising. So let's open head.S. Some address called "vectors" is put to the vbar_el1.

Let's try to find "vectors". Too many results... Adjusting the search line a little... Entry. s file. Code start, code_end. Let's take a look. We found it! All the entries are here! Interrupts.

Exceptions and system calls. So basically all of them. This is the code for kernel_ventry macro. It is not so easy to read. We have a big bunch of ifdef's and other macros. To help ourselves

to get through this macros jungle, we can try try to look at the real machine code, that is actually compiled and all the macros are unwrapped. Right now I have a running arm64 kernel on my computer.

Let's disassemble the kernel binary and try to find interrupt entry. Okay, it doesn't have any symbols left. So we need a system map file to navigate. And we immediately see vectors. 11,000..

Let's find it in the binary. Apparently all the entries are right here: the first one, the second one and so on. We are interested in that one. Irq means interrupt and zero means that when

it had happened we were executing user code and not kernel code. So this is exactly our interrupt. Let's find it in the binary. We've found it! This instruction is exactly what we were looking for.

When the interrupt arrives the program counter jumps directly to that particular instruction. Very good! We have found the interrupt entry! This macro monster unwraps into these three

instructions and the first instruction just jumps further, to 1148c, skipping the other two. I have no idea what is that. The next instruction is actually very interesting. A few minutes ago,

we discussed that the first kernel job is to save all the user registers to memory. Until that moment the kernel can't execute any instruction that can modify any of the registers.

So the kernel wants to save registers. That instruction moves the stack pointer to allocate a bit of stack memory to save registers. In the source code we see that it was moved by pt_regs

size. struct pt_regs is the core element here. It keeps all the saved registers and we will see that struct a lot. And when we replace the user program counter with the signal handler address,

we will replace it in this struct. Okay the space is allocated and we can move forward. This is a tricky check for stack overflow which is not so interesting and after that we jump to 11e3c.

which is called el0t_64_irq. Let's look at it. STP stands for "store pair". So here we are finally saving all the registers to the memory next to the stack pointer. x0, x1, x2, x3 and so on. A bit

later it saves the program counter. Remember what we discussed: when the CPU jumps to the interrupt entry code, the hardware saves the interrupted program counter to a separate register because

we don't want to lose it. In arm64 this register is called elr_el1. Here we load elr_el1 to x22 and store x22 to the pt_regs. This way the user program counter is saved and this is exactly the

place in memory where we will replace the return program counter with our signal handler address. Very good, this important part has been found as well. This function does a few more things,

and in the end it finally jumps to the function.. That one: c3dd0. Let's find its name in the system map. The good thing is that in ASM code we put a pointer to saved registers into x0 register.

Which is the first argument for any C function. So this C function successfully gets the pointer to saved pt_regs. A few more jumps and we are in the code that actually handles the interrupt. When

the interrupt job is done, the kernel wants to resume the process. exit_to_usermode function, one more jump and we found the place where the kernel checks if we have any signals arrived.

Do_notify_resume checks the sig_pending flag which was set by another process while sending us a signal. If the flag is set, we go to do_signal function. Next we go to handle_signal,

setup_rt_frame, setup_return.. And here we replace the stored program counter with a signal handler address. It was the last piece of our puzzle. Now all the functions return, and we eventually find

ourselves here. The only job left is to resume the user process. The next instruction is a branch to the address 12a60. Let's find its name. Its name is return_to_user. Let's find it in the binary.

It does a little bit of kernel magic, and in the end restores all the registers. But the program counter is not yet restored it is done in several steps we load the saved program counter from stack

memory to x21. After that we move it from x21 to elr_el1. And the last instruction, "eret", takes that elr_el1 value and jumps there. We need a dedicated instruction to jump back to the user

code because we need a single instruction that will both jumps and changes a privilege level back from kernel to user. eret does both. And since we have replaced the saved program counter with

the foo function address we will jump to it, to our signal handler. And when the signal handler is finished we... wait.. We want to continue the main code but we can't jump there. We have lost

its address and all the register values the CPU had while it was executing the main code, are now completely changed by the signal handler code! We broke the program! What could work is if after the

signal handler we could jump back to the kernel. The kernel can restore the original program counter and all the other registers. Therefore we will be able to continue the main code execution.

And we actually have to do something like that. Let's improve our signal delivery mechanism. But first we need to understand how the stack pointer behaves when we enter kernel space. Our process,

as any other user process, uses stack. It keeps their local variables and other things. If it faces some instruction like this it will shift the stack pointer by 100 bytes. Nothing special. But

when the interrupt arrives the first instruction of the kernel code tries to move the stack pointer. Is it the same stack pointer? No. If the kernel code would use the same stack pointer, some

kernel local variables could be exposed to the user space stack which is not so secure. We don't want to expose any kernel information to the user space memory. If we have any instruction involving

the stack pointer it behaves differently depending on whether the CPU is executing userspace code or kenrelspace code. If the CPU is in user space mode, any reference to stack pointer actually

leads to sp_el0 internal register. Which points to the user space stack.If the CPU is in kernel mode, any reference to stack pointer actually lead to sp_el1 internal register. Which points to

another stack - kernel stack. It works the same as user stack but it lives in kernel memory, which cannot be accessed from the user space. So the first instruction

in the interrupt entry doesn't move the user space stack pointer. It moves a kernel stack pointer. And the user stack pointer is also a part of saved pt_regs. The kernel reads sp_el0 and puts

it into pt_regs. Now we go further. Handle the interrupt and want to resume the process. But we find out that we have a signal arrived. Now we will replace the saved program counter inside the

saved pt_regs. It means that we are overwriting the original program counter in pt_regs and losing it. But we will need it. When the signal handler is finished, we want to continue the execution

of the original code. And not only the program counter. We need to restore all the original registers because the CPU registers will be corrupted by the signal handler code. So we need

to keep the original registers. But where? When we leave the kernel and jump to the signal handler, the kernel stack has to be empty. Why is that? Let's pretend that we have some leftovers when we

return from the kernel to the signal handler. But it could be that while we were executing a signal handler, we receive another interrupt. We jump to the kernel code, allocate more space in the

kernel stack and at the very end we find out that we have another signal arrived. A nested signal. From the kernel we will return to another signal handler with one more leftover in the stack. The

kernel stack is limited in size. If we get too many nested signals, the kernel stack will be overflowed. Which is very bad. But user stack is not so strictly limited. A very convenient place

for us to save the original registers especially if we think about nested signals is the user stack . Look: set_up_rt_frame, get_sigframe, we get user stack pointer from pt_regs. We

shift it by the size of the saved pt_regs. We actually shifted a bit more to save some extra stuff but it is not so interesting. And we save the pointer to allocated space into user->sigframe

variable. Now we return back go to setup_sigframe. sf pointer points exactly to the same place and here we copy all the pt_regs from kernel stack into that user stack area. When everything is

saved, we do a program counter replacement trick and jump to the signal handler. And signal handler code uses the stack above the saved original registers. When the signal Handler finishes its

job we want to restore original registers and jump back to the main code. The kernel is the only one who can perform such a trick. But how to trick the signal handler and make it jump to the kernel in

the end? Hmm, its last instruction is "ret". "Ret" instruction jumps back to the address specified in the link register which is x30. So we can replace this register with a fake value which will lead

the CPU to return to our well prepared code. Sometimes such a code is called a trampoline. What kind of instructions do we need there. The only way to intentionally jump to the kernel code

is to invoke a system call. Wo we basically need only two instructions: put the system call number and execute SVC instruction which is arm64 system call instruction. And we have a bunch of "DO NOT

TOUCH IT" comments for better maintainability. Trampoline code lives in the kernel source and the kernel maps it to the user space memory of each process. So, when the signal handler is finished,

"ret" instruction will lead the CPU to the trampoline. Trampoline code invokes a system call and at this point we jump to the kernel entry again. So we walk again through the whole

procedure of saving current registers to kernel stack pt_regs and we find ourselves in the system call handler. The system call we invoked is called "sigreturn". It's main purpose is to copy original

registers back from user stack to pt_regs. So when the kernel resumes the user process, it will restore the original registers from pt_regs, which belong to the original code, and the CPU continues

the original code as if nothing happened. And the last part for those who are still watching. I have another video about the interesting compiler bug. Compiler generated a code reading the local

variable beyond the stack pointer. Now we see why is it incorrect even in user space code. A signal can arrive at any moment. And kernel will save the original registers directly

above the stack pointer if user code keeps a local variable beyond the stack pointer, the kernel will override it while saving original registers or executing the signal handler.