The normal syscall table is allocated at the start of SceSysmem with enough space for theoretical maximum of
3840 syscalls. Because of randomization, many slots are unoccupied and will not be used. Normal syscalls are numbered from
0xFFF. "Fast" syscalls (run with interrupts disabled) are reserved in slots
0xFF. However, as of 1.69, there has been no observed usage of fast syscalls.
Syscall slots are assigned to modules when they are started (modules loaded before SceKernelModulemgr have their slots assigned after the module manager starts). However, the syscall table itself is not written to until the first application/library imports this function. After the table is written to, however, a syscall is not erased from the table until after the module is unloaded.
On startup, the syscall table is allocated and every slot points to a placeholder function that returns
SCE_KERNEL_ERROR_NOSYS. When an entry is added, the function pointer is overwritten with the exporting function. When an entry is removed, the placeholder function replaces the slot. This way, upon syscalls, the kernel does not need to check if the slot is a valid function.
There is a global counter keeping track of the "top" of the list of syscall slots. When a new module is started (after it is loaded), this counter is first incremented for a random value between 1 and 5 (unless incrementing it will overflow). Next, for each exporting entry, the syscall number is assigned by looking at the slot pointed to by the "top" pointer. If it points to an allocated slot, it will increment (with wraparound) until it finds an unallocated slot. If all slots are allocated, it returns error
SCE_KERNEL_ERROR_MODULEMGR_SYSCALL_REG. Once it finds an unallocated slot, it writes the slot number to a buffer in the module information structure in kernel memory and increments the top pointer.
When a user application/library imports a kernel syscall, the resolver looks to see if the slot assigned to that function is written to yet. If it is not, the kernel will fill the assigned slot with the exporting function. (This is done at the time the user library starts, not when it is loaded to memory).
When a library is unloaded, any module with exporting functions will have all the slots erased (replaced with the placeholder function). It knows what to erase based on the slot assignment information.
Because of the time between a slot is allocated and the time it is written to, there could be collisions. For example, assume there is one empty slot left. Two modules each exporting syscalls are loaded and both of them are assigned the final free slot. One user library is loaded that imports from the first module. Then it imports from the second module. At this point, the function pointer exported by the first module is replaced with the second one.
It is unlikely this would lead to any security vulnerabilities, but it could create system instability. However, if the system has 3000+ syscalls loaded, it may already be in an unstable state.
Syscall numbering depends on both the random increment per each module load and also the order modules are loaded. (The actual order of module loads is dependent on ASLR and later by what application the user decides to open first). Therefore, lower syscall numbers are more likely to be close upon each reboot. Because of the way syscall slots are allocated and the fact that it is unlikely that 3000+ slots are used, they will not be randomized within a module and are therefore arranged in NID order. Estimation of syscalls that are not imported by the current running app only requires knowledge of the relative position of that exported function and the location of any other syscall from that module. However, if a syscall is never imported by any running apps on the system, you cannot call it regardless if it is exported by the kernel.
To estimate syscalls, another useful way is to note the error code returned by syscalls. Many functions return specific error codes that other functions do not. In case of ambiguity, you can abuse the fact that syscalls next to each other do not change position, so some set of inputs to each syscall in order should return the same sequence of error codes. With good inputs, you can predict with high probability the syscalls for a certain module.