Kernel
The Vita has a purely modular kernel. All components of the kernel are skprx files found in the os0 partition and are listed in Libraries.
Security
KASLR
Since version 1.80 or so, the kernel implements kernel address space layout randomization to discourage ROP attacks.
Canaries
Since version 1.80 or so, the kernel makes use of stack canaries to detect stack buffer overflows and halts the system when an overflow is detected.
Memory Domains
Memory domains is a feature in ARM MMU that provides an easy way of showing and hiding groups of addresses as well as their permissions. When a syscall is made, the handler disables all access to memory domains for user memory so kernel code cannot directly access user memory. This means if a user pointer is passed in and the kernel forgets to check it and dereferences it directly, it will abort. In order to access user memory, special functions are used that temporarily enables all domains and the access is implemented with the ARM unprivileged access instructions LDRT and STRT to make sure the access functions cannot read or write in kernel memory space. As long as the domain disable code in the syscall hander is secure and the user memory access functions are secure, there is no need for additional checks implemented per function. Additionally all non-code pages are marked as "execute never" in both kernel and userland.
Syscall Randomization
The numbers assigned to syscalls change on each boot but the delta between the same functions exported by the same module will stay consistent.
NID Poisoning
Since version 2.10, SceKernelModulemgr replace the NID entries in the module import tables with junk data. That means that you can no longer map syscall numbers to NIDs.
Usermode stack pivoting protection
Since unknown version (seen on 3.18) the kernel will terminate an application if it notices that its stack pointer register is not pointing into the stack memory.
User&kernel heap overflow protection
dlmalloc, used for heap allocations, is compiled with -DFOOTERS=1 to enable more heap overflow checks. Additionally, a custom SceNetPs malloc implementation also does some heap overflow checks on its own.
List of kernel libraries
For a list of all kernel libraries, check out Libraries.