The Molecule team was the first to hack the Vita and responsible for the majority of the reversing work done on the Vita.
Non Secure Kernel Bootloader - enso
The commonly spread definition of CFW in PSP and PS3 scene was to be reboot-proof. Molecule achieved this on PSVita by exploiting a vulnerability in NSKBL eMMC init code.
Plugins API - taiHEN
Davee wanted a powerful API for patching PSVita OS, making it more a Custom FirmWare.
On 29/07/2016 HENkaku was released: http://henkaku.xyz/
HENkaku enables homebrew by patching out signature checks (works similar to SceKernelModulemgr#Module_decryption_and_signature_checks). HENkaku uses two kernel vulnerabilities: Vulnerabilities#Heap_use-after-free_in_sceNetSyscallIoctl and Vulnerabilities#sceIoDevctl_does_not_clear_stack_buffer and a usermode webkit vulnerability. HENkaku only works on firmware 3.60, however the kernel vulnerabilities are present in all firmware versions up to and including 3.60.
On 14/06/2015, Rejuvenate, which was the first public exploit that allowed to run unsigned usermode code, was released. http://yifan.lu/2015/06/14/rejuvenate-native-homebrew-for-psvita/
It was no surprise that crypto processes were not handled by the kernel (such was the case for previous Sony consoles). Libraries that deal with encrypted/signed content (SELF loading, PUP unpacking, etc) all make calls to the Secure World. The hypothesis was that like many large manufacturers at the time, secure world was used for cryptography and security tasks. Getting access to the secure kernel was even harder than non-secure kernel because there was much less exposure and much less information. However, with a vulnerability that abused some lightly documented features of the ARM architecture, the secure kernel was dumped on 06/09/2014, a little less than a year after owning kernel. Unfortunately, almost immediately, the team found that secure kernel was a red herring. There was no keys or any sensitive information in secure world (Sony was wiser here than most other ARM device makers), but the sole task of the secure kernel was to communicate with an external processor which the team named F00D Processor because of the
e_machine field of the ELF headers.
For about a year research was focused on getting kernel code execution. Through some ingenuity and a lot of luck, on 27/08/2013 the first kernel exploit on the Vita was realized. The vulnerability was an integer overflow leading to a heap overflow and a misconfiguration that allowed a small portion of kernel heap memory to be leaked. The exploit and tools were completed on 01/09/2013 and for the first time, kernel memory was revealed. After a week of dumping the large kernel codebase (there were many factors that made it a slow process), work begin in parallel to reverse the system and find more vulnerabilities.
On 18/08/2012, a vulnerability was discovered in PSM that allowed both for memory to be dumped and code to be executed. UVLoader was developed and in a couple of weeks, the first working native code homebrew ran on 12/09/2012. Although the source for UVLoader was released in anticipation for excitement in the homebrew community, there was no serious response from developers. Unfortunately, Sony used the source for UVLoader to secure the system in later updates and make userland code loading a much harder reality.
In early 2012, the first ROP exploit was achieved through the Web Browser. Memory dumps of the browser were obtained through a disclosed WebKit vulnerability that was not patched because Sony did not use the most up-to-date WebKit version. The same vulnerability allowed allowed ROP code execution. ROPTool was written to make creation and testing of ROP payloads easy.
Molecule has done some work on PSP in the past. Initial reversing of the PSP Emulator was done by members of Molecule including the first flash0 dump that opened the door for all future PSP emulator hacks.