Packages

PKG files for PSVita are essentially the same structure as PS3 and PSP game PKG files with different AES, RSA and ECC keys.

Keys
See Keys.

The overall algorithm is the same as PSP with one exception.

PS3 / PSP
On the PS3/PSP, the file table and the data is AES CTR encrypted using the 0x10 bytes from offset 0x70 in the file as the IV along with the PS3/PSP AES Key.

PSVita / PSM
On the PSVita for type 2, 3 and 4, they first encrypt the 0x10 bytes at 0x70 to create a new AES session key, then do the normal AES_CTR with 0x70 as the IV and the new session key as the AES CTR key.

PKG key id
At offset 0xE7 in the PKG file is a version byte. This is only in the PSP type PKG files inside the .ext section. If it is type 1, then use the PSP key as normal, if it is type 2, use the PSV2 key with the new session key step, and if its type 3, use the PSV3, etc.

Installing
Package Installer uses SceNpDrm to decrypt/extract the game package into a temporary directory. Then it calls ScePromoterUtil NID 0x86641BC6 on 1.69 and NID 0x716C81F4 on 1.80+ to "promote" (aka install) the files into the right directory in ux0. ScePromoterUtil is just a wrapper for a SceShellSvc call, so SceShell actually moves the files, then adds the LiveArea bubble and updates app.db accordingly. In theory you can use SceNpDrm and ScePromoterUtil separately and package your games in an alternative format.