Packages
PKG files for PSVita are essentially the same structure as PS3 and PSP game PKG files with different AES, RSA and ECC keys.
Algorithm
The overall algorithm is the same as PSP with one exception.
Keys
Some keys are common with PSP and PS3 depending on the type of package.
PS3 / PSP
On the PS3/PSP, the file table and the data is AES CTR encrypted using the 0x10 bytes from offset 0x70 in the file as the IV along with the PS3/PSP AES Key.
PSVita / PSM
On the PSVita for type 2, 3 and 4, they first encrypt the 0x10 bytes at 0x70 to create a new AES session key, then do the normal AES_CTR with 0x70 as the IV and the new session key as the AES CTR key.
PKG key id
At offset 0xE7 in the PKG file is a version byte. This is only in the PSP type PKG files inside the .ext section. If it is type 1, then use the PSP key as normal, if it is type 2, use the PSV2 key with the new session key step, and if its type 3, use the PSV3, etc.
Installing
Package Installer uses SceNpDrm to decrypt/extract the game package into a temporary directory. Then it calls ScePromoterUtil#scePromoterUtilityPromotePkgWithRif on FW 1.69 and ScePromoterUtil#scePromoterUtilityPromotePkg on FWs 1.80+ to "promote" (i.e. install) the files into the right directory in ux0. ScePromoterUtil is just a wrapper for a SceShellSvc call, so SceShell actually moves the files, then adds the LiveArea bubble and updates app.db accordingly. After getting usermode execution, you can use SceNpDrm and ScePromoterUtil separately and package your games in an alternative format, for example .vpk.