PSJailbreak Payload Reverse Engineering
1.it gets control at exploit_entry, which copies the rest of the payload to the fixed address 0x8000000000700000 and jumps to exploit_main.
2.exploit_main copies a resident part of the payload to another location, creates virutal usb device driver called "mod" with 3 functions, hooks some vsh functions via toc entry and does some permanent in-ram patching. when the work is done it zeroes itself out.
3.the resident part has basically 3 purposes: it manages virtual usb device, it does some on-the-fly patching and it hooks all the game disk file accesses from the vsh.
1.the virtual usb device is needed to make sure the original ps3jb device in plugged in. once the correct device is plugged (the one with the AAAAC0DE) device driver initializes the variable INITIALIZED to 1 (see kmod_func1 - probably "identify device", and kmod_func2 - "initialize device"). if one pluggs the device out, the function kmod_func3_call_panic "term device" is called which causes a kernel panic. all the virtual usb device code can be removed completely from the open psjb implementation since it's just a way of protection for the original ps3jb.
2.the on-the-fly patching part of the code is probably called on virtual memory page remapping and does additional patching in-place. it identifies if the pages requires patching byt calculating it's "hash" and comparing to the table entries. one of the patches enables developer menu/settings called "category_game_tool2.xml#root" which probably enables support of the pkgs and other dev stuff.
3.the hooks from the vsh are intended to redirect all on-bdvd file requests (or probably just "open") from vsh to the hdd saved backup. the launcher saves the base directory of the game started and after that all the file names are prepended with it. that's how the backup feature works. the lv1 still needs bdvd auth to launch the game, so the original disc in bdvd is still required.
4.Adds a Syscall (Syscall 36) which will be called by Backup Loader to activate the virtual bluray drive with the correct backed upped disk. 5.Patches the return value from hypercall 99 so that we can launch unsigned apps.
the code below is from my idb of the payload.