Basic Bluray disc authentication procedure
Whenever you insert a Blu-Ray disc, the PlayStation 3 drive will look at a special area of the disc called the PIC Zone (the BD ROM Mark is actually used in movie discs but not in game unlike what I first thought).
The PIC Zone (which stands for Permanent Information & Control data Zone) contains information/data related to the disc's authentication which is done by the Blu-Ray drive. This area cannot easily be dumped (you'd pretty much need a Blu-Ray drive with a hacked firmware) and of course that specific area cannot be burned on any kind of discs or with any kind of burners commercially available.
There is no absolute certainty to what data the PIC Zone actually contains. I initially thought that Sony would use a public/private cryptography cipher to authenticate the discs but that is quite unlikely considering the limited resources of the drive controller. There isn't any kind of hard cryptographic layer on the discs as I first expected, so the security on the discs themselves is much less invasive as I initially thought. Yet the fact that the PIC Zone can't be rewitten without any kind of special equipment does its job well when it comes to preventing backups.
The authentication procedure itself is done through the use of a per Bluray Drive key pair. One being located on the drive controller itself while the other is stored AES256CBC encrypted in the EID2 area located on Flash. This key is also used while updating the Bluray Drive, which firmware's will be physically re-encrypted using that very same key and stored that way. As such you cannot swap a Bluray Drive controller board from one ps3 to another - see Remarry Bluray Drive.
This also means that physically dumping the drive's firmware would lead nowhere with it being stored in an encrypted form. The only way to get a plain version of it would be to dump the Bluray Drive controller's ram at runtime. Beside although I am not entirely sure about this, it is very unlikely that a command exists to read the firmware from the Bluray Drive, should it exist, the dumped binary would still be encrypted, thus connecting the Bluray Drive to a computer literally leads nowhere.
Finally once the authentication procedure is done, there is another protection which happens to be a per sector software cryptographic layer.
algo for masterdiscs
genelib.dll (from SDK generator tools) has the static keys for masterdiscs
you can also get it from sv_iso_spu_module
The cryptographic layer is the very same one used on PS3 master discs as on retail ones with the exception of the key being used. Masterdiscs are identified through their special masterdisc sectors located at offset 0x7000 (sector 14) on the disc.
The encryption itself is done in sector ranges rather than files, where the key for each sector is defined by the address of the said sector in correlation with an initial static key. On retail discs, there is a per disc key located at offset 0x800 on the disc with a header composed of "Playstation3" and the discs' title id such as "PlayStation3 BCES-00141". I assume that this key is in encrypted format and likely decrypted through lv2 by appldr.
The software cryptographic layer is done in such way that the disc sectors will be transparently decrypted so long as you are using proper syscalls (e.g. running a game or a playstation 3 application), this explains how easy it is to dump a disc's content whenever you can run PS3 code on top of lv2.
See also SELF File Format and Decryption
The EBOOT.BIN as well as other self and sprx binaries themselves aren't tied to the disc's encryption, however their metadata may contain specific flags (in fact the ones on game discs always do) that will prevent them from being loaded from anywhere other than an authenticated disc (masterdiscs != authenticated discs) This would explain why someone on a debug console for instance can't just grab the games' binaries, put them on a masterdisc and hope for the game to just play. Other flags are being involved such as a "no debug" flag that will pretty much prevent you from loading your binary into sony's debugger.
Because the binaries on the discs still have to be signed as they are verified and decrypted by appldr, in the event that you would somewhat trick the drive into thinking that your disc is a genuine PS3 disc, you could still not have your own fself ("fake" secure elf, complied with Sony's sdk) in there and get it to run, thus this would never lead to homebrews no matter what some clueless people may claim about it.
Old (very incomplete) source: http://lan.st/showthread.php?t=1722