ENCDEC Device Reverse Engineering

Introduction

• ENCDEC is the encoder/decoder, integrated inside South Bridge
• The following information was reverse engineered from LV1, Storage Manager in LPAR1, sb_iso_spu_module.self and sv_iso_spu_module.self.

Linux Driver ps3encdec

• I'm using this driver to set/clear my ATA and VFLASH keys.
• Tested on Linux 3.5.1.
• You can send all supported ENCDEC commands with this driver.

Interesting Facts

• HDD sectors arrive in LV1 with tweak values already XORed.
• But VFLASH sectors are first encrypted/decrypted with ENCDEC keys and the fact is that VFLASH sectors are NOT already XORed with tweak values. LV1 does pre- and post-XORing with tweak values.
• LV1 allocates a DMA region where it stores the sector number for each requested sector and ENCDEC encryptes these tweak values.
• After that, LV1 XORes encrypted tweak values.
• See encdec_device_enqueue_decsec_request and EdecXTS_XorWithMask in LV1.
• I patched my LV1 for testing and killed XORing with encrypted tweak values. After that VFLASH sectors were encrypted/decrypted without tweak values, only with XTS data key.

Establish Secure Communication Channel

• First host and ENCDEC device exchange random numbers.
• From the exchanged random numbers host and ENCDEC device compute the session key.
• ENCDEC commands, e.g. to set ATA keys, are encrypted with the session key and AES-CBC-192.
• Before a secure communication channel is established, host and ENCDEC device use static AES-CBC-192 keys to encrypt communication data. The static keys can be found e.g. in sb_iso_spu_module.self or sv_iso_spu_module.self.
• Static ENCDEC keys depend on SB bus version. To get your SB bus version, read v2 of repository node SB bus id.

SB bus id from 3.15:

....bus. 00 00 00 00 62 75 73 01
id...... 69 64 00 00 00 00 00 00
........ 00 00 00 00 00 00 00 00
........ 00 00 00 00 00 00 00 00
........ 00 00 00 00 00 00 00 01
........ 00 00 00 00 03 00 01 03        <-------- SB bus id v2

# Dumping it with ps3lv1call-tools

~/ps3lv1call-tools$ sudo ./ps3lv1call 91 1 0x0000000062757301 0x6964000000000000 0 0
0000000000000001 0000000004000103

• During the communication, host and ENCDEC device use random IVs which are sent unencrypted together with encrypted payload.
• The ENCDEC commands, which are encrypted with the session key, contain magic 24 bytes which are checked by ENCDEC device and if some bits are not correct then the command is denied. The magic bytes can be found in sb_iso_spu_module.self too.
• The format of ENCDEC command to set ATA keys is slightly different from the ENCDEC command to set ENCDEC keys.

ENCDEC Commands

KGEN1 (0x81)

• Used by host to send host random to ENCDEC device and receive ENCDEC random from ENCDEC device.
• Sent data is encrypted with host static key and received data is encrypted with ENCDEC static key.

KGEN2 (0x82)

• Used by host to send host and ENCDEC randoms to ENCDEC device.
• Sent data is encrypted with host static key.

KSET (0x83)

• Used by host to send ENCDEC command to ENCDEC device.
• Sent data is encrypted with the session key.
• It contains magic 24 bytes from sb_iso_spu_module.self.

KGEN_FLASH (0x84)

• The command resets ENCDEC device state.
• LV1 calls this command "EdecKgenFlash", probably correct name is "EdecKgenFlush" :)

SB_CLEAR (0x87)

• No clue what it does but it is sent by Storage Manager after ATA keys are cleared.

Linux Application

• Here the Linux application i use with my ps3encdec device driver to set/clear ATA/ENCDEC keys on my PS3.

See my GIT repo: http://gitorious.ps3dev.net/ps3linux/encdec