Debug Qualcomm TEE and secure devices

TEE logs aren’t enabled in the current release.

Debug provides a set of common logging and debugging techniques to troubleshoot issues in Qualcomm TEE, trusted and client applications, and secure devices. Important Run all the SSH commands in the SELinux Permissive mode. The Enforcing mode will be supported in the future. For instructions on how to connect to the device, see Qualcomm Linux Build Guide.

Debug Qualcomm TEE

Qualcomm TEE kernel logs, also known as the TrustZone diag log, can be used to debug errors that occur in Qualcomm TEE. The TrustZone diag log is available in the Linux kernel driver, which redirects the logs.

Connect to the device as the root using SSH.
Capture the TrustZone logs using the following command:
cat /proc/tzdbg/log > tzbsp_log.txt

The error codes in tzbsp_log.txt are encoded in hexadecimal. You can run the following tool to decode tzbsp_log.txt from hexadecimal to string.

Go to <TZ.XF.X.X path>/trustzone_images/ssg/bsp/tz/build/tz/A53_64/<BuildFlavor>

Run the following commands using python 3.

python3 print_tz_log.py -l tzbsp_log.txt -e errorCodesDict.txt -t <TZ.XF.X.X path> -o tzbsp_log_decode.txt

For example:

python3 print_tz_log.py -l tzbsp_log.txt -e errorCodesDict.txt -t //crmhyd/nsid-hyd-05/TZ.XF.5.0-07927-KODIAKAAAAANAAZT-1 -o tzbsp_log_decode.txt

For device log collection, the TrustZone diag log buffer is part of the RAM dump, which can be parsed using qsee.elf from TZ.XF software in the crash dump parser tool. For offline or off-device log collection, the TrustZone diag log buffer is part of the RAM dump, which can be parsed using qsee.elf (trustzone\_images/ssg/bsp/qsee/build/\${tz\_bid:EACAANAA}) from the TZ.XF software in the crash dump parser tool.

Debug using secure crash dump

You can debug Qualcomm TEE using the RAM dump. The execution region dump of Qualcomm TEE is collected using secure crash dumps. Devices that trigger the fuse with stage 2 sec.elf are known as secure boot-enabled devices. To debug on these devices, see SecTools v2: Secure Debug User Guide. Note The SecTools guides are available to licensed developers with authorized access.

Debug trusted and client applications

The trusted application logs, also known as Qualcomm TEE logs, are used to debug the errors in trusted applications. To debug errors in the client application, the kernel and journalctl logs are used. For online or on-device log collection, Linux collects the Qualcomm TEE/kernel logs at runtime. You can connect to the device using SSH and use the following commands:

To collect the Qualcomm TEE logs from Linux:
cat /proc/tzdbg/qsee_log > qsee_log.txt
For client applications, to collect the kernel and logcat logs:
cat /dev/kmsg > kernel_log.txt journalctl > journalctl.txt
For offline or off-device log collection, the Qualcomm TEE log is available in RAM dumps along with the kernel and journalctl logs.

Debug on secure devices

As part of the secure boot procedure, blowing debug disable fuses disable debugging capabilities on the devices. This includes RAM dumps, INV, and NINV debug on the subsystems. The debug policy feature allows control over the debug capability for a device enabled with secure boot. The debug policy image allows debug capabilities such as JTAG re-enable (INV debug), RAM dump, and TrustZone logging (NINV debug) on commercial secure devices. For security reasons, the serial number of the device controls the debug policy for secure RAM dumps, Qualcomm TEE logs, and JTAG. Enabling JTAG on the Qualcomm TEE subsystem disables the device security with respect to hardware key generation. As a result, existing secure storage like user data, SFS, and RPMB becomes inaccessible. Sometimes, the device may prompt for a factory data reset. Use the following command to debug on secure devices:

<meta>/common/sectoolsv2/ext/linux/sectools secure-debug --security-profile <meta>/common/sectoolsv2/<chipset>_security_profile.xml --generate --outfile apdp_out.mbn --all-flags --sign --signing-mode LOCAL --oem-id=0x1 --root-certificate=./RSA-OEM-KEYS/qpsa_rootca.cer  --ca-certificate=./RSA-OEM-KEYS/qpsa_attestca.cer --ca-key=./RSA-OEM-KEYS/qpsa_attestca.key --oem-product-id=0xabcd --serial-number=0xabcdabcd

Ensure that you configure the OEM_ID, PRODUCT_ID, serial number and keys, and certification paths appropriately. For more information, see SecTools v2: Secure Debug User Guide. Note The SecTools guides are available to licensed developers with authorized access.

Flash APDP on device

To flash APDP on the device, run the following command:

fastboot flash apdp_a <path to apdp.mbn>

Table : Debug policy flags for dump collection

Stage	Full dump	Mini dump
Stage	Applications (DCC and scan dump) aDSP/Video/RPM/ SLPI	Modem/Qualcomm TEE/Secure dump	TZDiag
Non‑secure	No debug policy needed	No debug policy needed	No debug policy needed
Stages 1 secure	No APDP image needed	No APDP image needed	No debug policy needed
Stages 2 secure	`--nonsecure-crash-dumps`	–offline-crash dumps with device serial number. See below for Available minidumps	See below for TZDiag flags and encryption configuration

TZDiag flags and encryption configuration

QCS6490/QCS5430: “–logs” with device serial number QCS9075: “–tz-diag-logs” with device serial number Or Encrypted TZDiag with --nonsecure-crash-dumps + TZDiag encryption public key/exp in devcfg can be configured in the following location:

/trustzone_images/ssg/securemsm/trustzone/qsee/mink/oem/config/<chipset>/oem_config.xml

Available minidumps

Apps minidump: --apps-encrypted-mini-dumps
Modem: --mpss-encrypted-mini-dumps
WLAN: --wlan-encrypted-mini-dumps
aDSP minidump: --adsp-encrypted-mini-dumps
cDSP minidump: --cdsp-encrypted-mini-dumps

See KBA-191202045020-1 (ZIP). For more information, see MiniDump Software User Guide. Note The SecTools and MiniDump guides are available to licensed user with authorized access.

Qualcomm TEE/TrustZone diag log collection on secure device

On the secure device, the Qualcomm TEE/TrustZone log that’s collected from Linux is disabled by default. Qualcomm provides an encrypted log feature for logging. Follow these steps for enabling this feature:

Generate an RSA key for encryption using:
openssl genrsa -out rsa_key 2048

Show RSA key information and modulus using:

openssl rsa -in rsa_key -text
openssl rsa -in rsa_key -modulus
Private-key: (2048 bit)
modulus: 00:a0:48:99:99:83:26:65:57:fc:75:52:25:45:53:
92:fc:27:29:cb:14:35:94:7c:89:bc:d4:0a:c6:3d:
0d:6d:8a:7d:72:1d:e3:4f:f0:32:66:41:a9:f6:c1:
2f:79:aa:58:ea:57:3b:29:6d:cf:40:33:4e:ad:ec:
bf:78:44:4b:28:52:c8:e3:6e:77:01:e5:a3:c6:25:
65:8c:8b:cc:32:20:2d:29:58:03:f0:d5:b7:f4:c0:
d6:09:b2:8e:59:c1:3c:ac:e5:61:04:36:78:e3:da:
95:b3:e3:b7:71:90:50:ee:a9:70:5a:15:1a:af:d9:
a5:4f:c2:70:f1:f8:f1:67:d1:78:0e:b8:95:6e:93:
73:6a:23:f1:31:e1:e2:49:ff:18:54:a3:73:d0:70:
91:de:7a:92:53:11:aa:cb:b0:f9:d0:e1:83:9f:74:
67:bc:1a:89:6d:b1:d2:de:4f:ab:3c:1c:63:c9:bc:
75:f0:c0:80:fc:db:73:d1:8a:e3:f4:60:57:dd:66:
f1:3a:fa:18:ed:7f:47:72:3e:49:50:94:8e:19:ae:
6b:69:62:3d:74:ca:44:fb:d4:1c:1d:59:43:30:31:
0d:fb:ab:70:44:9d:d9:d0:ce:cb:43:f3:2a:98:a4:
83:e7:76:ae:a8:b8:ea:63:64:e1:11:1b:99:92:b3: 9b:3f
publicExponent: 65537 (0x10001)

Note The modulus is used in the pub_mod in oem_config.xml file. The pub_exp exponent is usually 65537. 0x10001 is known as the publicExponent.

Set the RSA public key (exponent and modulus) in the trustzone_images/ssg/securemsm/trustzone/qsee/mink/oem/config/<chipset>/ oem_config.xml file. Enable this feature by adding the following lines to the oem_config.xml file using:

<driver name="NULL">
<global_def>
<var_seq name="pub_mod" type=DALPROP_DATA_TYPE_STRING>
a048999983266557fc755225455392fc2729cb1435947c89bcd40ac63d0d6d
8a7d721de34ff0326641a9f6c12f79aa58ea573b296dcf40334eadecbf7844
4b2852c8e36e7701e5a3c625658c8bcc32202d295803f0d5b7f4c0d609b28e
59c13cace561043678e3da95b3e3b7719050eea9705a151aafd9a54fc270f1
f8f167d1780eb8956e93736a23f131e1e249ff1854a373d07091de7a925311
aacbb0f9d0e1839f7467bc1a896db1d2de4fab3c1c63c9bc75f0c080fcdb73
d18ae3f46057dd66f13afa18ed7f47723e4950948e19ae6b69623d74ca44fb
d41c1d594330310dfbab70449dd9d0cecb43f32a98a483e776aea8b8ea6364e1111b9992b39b3f
</var_seq>
<var_seq name="pub_exp" type=DALPROP_DATA_TYPE_STRING>
000000000000000000000000000000000000000000000000000000000000000
000000000000000000000000000000000000000000000000000000000000000
000000000000000000000000000000000000000000000000000000000000000
000000000000000000000000000000000000000000000000000000000000010001
</var_seq>
</global_def>

Note When the public key in the oem_config.xml file is updated, ensure that there are no new line characters, tabs, or spaces inserted between due to the Notepad or Wordpad editors.

Enable the encryption feature configuration flag from the trustzone_images/ssg/securemsm/trustzone/qsee/mink/oem/config/<chipset>/oem_config.xml file, using:
< props name="OEM_log_encr_enable" type=DALPROP_ATTR_TYPE_UINT32> 1 </props>
To build the TrustZone devcfg image, enter the OEM_ID field value and sign the devcfg.mbn image.
Flash the signed devcfg.mbn image using:
fastboot flash devcfg_a devcfg.mbn

Note Use devcfg.mbn for QCS6490 and devcfg_iot.mbn for QCS9100.

Collect the Qualcomm TEE/TrustZone log using:

cat /proc/tzdbg/qsee_log > qsee_log.txt
cat /proc/tzdbg/log > tz_log.txt

Qualcomm TEE/TrustZone diag log decryption steps

Download the Python decryption tool decrypt_tzdiag_qsee_log_tools.py from KBA-200917004544-1 (ZIP).

To install, run the following commands:

Python Version 3.x
pip install pycryptodome
pip install cryptography

To decrypt, run the following command:

python decrypt_tzdiag_qsee_log_tools.py -pk <RSA private key file> -a RSA -I <input encrypted qsee/tz diag log collected from device> -o <decrypted qsee/tzdiag log filename>

After successful decryption:
1. Navigate the plain text of the Qualcomm TEE log to a readable string format.
2. Convert the hexadecimal encoded error codes to string, using:
  print_tz_log.py

Next steps

To learn how to develop and run trusted and client applications, see Develop trusted and client applications.
To configure Qualcomm TEE for securing devices that handle sensitive data and run trusted applications, see Configure security services.
To customize memory and SEPolicy, see Customize security services.

Introduction

Architecture

Key Documents

Technologies

System

Release Notes

Debug Qualcomm TEE

Debug using secure crash dump

Debug trusted and client applications

Debug on secure devices

Flash APDP on device

TZDiag flags and encryption configuration

Available minidumps

Qualcomm TEE/TrustZone diag log collection on secure device

Qualcomm TEE/TrustZone diag log decryption steps

Next steps