APT3 installs a backdoor on the compromised system by modifying the Windows Registry to ensure persistent access. This involves creating a registry key that launches the backdoor executable during system startup.