Initrd contains the systemd binary and enough libraries, services, and kernel modules to get booted this far. The system failed at switch root which is where the real root disk is mounted. Initrd can contain as much or as little as needed to get a working system which can be a lot of you are using a network filesystem as a root for instance.
I’n Windows it is not stored in a keyring but instead in the registry. This has basically the same security threat model as a local key file.
The ssh-agent on Linux will do what you want with effectively the same security. The biggest difference being that it doesn’t run as a system service but instead runs in userspace which can make it easier to dump memory. There are some other agent services out there with additional security options but they don’t change the threat model much.