Integrating git and ssh into KDE Plasma with terminals such as Konsole is not the most straightforward process. It certainly doesn't work out of the box like with Gnome or Mate.

If it's acceptable to always commit keys to the agent edit or create ~/.ssh/config and write:

AddKeysToAgent yes

There might be the need to also customize the ssh-askpass command. A general rule of thumb here (so that it uses whatever is stored within $SSH_ASKPASS for example) is for it to run without an attached screen. A git related example is:

setsid -w git fetch

# KDE

Gnome-like git integration in KDE Plasma