Confirm we are properly handling removal of signed keys

[ryan-norton]

If a user passes a signed key to ssh-add -d do we properly remove it, or only if the unsigned copy is passed in? We should confirm and ensure we are doing the right things here.