Making SSH and SCP work for automated deployments via in GitLab or any other CI/CD system

So you want to copy your build artifacts to some server but ssh complains about host key verification and missing credentials?

This is really about automated passing of credentials and host key verification.

User and Password: use public keys

On the server, create a new key pair. It’s pretty important that you don’t use your personal key pair since we’re going to share the private key with GitLab. Don’t do that with your personal private key — it is really your identity file.

ssh-keygen -f id_rsa.yourserver

Make sure you leave all passphrases empty. Just hit return when asked to enter a phrase.

This will create a id_rsa.yourserver and id_rsa.yourserver.pub. That’s your fancy key pair. Now add the public key id_rsa.yourserver.pub to you server’s authorized keys file:

cat id_rsa.yourserver >> ~/.ssh/authorized_keys

Inspect the new private key via cat id_rsa.yourserver

 — — -BEGIN RSA PRIVATE KEY — — -
MIIEpAIBAAKCAQEAufBB3Xm52WYtSxjZKpSInqoEejk1ZP7QOzYvdOzxpOvZVPwF
V2WriNDd6mCex6d4ULxZnxswb8ZjXnCiUDVtnHADA6AH6y3UvMUqfzxs5QElgQKG
NlI/ . . I REMOVED TWENTY LINES FOR BREVITY . . . . . . . . .HzG
zhbJE9Kv41bpADRMG+HmGU9CpHbN4JkY1gNKJBxrrxTkeR5zkBGv7exSTM5+9Bun
pxX5cmczky2/BBovOur3soBM4QkQPwl26E9k0WLrT88YjhO1z0mMJmfDPCgZgh87
iPuAmQKBgQCIAk1QXQC+ . AND THIS KEY IS NOT A REAL KEY . . aIaUC
lwaj0yhdJPTWsHqWc0Rhhbuk7oEjn8NriyduPLU/0d+iA9bQx9ETi+kCF+A3sYY+
qIUq8j28hIDBJGUZldQxhhHIXwICG47/GzLEw7zop0szez++gOCZTw==
 — — -END RSA PRIVATE KEY — — -

and add the contents that should look something like above (just way bigger) to your GitLab CI/CD variables. It’s within your project under settings, then CD/CD. Expand the Variables section and set

• type: File
• Key: IDENTITY
• Value: the private key above (your key, not the gibberish above)

Then in your .gitlab-ci.yml you can reference that identity file as $IDENTITY. For example as parameter to ssh

ssh -i $IDENTITY you@yourserver.com "systemctl restart your.service"

Unfortunately, only. This is not enough:

Host key verification

ssh wants to make sure that you are connecting to the right server, so the server will present a fingerprint of its host key. Only if that key is known to ssh it will let you connect to that server

On your developer machine ssh will show you the fingerprint on screen and ask you if you really want to connect to that server. You will have to type yes usually. Then, the fingerprint will be added to your ~/.ssh/known_hosts file so ssh doesn’t need to ask again for that server.

Unfortunately, typing isn’t an option in automated scenarios like your CI/CD runner environments. You can either

• disable host key verification (for security reasons, don’t!)
• set upSSHFP records in your DNS
• or simply ship a known_hosts file in your repository

Crafting the known_hosts file

The simplest solution is to inspect the verbose (the -v option) output of ssh -v you@some_server.comwhen connecting to the server. Look for

debug1: Server host key: ecdsa-sha2-nistp256 SHA256:4F7nGBwJqx97Er3XXXXXXXXX/sS4MGYNvrwYYRqBqL8
debug1: Host ‘10.0.2.100’ is known and matches the ECDSA host key.
debug1: Found key in /home/mart/.ssh/known_hosts:2

Yay, this known_hosts:2 from the last line above tells you that you want to copy line number 2. Extract that to a file and name it conveniently.

sed -n ‘2,+1p’ ~/.ssh/known_hosts > known_hosts

Now, check if that works:

ssh -v -o "UserKnownHostsFile=./known_hosts" you@some_server.com

If you can login, great! Add that known_hosts file to your repo and you can then copy stuff in GitLab’s CI/CD runner.

scp \
  -o "UserKnownHostsFile=./known_hosts" \
   -i $IDENTITY some_build \
  you@some_server.com:/some/target

And it works!

Final words

There’s much more to be automated here :-) And yes, this is more of a general documentation for myself on how to do stuff. Because once it runs I usually don’t touch it that often and then forget about how I managed to get it to work in the first place.