How to Set Up SSH Keys on Linux for Password-Free Authentication

Overview

SSH key authentication replaces your server password with a cryptographic key pair — one private key that stays on your local machine, and one public key that lives on your server. When you connect, your SSH client proves identity by signing a challenge with your private key. Your server verifies it using the public key. No password ever crosses the wire.

Most people set this up after getting a new VPS or dedicated server and getting tired of typing a password every time they connect. It’s also a hard requirement on many production setups because password-based SSH is a serious attack surface. If your server is publicly accessible, bots are hammering port 22 constantly — key-only auth shuts that attack vector down entirely.

This guide covers generating an Ed25519 key pair on your local Linux machine, deploying the public key to a remote server, and optionally disabling password login. The same process applies whether you’re on Ubuntu 22.04 LTS, Debian 12, Rocky Linux 9, or most other modern distributions.

Prerequisites

• A local Linux machine (desktop or laptop) running OpenSSH client — installed by default on Ubuntu, Debian, Fedora, Arch, and most others
• A remote Linux server you can currently log into — either by password SSH or via your hosting control panel’s web console
• SSH access to the remote server on port 22 (or a custom port if you’ve changed it)
• A non-root user account with sudo privileges on the remote server, or root access directly
• OpenSSH 6.5 or later on both machines — run ssh -V to check. Ed25519 keys require this version minimum. Any server provisioned in the last five years will have it.

Step-by-Step Instructions

Step 1: Generate an Ed25519 Key Pair Locally

Run this on your local machine, not the server. Ed25519 is the algorithm I’d recommend — it’s faster than RSA 4096 and the keys are shorter, which means fewer headaches with tools that truncate long strings. RSA 4096 is still fine if you have compatibility requirements, but Ed25519 is the modern default.

ssh-keygen -t ed25519 -C "your_email@example.com" -f ~/.ssh/id_ed25519

The -C flag adds a comment to the key — typically your email. It doesn’t affect functionality but helps you identify which key is which when you have several. The -f flag sets the output filename explicitly so there’s no ambiguity.

You’ll be prompted for a passphrase. Use one. A passphrase encrypts your private key on disk, so if your laptop is stolen the key is still useless without it. You can add the key to ssh-agent so you only type the passphrase once per session — covered in the FAQ below.

After running the command, you’ll have two files:

• ~/.ssh/id_ed25519 — your private key. Never share this, never copy it to the server, never paste it anywhere.
• ~/.ssh/id_ed25519.pub — your public key. This is what you copy to the server.

Step 2: Copy Your Public Key to the Remote Server

The cleanest way is ssh-copy-id, which handles file permissions automatically. Permissions on ~/.ssh/authorized_keys are one of the most common reasons SSH key auth fails silently.

ssh-copy-id -i ~/.ssh/id_ed25519.pub your_user@your_server_ip

You’ll be prompted for your current SSH password one last time. After that, your public key is appended to ~/.ssh/authorized_keys on the server with correct permissions set automatically.

📝 Note: If ssh-copy-id isn’t available on your local machine (uncommon but possible on some minimal installs), you can do it manually:

# Run this on your local machine — it pipes your public key directly into authorized_keys on the server
cat ~/.ssh/id_ed25519.pub | ssh your_user@your_server_ip "mkdir -p ~/.ssh && chmod 700 ~/.ssh && cat >> ~/.ssh/authorized_keys && chmod 600 ~/.ssh/authorized_keys"

The chmod 700 on ~/.ssh and chmod 600 on authorized_keys are not optional. OpenSSH will refuse to use keys if these permissions are too open — this is a deliberate security check, not a bug.

Step 3: Test Key-Based Login Before Disabling Passwords

⚠ Warning: Don’t disable password authentication until you’ve confirmed key login works. If you lock yourself out, you’ll need to use your hosting provider’s web-based console to recover access.

Open a new terminal window — keep your existing session open — and test the connection:

ssh -i ~/.ssh/id_ed25519 your_user@your_server_ip

If it connects without prompting for your server password (only your key passphrase if you set one), key auth is working. If it still asks for your server password, skip to the troubleshooting section below.

Step 4: Disable Password Authentication (Recommended)

Once key login is confirmed, disabling password auth removes a major attack vector. On your remote server, edit the SSH daemon config:

sudo nano /etc/ssh/sshd_config

Find and set these three directives. They may already exist but be commented out with # — uncomment them and change the values:

PasswordAuthentication no
PubkeyAuthentication yes
AuthorizationKeysFile .ssh/authorized_keys

📝 Note: On Ubuntu 22.04 and later, there’s a second config file at /etc/ssh/sshd_config.d/50-cloud-init.conf that can override your main config. Open it and check — if it contains PasswordAuthentication yes, either remove that line or change it to no. This trips up a lot of people and the error isn’t obvious at all.

Save the file, then restart the SSH service:

# On systemd-based distros (Ubuntu, Debian, Rocky Linux, AlmaLinux)
sudo systemctl restart sshd

⚠ Warning: Don’t close your current SSH session yet. Open another terminal and confirm you can still log in with your key. Only then is it safe to close everything.

Step 5: Set Correct Permissions If You Configured Keys Manually

If you copied the key manually rather than using ssh-copy-id, double-check permissions on the server:

chmod 700 ~/.ssh
chmod 600 ~/.ssh/authorized_keys

The home directory itself also can’t be world-writable — chmod 755 ~ if you’re unsure.

Common Issues & Troubleshooting

SSH Still Asks for Password After Copying the Key

Almost always a permissions problem on the server. SSH is deliberately strict about this. Log in via your hosting console and run:

ls -la ~/.ssh/
stat ~/.ssh/authorized_keys

~/.ssh must be 700, authorized_keys must be 600, and both must be owned by your user — not root. Also check your home directory isn’t world-writable (ls -ld ~). If it is, OpenSSH ignores the key entirely.

Permission Denied (publickey) Error

This means the server rejected your key. Run SSH in verbose mode to see exactly what’s happening:

ssh -vvv -i ~/.ssh/id_ed25519 your_user@your_server_ip 2>&1 | grep -A2 "Offering|Authentications"

Look for lines showing which key was offered and whether it was accepted. Common causes: wrong key file specified, public key not in authorized_keys, or PubkeyAuthentication disabled in sshd_config.

Ubuntu 22.04: Password Auth Still Works After Setting PasswordAuthentication no

This is the /etc/ssh/sshd_config.d/50-cloud-init.conf override issue mentioned in Step 4. Cloud-init drops that file during provisioning and it takes priority. Edit it directly:

sudo nano /etc/ssh/sshd_config.d/50-cloud-init.conf
# Change or remove the line: PasswordAuthentication yes
sudo systemctl restart sshd

Agent Admitted Failure to Sign Using the Key

Your key is loaded into ssh-agent but the agent can’t use it, often because the key was added to the agent but the passphrase wasn’t accepted correctly, or the agent lost its state after a system resume. Fix it by re-adding the key:

ssh-add -D             # clears all keys from the agent
ssh-add ~/.ssh/id_ed25519   # re-adds your key, prompts for passphrase

SELinux Blocking SSH Key Auth on Rocky Linux / AlmaLinux

On SELinux-enforcing systems, if you created ~/.ssh or authorized_keys in an unusual way (e.g. copied from another location), the SELinux context might be wrong. Fix the context and restart:

restorecon -Rv ~/.ssh
sudo systemctl restart sshd

This is annoyingly underdocumented. The SSH daemon won’t log a useful error — it just falls back to password auth silently.

FAQ