JDPFu.com 2024

ssh-keygen -t ed25519
ssh-copy-id -i ~/.ssh/id_ed25519.pub userid@remote

$ egrep -i Failed /var/log/auth.log*

Listen on a Non-standard Port

This single change is the single most effective for ending unwanted ssh connection attempts. For some reason, attackers don’t bother looking for ssh on non-standard ports, at least not yet. Pretty much any port that is not 22 will work.

Ok, this can be a major hassle if you do it on internal machines. After all, your machines all expect to connect via ssh on port 22 after all. The real concern is from the outside bad guys , not the internal people trying to get work done. There are 2 ways to handle this. I’ll lead with the easier.

a) Use Port Translation on the Router to forward some high port, perhaps 34022, to an internal ssh-gateway on port 22. This means your outside world connection isn’t on port 22 and your internal users and programs still connect on the default port – 22. Pretty simple. You make the change in 1 place. The only downside is that when you are outside , then you need to remember which port to connection to. That’s easy too – use the ~/.ssh/config file for this. Read more about using ~/.ssh/config files. The short version is:

host remote_name_for_server

  user pete

  hostname remote5433.dyndns.org

  port 34022

I use different host settings for my machine when outside the LAN, so this is easy and I don’t need to recall what the actual port number used is. Win-Win. Inside " blog " connects on port 22. Outside, " blog.jdpfu.com ", if there were an ssh connection, would connect on some high port. This blog server doesn’t support external ssh connections, I have to hop onto another box on the same LAN to get there first.

b) The other way is to change the port on your gateway ssh server. Edit your /etc/ssh/sshd_config and change the Port 22 line to something else. You’ll need root access and will want to restart sshd. Be certain to use sudoedit /etc/ssh/sshd_config – this is safer than other alternatives for a few reasons.

Port 34022

Now you just need to remember to use that port for all furture ssh connections. That can be a hassle or you can just setup the alternate port in your ~/.ssh/config files on every client machine, everywhere. Be certain to change the router forwarding rule to this new port as well.

Use ssh-key-based connections

How often does something that is more secure actually make it easier? key-based ssh connections are easy and a one-time setup thing. Keys are 1,000,000,000,000x more secure than passwords. Use them, live by them, love them. Allowing passwords for authentication is actually crazy.

No Remote Root Logins

sudoedit your /etc/ssh/sshd_config and change the PermitRootLogin yes to either

PermitRootLogin no

or

PermitRootLogin without-password

Simple. The without-password will leave your automatic jobs from remote machines which run as root working. It should be stated that your PAM settings may bypass this PermitRootLogin setting, so test your results.

We can limit the source for remote root access, which isn’t as good as preventing it completely, but security is about trade-offs and mitigations. To allow a subnet to login as root:

AllowUsers root user1

Be certain to restart the sshd after changing these settings.

$ sudo /etc/init.d/ssh restart

or if your distro is forcing that service controller crap …

$ sudo service ssh restart

The rest of the suggestions are handled by Fail2Ban.

Fail2Ban

Fail2ban completely rocks.

From the package description:

Monitors log files (e.g. /var/log/auth.log, /var/log/apache/access.log) and temporarily or persistently bans failure-prone addresses by updating existing firewall rules.

Installation:

$ sudo apt-get install fail2ban

Configuration for ssh – well, there is good news, Fail2Ban is configured for ssh on port 22 already. The only items you may wish to change are the default lockout times for blocked IPs. The bad news is you need to install Fail2Ban on each of your systems. That’s pretty trivial, IMHO. For fun, just to see how many hack attempts are on port 22, change your router port forwards for a few days from the high port back to port 22 and you’ll see hundreds of ssh login attempts from all over the internet.

Fail2ban can interfere with other firewall tools and rules. Be certain that if you are doing anything complex with the firewall, that fail2ban is still working. I found that iptables-save and iptables-restore had issues with fail2ban.

Watching the Fail2Ban log files

Fail2Ban is happy to work silently for you, but I like to know what’s happening, even if it is boring. Here’s a simple script to monitor some fail2ban.log files on multiple servers.
#!/bin/bash

1. ########################################################
2. Check the Fail2ban log file on all important servers
3. ########################################################
   LOGFILE=/var/log/fail2ban.log

1. Server list for root logins
   ROOT_SRVS=“srv1 srv2 srv3 srv4 srv5 srv6”

1. Server list for normal user logins
   USER_SRVS=“serverB remote4534.dyndns.org serverA”

1. ########################################################
   Chk_Fail_Log_Root(){
   echo "===
   Working on $D
   "
   ssh root@$D “if [[ -r $LOGFILE ]] ; then egrep ‘WARNING|ERROR’ $LOGFILE; fi”
   }

1. ########################################################
   Chk_Fail_Log_USER(){
2. Be certain to change the userid
   echo "===
   Working on $D
   "
   ssh pete@$D “if [[ -r $LOGFILE ]] ; then sudo egrep ‘WARNING|ERROR’ $LOGFILE; fi”
   }

1. ########################################################
2. main
   for D in $ROOT_SRVS ; do
   Chk_Fail_Log_Root
   done

for D in $USER_SRVS ; do
Chk_Fail_Log_USER
done

echo "===
Working on localhost
"
if [[ -r $LOGFILE ]] ; then egrep WARNING $LOGFILE; fi
echo "
DONE.
="


This script is nearly identical to what has been running here for a few years. Only the server lists and userid are changed.
Sample output:

===

Working on foo

2015-10-03 07:00:46,924 fail2ban.actions.action: ERROR  iptables -D INPUT -p tcp -m multiport —dports ssh -j fail2ban-ssh

2015-10-03 08:50:50,516 fail2ban.actions: WARNING [pam-generic] Ban 61.160.223.35

2015-10-03 08:50:52,516 fail2ban.actions: WARNING [ssh] Ban 61.160.223.35

2015-10-03 12:37:31,129 fail2ban.actions: WARNING [pam-generic] Unban 61.160.223.35

2015-10-03 12:37:32,994 fail2ban.actions: WARNING [ssh] Unban 61.160.223.35

Fail2Ban for other Services

Fail2Ban includes settings for lots of services like apache, IMAP, Qmail, postfix, wsftpd, etc… If you have created a new service that has a log file, then you can create your own filter and action for Fail2Ban to follow. There are lots of examples that you can follow. Just a little knowledge about regex is needed.

Fail2ban doesn’t help with coordinated attacks from thousands of different internet IPs. These attacks generally come from 10,000 – 500,000 different IPs with each trying a different password for each of their 3 attempts before getting blocked. Since the attempts are spread over many thousands of different IPs, fail2ban doesn’t really work. OTOH, if we use ssh-keys for authentication, it just doesn’t matter. For those sorts of attacks, port knocking might be useful, but once the ssh daemon is running, we still need the protection of fail2ban, so it still needs to be installed.

For me, if I install openssh-server, I also install fail2ban, period. It has become a habit. If you are an ansible user, here’s an easy way to check+install as needed fail2ban on all your machines.

$ ansible all -m apt -a “name=fail2ban state=present”

The output I saw was all

blog | success >> {

    “changed”: false

}

How cool is that?

Fail2ban Email

If you have an MTA setup, then fail2ban will email any blocked IPs. If you’ve changed the default ssh port, this doesn’t happen very often. The same IP from the script above caused this email:

Hi,

The IP 61.160.223.35 has just been banned by Fail2Ban after

10 attempts against ssh.
Here are more information about 61.160.223.35:
% [whois.apnic.net]

% Whois data copyright terms    http://www.apnic.net/db/dbcopyright.html
% Information related to ‘61.160.0.0 – 61.160.255.255’
inetnum:        61.160.0.0 – 61.160.255.255

netname:        CHINANET-JS

descr:          CHINANET jiangsu province network

descr:          China Telecom

descr:          A12,Xin-Jie-Kou-Wai Street

descr:          Beijing 100088

country:        CN

admin-c:        CH93-AP

tech-c:         CJ186-AP

mnt-by:         MAINT-CHINANET

mnt-lower:      MAINT-CHINANET-JS

mnt-routes:     maint-chinanet-js

changed:        hostmaster@ns.chinanet.cn.net 20020209

changed:        hostmaster@ns.chinanet.cn.net 20030306

status:         ALLOCATED non-PORTABLE

source:         APNIC
role:           CHINANET JIANGSU

address:        260 Zhongyang Road,Nanjing 210037

country:        CN

phone:          86-25-86588231

phone:          +86-25-86588745

fax-no:         +86-25-86588104

e-mail:         ip@jsinfo.net

remarks:        send anti-spam reports to spam@jsinfo.net

remarks:        send abuse reports to abuse@jsinfo.net

remarks:        times in GMT8

admin-c:        CH360-AP

tech-c:         CS306-AP

tech-c:         CN142-AP

nic-hdl:        CJ186-AP

remarks:        www.jsinfo.net

notify:         ip@jsinfo.net

mnt-by:         MAINT-CHINANET-JS

changed:        dns@jsinfo.net 20090831

changed:        ip@jsinfo.net 20090831

changed:        hm-changed@apnic.net 20090901

source:         APNIC

changed:        hm-changed@apnic.net 20111114
person:         Chinanet Hostmaster

nic-hdl:        CH93-AP

e-mail:         anti-spam@ns.chinanet.cn.net

address:        No.31 ,jingrong street,beijing

address:        100032

phone:          +86-10-58501724

fax-no:         +86-10-58501724

country:        CN

changed:        dingsy@cndata.com 20070416

changed:        zhengzm@gsta.com 20140227

mnt-by:         MAINT-CHINANET

source:         APNIC
% Information related to ‘61.160.0.0/16AS23650’
route:          61.160.0.0/16

descr:          CHINANET jiangsu province network

country:        CN

origin:         AS23650

mnt-by:         MAINT-CHINANET-JS

changed:        ip@jsinfo.net 20030414

source:         APNIC
% This query was served by the APNIC Whois Service version 1.69.1-APNICv1r0 (UNDEFINED)
Regards,
Fail2Ban

That is useful information, right? BTW, this happened less than 1 week after the US and Chinese Presidents agreed to stop hacking each other.

I decided to block the entire subnet, 61.160.0.0/16, at the router.

Allow Only Selected IPs to Connect

Use your firewall to allow only specific internet IPs to connect to your ssh server. This may not be possible, but if all the users will always connect to the machine from a specific list of static networks, allowing only those is much better than allowing the billions of other IPs access too.

sudo ufw allow from 55.0.0.0/10 to any port 34022

sudo ufw deny 34022

With road warriors, this is very hard, but I’ve been blocking entire countries based on flawed country-to-IP subnet tables. If none of your road warriors is in China, Russia, Romania, Ukraine, or the USA, then it is relatively safe to block the thousands of subnets allocated to those countries, if only to reduce the unwanted attempts. Country subnet lists are highly flawed, so you may get a call from a road warrior who cannot connect over ssh. I’ve never gotten any calls, but my guys probably are running their own ssh-mini-vpns and hop onto our systems using their home ISP connections anyway. If you might be locked out yourself, setting up a port-knocking solution that isn’t blocked world-wide would be smart as an added access protection method.

I have a perl script to read thousands of subnets in country subnet files and deny all access. It isn’t pretty, but it works. You can google to find the subnet files by country. Then look at your logs do decide who are the troublemakers.

Use ~/.ssh/config

This file will make using ssh easier, especially when userids and ports are non-standard. Learn more.

ssh-copy-id

This is a handy tool to push the public ssh-keys to another machine, so that key-based authentication is used going forward. The command warns to check the modifications to the remote file, but I’ve never seen it do the wrong thing, even when pushing root keys to a non-root backup userid and remote server. Very handy.

ssh-copy-id userid@on_remote_machine

Simple. You’ll be prompted for login the password for the last time – don’t forget to change the /etc/hosts.allow and sshd_config to reflex your new level of security. Console logins do not use ssh, so if you have console access, you have an alternative method of access with that old, crusty, password.

Key-based Authentication, Unless Internal

This will force ssh keys to be used for logins, unless the person is on the internal network:

PasswordAuthentication no

ChallengeResponseAuthentication no

Match Address 10.0.0.0/8,172.16.0.0/12,192.168.0.0/16

    PasswordAuthentication yes

Allow Specific Users SSH Access

Put the userids you’d like to allow to ssh into the “sshlogin” group. Probably need to create that group. Another group name is fine too.

Then add this to the sshd_config file:

AllowGroups sshlogin

More details.

Summary

Be certain to move your ssh listener to a non-standard port. That alone is the greatest change you can make. It really is sad how well that works. If you are afraid to change your sshd settings, at least install Fail2Ban to block repeated failed attempts automatically. Those 2 changes will make your ssh connections from the internet almost impossible to hack without access to YOUR system with ssh-keys.

If you have a high value server or have pissed off some hackers on the internet, it would be smart to use additional setting.

Some people