WordPress is under attack! Watch it! Password Protect it!

What? What do you mean? There’s already a password. Yes, you need to log in when you want to put up a new blog post or do maintenance of some sort. However, that doesn’t mean that you can’t have an additional layer of protection. Not only can you have it, WordPress actually recommends it here: https://codex.wordpress.org/Brute_Force_Attacks

I looked in my nginx access log and I saw a bunch of messages that looked like this:

95.219.148.136 - - [16/Nov/2017:06:34:33 -0800] "GET /wp-login.php HTTP/1.1" 402 195 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.1"
95.219.148.136 - - [16/Nov/2017:06:34:34 -0800] "GET / HTTP/1.1" 200 21587 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.1"
202.152.71.21 - - [16/Nov/2017:06:40:48 -0800] "GET /wp-login.php HTTP/1.1" 402 195 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.1"
202.152.71.21 - - [16/Nov/2017:06:40:49 -0800] "GET / HTTP/1.1" 200 21589 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.1"
177.221.4.36 - - [16/Nov/2017:06:55:42 -0800] "GET /wp-login.php HTTP/1.1" 402 195 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.1"
177.221.4.36 - - [16/Nov/2017:06:55:42 -0800] "GET / HTTP/1.1" 200 21589 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.1"

After doing some investigation, it appeard to be the sathurbot attacking my blogsite. It’s some sort of distributed piece of malware that attacks poorly maintained or blogs with weak passwords. The malware tries to attack the wp-login and something else. You can read more about it here: https://www.welivesecurity.com/2017/04/06/sathurbot-distributed-wordpress-password-attack/.

The first thing I did to counter this issue was configure Cloudflare to under attack mode. This gives the client a short delay when connecting to your site so that can’t get to the file. This should stop the entries in the log completely, immediately. Since I don’t want users to see the delay all of the time, I decided after the attacks slowed to have nginx password protect the file so that when trying to request it, nginx will ask for a password as well. This way, you’ll need to authenticate twice to get into WordPress, but it’s okay. The extra trouble gives me peace of mind that I’ll less likely be attacked.

With nginx, I did it this way:

location ^~ /wp-login.php {
 auth_basic "Administrator Login";
 auth_basic_user_file /etc/nginx/conf.d/.htpasswd;
 include fastcgi.conf;
 fastcgi_intercept_errors on;
 fastcgi_pass php-wphandler;
 fastcgi_buffers 16 16k;
 fastcgi_buffer_size 32k;
}

The .htpasswd is a hashed file. You can create it with the htpasswd command that comes with the apache2-utils package. The file would look something like this:

alton:[email protected]$SDFapr1$yDoxiXVW$aFe

Now in my logs, I get 401 messages instead of 402 messages.

172.68.242.50 - - [29/Nov/2017:09:36:50 -0800] "GET /wp-login.php HTTP/1.1" 401 195 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.1" "134.196.23.66"
172.68.246.96 - - [29/Nov/2017:09:45:48 -0800] "GET /wp-login.php HTTP/1.1" 401 195 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.1" "193.93.187.11"
162.158.91.51 - - [29/Nov/2017:09:49:22 -0800] "GET /wp-login.php HTTP/1.1" 401 195 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.1" "93.172.55.76"
141.101.77.120 - - [29/Nov/2017:10:08:03 -0800] "GET /wp-login.php HTTP/1.1" 401 195 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.1" "41.100.125.248"

I also know that they’re less likely to hack my site. 🙂

Happy blogging!

Using vim-cmd to remedy a bsod

Here’s a great tutorial for vim-cmd if you haven’t had experience with it before by my friend, Steve Jinwww.doublecloud.org/2013/11/vmware-esxi-vim-cmd-command-a-quick-tutorial/

This is a real-world situation I got myself into when I tried connecting to my client VM and found a BSOD that looked like this:

It’s pretty obvious that the reason for the crash is the USB stick that’s plugged in from the usbuhci.sys line in the blue screen. Since I tunnel into my client VM via SSH and VNC, the easiest way for me to shutdown my VM and remedy this issue is through vim-cmd. This only works if you have SSH allowed onto your ESXi host or if you are connecting to the host with the VMware CLI or vMA or whatever they’re calling it these days. I have the former.

The first thing I do after logging into the ESXi host as root is run:

vim-cmd vmsvc/getallvms

I need to know which one of my VMs is the one to manage. I get this:

Vmid Name File Guest OS Version Annotation
1 windows7 [BIG_DISK] windows7/windows7.vmx windows7_64Guest vmx-07
3 thimble [BIG_DISK] thimble/thimble.vmx ubuntu64Guest vmx-08
4 chunli [Datastore 2] chunli2/chunli2.vmx ubuntu64Guest vmx-08
5 zangief [Datastore 2] zangief2/zangief2.vmx ubuntu64Guest vmx-11

With this information, I know that it’s VM 1, so I power it off by running:

vim-cmd vmsvc/power.off 1

Thinking the USB issue might be a fluke, I try to power the VM back on to see if it will boot.

vim-cmd vmsvc/power.on 1

I see that it starts booting, but as the resolution changes on the VM, my VNC viewer freezes. Since I normally don’t know exactly when it freezes, I didn’t know when I got the BSOD again.

Until I decided to at look at the vmware.log file. This is what I saw there:

2017-10-18T22:27:05.519Z| svga| I125: SVGA disabling SVGA
2017-10-18T22:27:05.545Z| svga| W115: WinBSOD: (20) 'Technical information: '
2017-10-18T22:27:05.545Z| svga| W115:
2017-10-18T22:27:05.546Z| svga| W115: WinBSOD: (22) '*** STOP: 0x000000D1 (0xFFFFF88000BF2000,0x0000000000000002,0x0000000000000001,0'
2017-10-18T22:27:05.546Z| svga| W115:
2017-10-18T22:27:05.546Z| svga| W115: WinBSOD: (23) 'xFFFFF88004206E49) '
2017-10-18T22:27:05.546Z| svga| W115:
2017-10-18T22:27:05.557Z| svga| W115: WinBSOD: (26) '*** usbuhci.sys - Address FFFFF88004206E49 base at FFFFF88004200000, DateStamp'
2017-10-18T22:27:05.557Z| svga| W115:
2017-10-18T22:27:05.557Z| svga| W115: WinBSOD: (27) ' 57b37a29 '
2017-10-18T22:27:05.557Z| svga| W115:
2017-10-18T22:27:05.557Z| svga| W115: WinBSOD: (30) 'Collecting data for crash dump ... '
2017-10-18T22:27:05.557Z| svga| W115:
2017-10-18T22:27:05.573Z| svga| W115: WinBSOD: (31) 'Initializing disk for crash dump ... '
2017-10-18T22:27:05.573Z| svga| W115:
2017-10-18T22:27:07.547Z| mks| W115: Guest operating system crash detected.

Okay, so I see that my hunch is correct. I guess it’s time I remove the USB device from the VM. So I power off the VM again and open up the vmx file and just start removing all instances of USB.

These are the lines I removed. Don’t worry about breaking anything. The hypervisor will put them back if you need them later. Back up your vmx file before doing it though just in case.

usb.pciSlotNumber = "34"
usb.present = "TRUE"
usb:1.speed = "2"
usb:1.present = "TRUE"
usb:1.deviceType = "hub"
usb:1.port = "1"
usb:1.parent = "-1"
usb.autoConnect.device0 = "path:1/1 autoclean:1"
usb:0.present = "TRUE"
usb:0.deviceType = "mouse"
usb:0.port = "0"
usb:0.parent = "-1"

After you’ve saved your changes, you’ll need to reload the changes so that ESXi will reread the .vmx file to remove the USB device. You can do this by running this command:

vim-cmd vmsvc/reload 1

Now you’re ready to power on the VM.

vim-cmd vmsvc/power.on 1

The VM powers up and I’m back in business. I just had to figure out the USB issue later. Turned out that I just needed to reconnect the device and reformat it. I haven’t seen the issue come up again.

 

Red Hat Enterprise Linux 7.3 is broken!

At least kernel-3.10.0-514.26.2.el7.x86_64.rpm is broken. With it, you will not be able to use a stack size lower than ~4.5MB.

Here’s some reading on why your applications would want to do this: https://www.systemcodegeeks.com/shell-scripting/bash/using-rlimit-and-why-you-should/

Here’s an excerpt:

Why do we care?

Security in depth.

First, people make mistakes. Setting reasonable limits keeps a runaway process from taking down the system.

Second, attackers will take advantage of any opportunity they can find. A buffer overflow isn’t an abstract concern – they are real and often allow an attacker to execute arbitrary code. Reasonable limits may be enough to sharply curtail the damage caused by an exploit.

Here are some concrete examples:

First, setting RLIMIT_NPROC to zero means that the process cannot fork/exec a new process – an attacker cannot execute arbitrary code as the current user. (Note: the man pages suggests this may limit the total number of processes for the user, not just in this process and its children. This should be double-checked.) It also prevents a more subtle attack where a process is repeatedly forked until a desired PID is acquired. PIDs should be unique but apparently some kernels now support a larger PID space than the traditional pid_t. That means legacy system calls may be ambiguous.

Second, setting RLIMIT_ASRLIMIT_DATA, and RLIMIT_MEMLOCK to reasonable values prevents a process from forcing the system to thrash by limiting available memory.

Third, setting RLIMIT_CORE to a reasonable value (or disabling core dumps entirely) has historically been used to prevent denial of service attacks by filling the disk with core dumps. Today core dumps are often disabled to ensure sensitive information such as encryption keys are not inadvertently written to disk where an attacker can later retrieve them. Sensitive information should also be memlock()ed to prevent it from being written to the swap disk.

You can try running the following commands:

ulimit -s 4096
/bin/true

and see this output:

-bash: /bin/true: Argument list too long

Really!? Find more at Red Hat Bug 1463241 – rlimit_stack problems after update.

If you’re using this kernel, I suggest you upgrade immediately. Your applications that might be written with these limits set wil fail.

 

Automated backup of AWS route53 zones

cli53! It’s the coolest tool you can use for Amazon DNS route53! This is the posting I had tried to follow for backing up my zone files.

Automated Backup Of AWS Route 53 Record Sets

I suspect that AWS changed the output of this command, so it no longer works. Here’s one that does:

cli53 list | awk '{print $2}' | grep -v Name | while read line; do cli53 export ${line} > ~/backup/${line}bk; done

With this command, it will grab all of the domains and back up each of the zone files.

Spinning up a bunch of virtual desktops in Amazon WorkSpaces (videos)

This was a pretty fun project that I had gotten so I figured I would share the experience. There are multiple use cases for virtual desktops. In our case, it’s ephemeral – only need them for a few days for a class so that all students can share the same experience without the need for anything but a web browser. They can probably get a better experience with the PCoIP client, but it could be against some company policies. Most companies will allow HTTPs out, so we figured this would be the easiest way.

The way Amazon WorkSpaces works is that each desktop is assigned to a single user and the users sit in the directory service. The service I’m using is the Simple AD (Samba 4) as I had no need for a huge directory. To create the users, we will just need a UID (sAMAccountName in AD) and a password if using the API to create the desktops. If using the Amazon portal to create the desktops, you’ll need the first and last name and an email address as well. You can easily import a CSV file with this information, but for the sake of simplicity, I just use a generic account name and numbers.

After creating the directory and starting up a single desktop, I went to the “Programs” in the Control Panel and “Turn Windows Features on and off” and “Features” to install the “AD DS and AD LDS Tools”. More information on the RSAT tools is available here: https://wiki.samba.org/index.php/Installing_RSAT 

Here’s a short video on how to do it:

Once the RSAT tools are installed, the “dsadd” command will be available to add users. This is the script I’m using that asks for the users and then creates the users:

echo off
set /p users=Number of users to create:
echo "Creating %users% students"
set count=0
:createusers
    set /a count+=1
    echo creating student%count%
    dsadd user "cn=student%count%,cn=users,dc=corp,dc=amazonworkspaces,dc=com" -samid student%count% -pwd Student%count%
    if "%count%"=="%users%" goto done
    goto createusers
:done

The script will create users with the username student# with passwords Student# – the capital “S” is just for password complexity.

After creating the users, we can go and create the desktops. To do this, I used awscli. On a Mac or Linux system, it can be easily installed running “easy_install awscli”. After installation, there will be a config and credentials file that should be configured in the .aws directory in your home directory. Once that’s set, you can check to see what workspaces you have by running “aws workspaces describe-workspaces” – that gives you an idea of what your workspaces look like. The minimal template I’m using for workspaces looks like this:

{
 "Workspaces" : [
 {
   "DirectoryId" : "d-9267258c77",
   "UserName" : "%username%",
   "WorkspaceProperties": {
   "RunningMode": "AUTO_STOP"
 },
   "BundleId" : "wsb-gw81fmq2p"
 }
 ]
}

The DirectoryId is the directory service where the users are housed, I’ll be replacing the %username% with student#, and I added RunningMode just to save on costs – they’ll automatically suspend after an hour of idling. It takes about 90s to spin back up if they suspend. The BundleId is the VM that you want to provision. This one is the customized one for our classroom.

With the template in place, we’re ready to run the script:

#!/bin/bash

echo "Number of Desktops to Create [20?]"
read desktops
echo $desktops
COUNTER=0
         while [  $COUNTER -lt $desktops ]; do
             let COUNTER=COUNTER+1
             echo Creating Desktop number $COUNTER
        sed "s/%username%/student$COUNTER/g" create-workspaces.json > /tmp/student$COUNTER.json
        aws workspaces create-workspaces --cli-input-json file:///tmp/student$COUNTER.json
         done
echo Created $desktops Desktops.

You can remove the temporary files in /tmp afterwards.

Here’s a short video of the scripts in action.

Have fun with your desktops!

AIX notes … ipfilter, unzip, zlib, openssh, openssl

I had the privilege of experiencing AIX for the very first time this week. Hopefully this can save someone else time.

Some packages that aren’t installed by default that you might want include openssl, openssh, unzip, zlib, and IPFilter.

I would probably start with openssl/openssh. In AIX 7.2, you can do it in the OS installer. To do it outside of the installer, keep the installation cd in and run the following commands:

mount -V cdrfs -o ro /dev/cd0 /mnt
cd /mnt/usr/sys/inst.images/
installp -ac -Y -d . openssh.base openssl.base openssl.man.en_US openssh.man.en_US
lssrc -s sshd
umount /mnt

The default partitions aren’t big enough! Fortunately, it’s very easy to extend the partitions. You can do so with the following commands:

chfs -a size=+4G /opt
chfs -a size=+4G /var
chfs -a size=+4G /home
chfs -a size=+4G /usr
chfs -a size=+2G /tmp
chfs -a size=+4G /admin

Installing 3rd party software:

You can download unzip from:  https://public.dhe.ibm.com/aix/freeSoftware/aixtoolbox/RPMS/ppc/unzip/unzip-6.0-3.aix6.1.ppc.rpm. You can install it with “rpm -i” just like in Linux. Another open for unzipping files without unzip is using jar. You can run “jar -xvf” on a file and it can unzip it as well.

If you need the zlib library, you can get it from:  https://public.dhe.ibm.com/aix/freeSoftware/aixtoolbox/RPMS/ppc/zlib/zlib-1.2.11-1.aix6.1.ppc.rpm. The library was listed in the README, but I didn’t install it and the VEN seemed to work without it.

You can install IPFilter from https://www-01.ibm.com/marketing/iwm/iwm/web/reg/pick.do?source=aixbp. It will require a login, but not a serial number. Just create a login and download. Installing IPFilter is a little different. It installs like an AIX package, with installp. Unzip the contents of the IPFilter_Fileset.zip and go into the IPFilter_Fileset directory and run the following commands:

inutoc .
installp -ac -gXY -d. ipfl

 

Help! SSH attack on Ubuntu 14.04.2 LTS – use sshdfilter with ufw

If you’ve got a *Nix machine on the public Internet, you will experience this at one time or another. If you run sshd on its’ known port 22, some script kiddie out there will attack it. With that, you’ll see that you’ll have a bunch of connections that probably and hopefully will never succeed. I saw these messages in my /var/log/auth.log:

Nov 15 06:44:26 chunli sshd[20634]: Failed password for root from 43.229.53.13 port 41751 ssh2
Nov 15 06:44:26 chunli sshd[20636]: Failed password for root from 43.229.53.13 port 41921 ssh2
Nov 15 06:44:26 chunli sshd[20638]: Failed password for root from 43.229.53.13 port 42948 ssh2
Nov 15 06:44:26 chunli sshd[20546]: message repeated 2 times: [ Failed password for root from 43.229.53.13 port 27586 ssh2]

I’ve experienced this before and to alleviate this problem, I used sshdfilter in the past. I know that there are others out there like sshblack that can do blacklisting of attackers or one of the easiest ways to alleviate this problem is just to run ‘sudo ufw limit ssh’. This is a way to block them from coming back for a while.

To get started, first stop SSH:

sudo initctl stop ssh

Then move /etc/init/ssh.conf out. We’ll start SSH with sshdfilter instead.

You’ll get a message in the auth.log here:
auth.log:Nov 16 19:54:36 chunli sshd[1150]: Missing privilege separation directory: /var/run/sshd

Because of this, we’ll just add a line in the /etc/init.d/sshdfilter file like this:

start() {
 echo -n $"Starting sshdfilter: "
 export PATH=$PATH:/usr/local/sbin
 mkdir /var/run/sshd
 sshdfilter
 RETVAL=$?
 return $RETVAL

In /etc/sshdfilterrc, I changed the chain from:

#chain=’SSHD’ to

chain='ufw-reject-input'

Under the “Add a block rule” section, I changed some iptables commands to ufw commands. You can see the comments ones here:

#firewalladd='iptables -A $chain -p tcp -s $ip --dport 22 -j DROP'
#firewalladd='iptables -A $chain -p tcp -s $ip --dport 22 -j DROP'
firewalladd='ufw insert 1 reject proto tcp from $ip to any port 22'
# Delete a block rule:
firewalldel='ufw delete reject proto tcp from $ip to any port 22'
#firewalldel='iptables -D $chain -p tcp -s $ip --dport 22 -j DROP'

That’s about it! With that, my machine started to grow a big list of IP addresses to block from failed logins.