Tag: Ubuntu

Updating Or Upgrading When /boot Is 100% Full

I recently ran into an issue on a test server with a particularly small /boot partition. I attempted to run the typical sudo apt update && sudo apt upgrade and it failed. This was part of the error:

Preparing to unpack .../linux-image-4.4.0-112-generic_4.4.0-112.135_amd64.deb ...
Done.
Unpacking linux-image-4.4.0-112-generic (4.4.0-112.135) ...
dpkg: error processing archive /var/cache/apt/archives/linux-image-4.4.0-112-generic_4.4.0-112.135_amd64.deb (--unp ack):
 cannot copy extracted data for './boot/vmlinuz-4.4.0-112-generic' to '/boot/vmlinuz-4.4.0-112-generic.dpkg-new': failed to write (No space left on device)
No apport report written because the error message indicates a disk full error

I tried running sudo apt autoremove as suggested to clear out the old Linux files. This ran into an error which looked like so:

cole@ubuntu-docker:~$ sudo apt autoremove
Reading package lists... Done
Building dependency tree
Reading state information... Done
You might want to run 'apt-get -f install' to correct these.
The following packages have unmet dependencies:
 libstdc++-5-dev : Depends: libstdc++6 (>= 5.4.0-6ubuntu1~16.04.6) but 5.4.0-6ubuntu1~16.04.5 is installed
 libstdc++6 : Depends: gcc-5-base (= 5.4.0-6ubuntu1~16.04.5) but 5.4.0-6ubuntu1~16.04.6 is installed
 linux-image-extra-4.4.0-112-generic : Depends: linux-image-4.4.0-112-generic but it is not installed
 linux-image-generic : Depends: linux-image-4.4.0-112-generic but it is not installed
 Recommends: thermald but it is not installed
E: Unmet dependencies. Try using -f.

Running sudo apt -f install as suggested left me with:

cole@ubuntu-docker:~$ sudo apt -f install
Reading package lists... Done
Building dependency tree
Reading state information... Done
Correcting dependencies... Done
The following packages were automatically installed and are no longer required:
 libzip4 linux-headers-4.4.0-101 linux-headers-4.4.0-101-generic linux-headers-4.4.0-87
 linux-headers-4.4.0-87-generic linux-headers-4.4.0-91 linux-headers-4.4.0-91-generic linux-headers-4.4.0-92
 linux-headers-4.4.0-92-generic linux-headers-4.4.0-93 linux-headers-4.4.0-93-generic linux-headers-4.4.0-96
 linux-headers-4.4.0-96-generic linux-headers-4.4.0-97 linux-headers-4.4.0-97-generic linux-headers-4.4.0-98
 linux-headers-4.4.0-98-generic linux-image-4.4.0-101-generic linux-image-4.4.0-87-generic
 linux-image-4.4.0-91-generic linux-image-4.4.0-92-generic linux-image-4.4.0-93-generic
 linux-image-4.4.0-96-generic linux-image-4.4.0-97-generic linux-image-4.4.0-98-generic
 linux-image-extra-4.4.0-101-generic linux-image-extra-4.4.0-87-generic linux-image-extra-4.4.0-91-generic
 linux-image-extra-4.4.0-92-generic linux-image-extra-4.4.0-93-generic linux-image-extra-4.4.0-96-generic
 linux-image-extra-4.4.0-97-generic linux-image-extra-4.4.0-98-generic
Use 'sudo apt autoremove' to remove them.
The following additional packages will be installed:
 libstdc++6 linux-image-4.4.0-112-generic
Suggested packages:
 fdutils linux-doc-4.4.0 | linux-source-4.4.0 linux-tools
The following NEW packages will be installed:
 linux-image-4.4.0-112-generic
The following packages will be upgraded:
 libstdc++6
1 upgraded, 1 newly installed, 0 to remove and 61 not upgraded.
33 not fully installed or removed.
Need to get 0 B/22.3 MB of archives.
After this operation, 66.9 MB of additional disk space will be used.
Do you want to continue? [Y/n] y
Setting up gcc-5-base:amd64 (5.4.0-6ubuntu1~16.04.6) ...
(Reading database ... 444689 files and directories currently installed.)
Preparing to unpack .../libstdc++6_5.4.0-6ubuntu1~16.04.6_amd64.deb ...
Unpacking libstdc++6:amd64 (5.4.0-6ubuntu1~16.04.6) over (5.4.0-6ubuntu1~16.04.5) ...
Processing triggers for libc-bin (2.23-0ubuntu10) ...
Setting up libstdc++6:amd64 (5.4.0-6ubuntu1~16.04.6) ...
Processing triggers for libc-bin (2.23-0ubuntu10) ...
(Reading database ... 444689 files and directories currently installed.)
Preparing to unpack .../linux-image-4.4.0-112-generic_4.4.0-112.135_amd64.deb ...
Done.
Unpacking linux-image-4.4.0-112-generic (4.4.0-112.135) ...
dpkg: error processing archive /var/cache/apt/archives/linux-image-4.4.0-112-generic_4.4.0-112.135_amd64.deb (--unp ack):
 cannot copy extracted data for './boot/vmlinuz-4.4.0-112-generic' to '/boot/vmlinuz-4.4.0-112-generic.dpkg-new': f ailed to write (No space left on device)
No apport report written because the error message indicates a disk full error
 dpkg-deb: error: subprocess paste was killed by signal (Broken pipe)
Examining /etc/kernel/postrm.d .
run-parts: executing /etc/kernel/postrm.d/initramfs-tools 4.4.0-112-generic /boot/vmlinuz-4.4.0-112-generic
run-parts: executing /etc/kernel/postrm.d/zz-update-grub 4.4.0-112-generic /boot/vmlinuz-4.4.0-112-generic
Errors were encountered while processing:
 /var/cache/apt/archives/linux-image-4.4.0-112-generic_4.4.0-112.135_amd64.deb
E: Sub-process /usr/bin/dpkg returned an error code (1)

I’ve clearly run into a loop that I’ll not be able to solve without removing some files from the /boot partition. I ran a simple ls -la to find the versions, and removed the oldest files. I ran the following scripts:

sudo rm *4.4.0-91-generic
sudo rm *4.4.0-92-generic
...

until I only had the most recent 6 versions. I checked df to see where I was at with disk space, and lo-and-behold I had it down to ~50%. Finally I ran the commands that were originally suggested:

sudo apt -f install
sudo apt autoremove
sudo apt update && sudo apt upgrade

Now my installation is all up to date and ready to rock and roll!

The moral of the story is: make sure to keep your Ubuntu installations up to date! Automating this process is ideal!

🙂 Cole

Prevent XML-RPC Brute Force Attacks – WordPress on Ubuntu 16.04

Overview

In my foray into cloud hosting, I’ve noticed that a few of our servers started to peak in their CPU usage more than what normal web traffic would cause(2-5% and 10-30% on our t2.medium and t2.small AWS EC2 servers respectively). After looking at the Apache2 logs, I found that there was a significant number of hits trying to look for combinations of phmyadmin, db, sql, and many other url keywords. This is obviously bothersome, but we’re all secure, so it didn’t cause much of an issue.

What did cause worry was the sheer number post requests to the /xmlrpc.php file. The IP addresses appear to be located in russia, and there were a handful different IP addresses that all had very similar origins. I’ve obscured the starts, but the IP range was ***.***.204.7-12. There is obviously enough traffic to blip our CPU usage on the medium server and enough to task the small server in a big way. To be more exact, there were ~78,000 requests in the log originating from those IP addresses over the past few days.

Why Is This Happening To Me?

There are many reasons hackers would like to gain access to your hosting server. One of the major reasons may be to steal resources to mine or farm bitcoin or some other cryptocurrency.

XML-RPC allows for very efficient brute force hacking, as it allows for hackers to check many username/password combinations in one single request. This is all at the cost of giving developers access to remote procedure calls.

XML-RPC Solution: Apache2 Deny From All

As I don’t use WordPress mobile, or other applications/plugins that require external access, I’ll be disabling access from external addresses.

The simplest solution to blocking this type of traffic on an Apache2 server is to simply deny access to the /xmlrpc.php file itself. To do this, you just have to add a few lines to your site configuration files in Apache2.

sudo vim /etc/apache2/sites-available/*yoursitehere*.conf

Your VirtualHost configuration should look something like the example below after adding the bolded lines.

<VirtualHost>
#Some stuff
    <files xmlrpc.php>
      order deny,allow
      deny from all
      # you can allow from your own IPs using the following line
      # allow from ###.###.###.###
    </files>
</VirtualHost>

Finally restart Apache2 and you’re on your way to preventing hackers from eating up your server availability.

sudo service apache2 restart

XML-RPC Solution: Security Plugins

Note: This may be a better solution for those of you who wish to use applications that require XML-RPC to function properly.

There are some security plugins that will either deny access to the file, blog IPs that are abusing the service, or will disable XML-RPC altogether. Jetpack is one of those security plugins that will aid in blocking brute force attacks. WordFence and it’s firewall are really great for blocking IP addresses that are abusing your servers with irrelevant traffic.

Conclusion

If you do decide to utilize a security plugin, keep in mind that their scans and extra filtering procedures may have an adverse effect on your servers speed and responsiveness as well. This is sometimes a necessary evil, but you should plan ahead accordingly.

Blocking access to certain system files may prevent you from accessing certain features. However, you may find your site security more important than whatever feature that may be.

After having denied access to the file, we’re back down to our low utilization on the t2.medium server of <1%. I’m glad to see my CPU Credit Balance stabilize!