Website slow-CPU 100% capacity- (1)

About 2 years ago MPH Creative were recommended to a new client that was experiencing very poor website performance. The site was slow when navigating or updating and was continually crashing, more often than not the site would not load at all and time out.

The Problem

The web agency they had in place appeared to be clueless and didn’t know how to solve the problem and communications eventually broke down completely. It was a risk for us to get involved considering the previous agency had built the custom WordPress site as well as hosted it and still didn’t know how to resolve the problem. The company was in desperate need of a trustworthy agency and so with reservation, we agreed to help. Sometimes, diagnosing a website that we haven’t built ourselves can be like finding a needle in a very large haystack.

The first task was to move the hosting away from the web agency. Once the site had moved, all appeared to operate normally, speed was good, we had no crashes and the client was happy – but surely it couldn’t be that simple?

After a month or two, the site was still running well and the problems were ultimately put down to poor hosting… but did we speak too soon?

Another month later and the old problems started to resurface, the site slowed and performance dropped rapidly which eventually led to the site being down more than up.

The client and hosting company came back to us reporting the issues, so we went through the process of ensuring all the obvious things were updated. But again, other than a load of ridiculously large images uploaded to the system nothing major jumped out. All the plugins, WordPress version and theme were updated and all operated as they should.

We advised the hosting company to increase the specification of the server as the site did appear to have a decent level of traffic but nothing out of the norm. This appeared to do the job for another month or so, until we received a panicked call from the client that the site had once again gone down and showed an ‘error establishing database connection’.

The hosting company was less than helpful, and we were sure the site didn’t have any fundamental problems that would reoccur sporadically. If a site doesn’t work, it doesn’t work, very rarely a site will keep developing issues for no apparent issue. Once again, we did all we thought possible on our side and the client eventually lost patience with the lack of assistance from the hosting company who just kept pointing the finger at us and reiterating the same query that the CPU was at 100% capacity.

The client changed hosting companies yet again and on changing, like magic, all appeared calm and the site went back to normal, leaving us to believe this was again a hosting issue. I’m not a lover of hosting companies at the best of times and have the opinion that many people do on broadband and mobile phone companies. They promise you the world but once you’ve signed on the dotted line and have your money they couldn’t care less.

Following the third change in hosting company, we had even less time than before when the site crashed and was almost inoperable.

We were sure the site itself wasn’t the cause, so we did some in-depth research online to see if others were having similar problems. We were amazed to see that this is a global problem with hundreds of discussions, forums and cries for help from website owners and the issues were exactly the same:

  • High CPU usage at 100% capacity
  • Site extremely slow
  • Site timing out
  • Unable to access WordPress admin
  • Hosting company providing no advice or help
  • Hosting company blaming the site build and to check plugins

After reading many posts and applying possible solutions – none of which worked, we then focused on outside influences. According to Google Analytics, traffic was steady with nothing alarming that would pull the site down, especially as the site was on a dedicated high spec server. There was a fair bit of referral traffic pinging the site that reflected quite a high bounce rate so we blocked these IP addresses in the site code. This halved the GA results but still the site was extremely slow.

 

DDoS-Attacker-CPU-chart

DDoS Attack and How it works

 

The next step was to test to see if the site was being attacked through more hidden, malicious methods. Accessing the site database, we applied a query code to reveal all traffic hitting the site. In an instant, it was black and white what was causing the trouble. The site was being massively hit by a sustained automated DDoS attack. Over just a few hours, the site had been hit over 45,000 times which was crippling CPU and causing the 100% overload pulling the site down.

 

Why does this not show in Google Analytics?

The attack avoids being tracked by Google libraries as Google relies on javascript to perform it’s tracking; since this would not have been run by these requests it did not show up in the GA metrics.

 

The Solution

From here we isolated the IP addresses that hit the site almost every second or more and applied a short piece of code to block these IP addresses. And wham, bam thank you, mam, the site was back and fully operational.

Now that we’d unearthed the problem, a concern dawned on us. Every time we’d changed hosting the automated attack lost the IP address of our client’s site and the attacked ceased, but within months the attackers had relocated the site and the attacks started again.

This was obviously a malicious and targeted attack put in place to bring our clients website down. For what reason we do not know, they provide a positive and worthwhile service so to attack such a company simply makes no sense.

If you want to check to see if your site is being targeted by DDoS attacks you can follow these steps:


 

Step 1

Via FTP connect to your website and locate the functions.php file. This is usually in wp-content/themes/*YOUR THEME NAME*/

Add the following code to the top of the functions page:
$wpdb->query( $wpdb->prepare("INSERT INTO temp_log
(user_agent, their_address, they_requested)
VALUES (%s, %s, %s)
",array($_SERVER ['HTTP_USER_AGENT'],$_SERVER['REMOTE_ADDR'],$_SERVER['REQUEST_URI'] )) );

Then save and re-upload.


 

Step 2

We then made some changes to log the traffic, and left it for this log to build up enough information to provide a useful profile. We then ran a query to extract anomalous traffic.

Access the database, if you don’t have direct access, alternatives like phpMyAdmin are usually available. Once you’ve done that click on the tab at the top that says “SQL” and create the following table:

 

CREATE TABLE ‘temp_log’.’temp_log’ (
`id` INT NOT NULL AUTO_INCREMENT,
`they_requested` VARCHAR(255) NOT NULL,
`user_agent` VARCHAR(255) NOT NULL,
`when_they_visited` TIMESTAMP NOT NULL,
`their_address` VARCHAR(50) NOT NULL,
PRIMARY KEY (`id`) );

 


 

Step 3

Next, click the “Query” tab and input the following:
SELECT *FROM temp_log ORDER BY id desc


 

Step 4

A list will appear presenting the IP address and times they hit the site
If you are being heavily attacked there will be a clear culprit as the IP will appear multiple times, possibly every second or so.

 


 

Step 5

If the page is being visited by one of the suspect IPs we have identified, we need to halt execution. We want to do this fairly early on, so we add the following code to the top of the wp-config.php file:
if (in_array($_SERVER['REMOTE_ADDR'], [IP ADDRESS 1, IP ADDRESS 2])) {
header("HTTP/1.1 503");
die();
}

The IPs take the form of a comma separated list, with each IP surrounded by quotes. The header is to inform the visitor of the nature of the response; theoretically, any code could be supplied, but 503 seemed appropriate. We then have the “die” since the program execution is terminating abnormally.


 

Result:

CPU usage should have dramatically reduced and site performance vastly increased.

What if the DDoS attack starts again?
Apply the same process to block the attackers and consider using a third partly buffer like Cloudflare (www.cloudflare.com) to block DDoS attacks. The first level package is free so well worth giving this a go.

 

 

 

If you’re not sure how to apply the above steps, but you think this might be the problem with your site, feel free to get in touch and we will do our best to help.

Comments