Resource Warning from our hosting services

[Anonymous]

I received a “resource warning” email from in motion hosting and I forwarded it to Nibbler but felt that I should also post about it here.

Quote:

Hello Roy,

We have been tracking unusually high resource usage coming from your account and wanted to reach out to you before it becomes a larger problem. We only send this message to you when we see a large amount of resources consumed in a very short amount of time, and even then we will ignore such an event if it is rare for your account.

When your usage is severe enough to begin affecting other customers we may need to suspend the account or ask you to upgrade to a higher plan until this is resolved; however, usually when we see this sort of behavior there are a few relatively simple solutions you can take to better optimize your site for resource usage.

Attached we have included some metrics we can use to troubleshoot research usage. This includes some data on the amount of hits we see from your access logs as well as some CPU usage data we collect from the server. If you have a common content management system like WordPress or Joomla, we will also include some data about those installations as well. If we can determine a common pattern in this data, we have included some automated suggestions to address the issues in each section. We also have more information here from our Support Center:

https://www.inmotionhosting.com/support/account-management/resource-overage

Please note: if nothing is immediately clear to you, please do not hesitate to reply back to this message and our Systems team will be happy to take a further look at the issue and provide further assistance.

Resource Information

Top Process for stayld5

/usr/local/cpanel/3rdparty/bin/perl /usr/local/cpanel/3rdparty/bin/awstats.pl -configforum.staylds.com -LogFile/etc/apache2/logs/domlogs/forum.staylds.com-ssl_log.bkup -update

/opt/cpanel/ea-php74/root/usr/bin/php-cgi /home/stayld5/public_html/staylds/forum/viewtopic.php

/opt/cpanel/ea-php74/root/usr/bin/php-cgi /home/stayld5/public_html/staylds/forum/viewtopic.php

Time Percent Usage Most CPU Used Longest-Running Process

0:00 – 03:00 22.51 php-cgi php-cgi

3:00 – 06:00 26.04 php-cgi php-cgi

6:00 – 09:00 28.01 php-cgi php-cgi

Recent 23.44

Information Parsed from: /var/log/apache2/domlogs/stayld5/forum.staylds.com-ssl_log for the last 24 hours Hourly hits and response codes

Hour Hits

5 3868

6 5816

7 5726

8 5806

9 5962

10 5116

11 4130

Response Code Hits

200 35995

302 9

304 2

403 157

404 130

406 4

503 10

508 117

Duplicate requests

Hits Response Access Location

104 200 GET /styles/prosilver/theme/stylesheet.css?assets_version=36

103 200 GET /styles/prosilver/theme/en/stylesheet.css?assets_version=36

102 200 GET /styles/prosilver/template/ajax.js?assets_version=36

101 200 GET /styles/prosilver/theme/forms.css?hash=b64464fb

101 200 GET /styles/prosilver/theme/icons.css?hash=64da33ce

101 200 GET /styles/prosilver/theme/utilities.css?hash=d8f72c42

100 200 GET /assets/css/font-awesome.min.css?assets_version=36

100 200 GET /assets/javascript/core.js?assets_version=36

100 200 GET /styles/prosilver/theme/base.css?hash=7c5543be

100 200 GET /styles/prosilver/theme/colours.css?hash=fcb2f289

Requests for non-static content

Hits Response Access Location Possible Solution

17661 200 GET /viewtopic.php

10106 200 GET /posting.php

4436 200 GET /ucp.php

457 200 GET /search.php

400 200 GET /download/file.php

318 200 GET /memberlist.php

145 200 GET /viewforum.php

104 403 GET /posting.php

95 200 GET /assets/fonts/fontawesome-webfont.woff2

85 200 GET /

Top User Agents

Hits Agent Potential Solution

20492 “facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)” Block Unwanted Users

8469 “Mozilla/5.0 (compatible; AhrefsBot/7.0; +http://ahrefs.com/robot/)” Block Unwanted Users

1902 “Mozilla/5.0 (compatible; SemrushBot/7~bl; +http://www.semrush.com/bot.html)” Set Crawl Delay

743 “Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/125

618 “Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like

423 “Mozilla/5.0 (iPhone; CPU iPhone OS 17_5_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, lik

354 “Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/1

307 “Python/3.10 aiohttp/3.9.3”

281 “Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like

262 “Mozilla/5.0 (compatible; DotBot/1.2; +https://opensiteexplorer.org/dotbot; help@moz.com)” Set Crawl Delay

Top IPs

Hits IP Reverse DNS Potential Solution

646 66.249.68.3 crawl-66-249-68-3.googlebot.com. Rate Limit Google Bots

460 51.222.253.4 proxy-ca003-ext2.a.ahrefs.com.

457 51.222.253.10 proxy-ca009-ext2.a.ahrefs.com.

441 51.222.253.11 proxy-ca010-ext2.a.ahrefs.com.

439 51.222.253.20 proxy-ca019-ext2.a.ahrefs.com.

435 51.222.253.14 proxy-ca013-ext2.a.ahrefs.com.

434 51.222.253.1 proxy-ca000-ext2.a.ahrefs.com.

432 51.222.253.7 proxy-ca006-ext2.a.ahrefs.com.

431 51.222.253.17 proxy-ca016-ext2.a.ahrefs.com.

429 51.222.253.5 proxy-ca004-ext2.a.ahrefs.com.

Best regards,

InMotion Hosting

https://www.inmotionhosting.com/contact

Available 24/7 via phone, chat, or email, and check out our customer-exclusive knowledge base:

https://secure1.inmotionhosting.com/amp/support

Additionally, our Support Center contains thousands of helpful articles and guides:

https://www.inmotionhosting.com/support

[Anonymous]

Thanks Roy.

I mentioned in the email (and posting here for everyone’s benefit)…

Have you ever logged into StayLDS and it took a very long time to load a page? That’s happened to me a lot. I jumped on inmotion’s portal and found one bot in particular (AhrefsBot) was hitting the site hard. I dismissed it because I figured it would be okay for a bot to mine StayLDS for search results or AI.

I blocked that bot and SemrushBot, which I also saw in the logs. [nerd]I used the robots.txt method, which both bots report that they honor.[/nerd]

I can’t say for sure that was the only problem.

I see in the logs inmotion sent that the main offender is “facebookexternalhit/1.1”. I’ve read that they don’t honor robots.txt blocks, so I’ll have to find another way.

[Anonymous]

I also banned user agent “facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)” using .htaccess.

That was a new thing to me, so not sure if it worked.

I’ll try to monitor the logs from time to time to see if traffic from those bad acting agents has gone away.

[Anonymous]

[attachment=0]bot.PNG[/attachment]

To give you an idea of how much traffic bots are soaking up. I tried to address the top three offenders.

facebookexternalhit (.htaccess)

AhrefsBot (robots.txt)

SemrushBot (robots.txt)

Immediately after setting up the blocks, the site statistics are still consistently showing a very high CPU usage, which is troubling.

I hear it can take several hours for the changes to take effect. The one that absolutely has to go is facebookexternalhit and unfortunately it has to be taken care of using a method I’m not very familiar with.

I’ll give it some time for the new rules to settle in and see where we’re at.

[Anonymous]

From everything I’ve read, the facebook bot is particularly aggressive. It doesn’t appear to honor robots.txt and finds a way to circumvent .htaccess.

I’m going to ban their IP ranges. The trouble is that it may inadvertently sweep up others in the process, so I’m posting the ranges here so I’ll know to go back and remove them once the facebook bot takes a hint.

173.252.64.0-173.252.127.255

69.171.224.0-69.171.255.255

51.222.253.

I’m sure there will be more. These buggers are pervasive.

[Anonymous]

See also:

viewtopic.php?t=10345” class=”bbcode_url”>viewtopic.php?t=10345

[Anonymous]

Thanks for all of this. I do occasionally have some issues with very slow loading. I had chalked it up to just slow connection on my end even though my fiber is usually quick. I don’t recall which day but I did have some trouble one day this week.

It looks like you might have things in hand, but I would suspect that InMotion may have encountered something like this in the past. Do you think they might assist with this particularly pernicious bot?

[Anonymous]

Thanks, Roy and Nibbler. I am WAY out of my league with this stuff. I hate that this sort of stuff is part of our modern world, but I am grateful for your much better knowledge than mine.

[Anonymous]

Thank you Nibbler!

[Anonymous]

The efforts appear to have worked. Here’s a graph of the bandwidth usage.

[attachment=0]bandwidth.PNG[/attachment]

The challenge is that these bots jump from IP to IP so we’ll have to be vigilant. That and there are a few more bots out there that I could look into banning.

[Anonymous]

Great job! Thanks again. What would we do without you?

I also wonder if this is related to the 2-3 new member requests we’ve been getting per week where the justification is always the copied “*Your activation is contingent….”? If it’s AI, it’s not learning.

[Anonymous]

Yes, I’ve seen lots of those lately. I’ve taken to deleting them without investigating. They all have strange email addresses too.

[Anonymous]

I just delete them as well. I’ve done blocking IP address here and there, but my guess is the IPs are spoofed or some sort of VPN. They seem to generally have believable usernames, but beyond that they’re universally off.

[Anonymous]

Thanks, o tech magician.

[Author]

Posts