- This topic is empty.
-
Posts
-
I received a “resource warning” email from in motion hosting and I forwarded it to Nibbler but felt that I should also post about it here. Quote:Hello Roy,
We have been tracking unusually high resource usage coming from your account and wanted to reach out to you before it becomes a larger problem. We only send this message to you when we see a large amount of resources consumed in a very short amount of time, and even then we will ignore such an event if it is rare for your account.
When your usage is severe enough to begin affecting other customers we may need to suspend the account or ask you to upgrade to a higher plan until this is resolved; however, usually when we see this sort of behavior there are a few relatively simple solutions you can take to better optimize your site for resource usage.
Attached we have included some metrics we can use to troubleshoot research usage. This includes some data on the amount of hits we see from your access logs as well as some CPU usage data we collect from the server. If you have a common content management system like WordPress or Joomla, we will also include some data about those installations as well. If we can determine a common pattern in this data, we have included some automated suggestions to address the issues in each section. We also have more information here from our Support Center:
https://www.inmotionhosting.com/support/account-management/resource-overage Please note: if nothing is immediately clear to you, please do not hesitate to reply back to this message and our Systems team will be happy to take a further look at the issue and provide further assistance.
Resource Information
Top Process for stayld5
/usr/local/cpanel/3rdparty/bin/perl /usr/local/cpanel/3rdparty/bin/awstats.pl -configforum.staylds.com -LogFile/etc/apache2/logs/domlogs/forum.staylds.com-ssl_log.bkup -update
/opt/cpanel/ea-php74/root/usr/bin/php-cgi /home/stayld5/public_html/staylds/forum/viewtopic.php
/opt/cpanel/ea-php74/root/usr/bin/php-cgi /home/stayld5/public_html/staylds/forum/viewtopic.php
Time Percent Usage Most CPU Used Longest-Running Process
0:00 – 03:00 22.51 php-cgi php-cgi
3:00 – 06:00 26.04 php-cgi php-cgi
6:00 – 09:00 28.01 php-cgi php-cgi
Recent 23.44
Information Parsed from: /var/log/apache2/domlogs/stayld5/forum.staylds.com-ssl_log for the last 24 hours Hourly hits and response codes
Hour Hits
5 3868
6 5816
7 5726
8 5806
9 5962
10 5116
11 4130
Response Code Hits
200 35995
302 9
304 2
403 157
404 130
406 4
503 10
508 117
Duplicate requests
Hits Response Access Location
104 200 GET /styles/prosilver/theme/stylesheet.css?assets_version=36
103 200 GET /styles/prosilver/theme/en/stylesheet.css?assets_version=36
102 200 GET /styles/prosilver/template/ajax.js?assets_version=36
101 200 GET /styles/prosilver/theme/forms.css?hash=b64464fb
101 200 GET /styles/prosilver/theme/icons.css?hash=64da33ce
101 200 GET /styles/prosilver/theme/utilities.css?hash=d8f72c42
100 200 GET /assets/css/font-awesome.min.css?assets_version=36
100 200 GET /assets/javascript/core.js?assets_version=36
100 200 GET /styles/prosilver/theme/base.css?hash=7c5543be
100 200 GET /styles/prosilver/theme/colours.css?hash=fcb2f289
Requests for non-static content
Hits Response Access Location Possible Solution
17661 200 GET /viewtopic.php
10106 200 GET /posting.php
4436 200 GET /ucp.php
457 200 GET /search.php
400 200 GET /download/file.php
318 200 GET /memberlist.php
145 200 GET /viewforum.php
104 403 GET /posting.php
95 200 GET /assets/fonts/fontawesome-webfont.woff2
85 200 GET /
Top User Agents
Hits Agent Potential Solution
20492 “facebookexternalhit/1.1 (+
http://www.facebook.com/externalhit_uatext.php )” Block Unwanted Users8469 “Mozilla/5.0 (compatible; AhrefsBot/7.0; +
http://ahrefs.com/robot/ )” Block Unwanted Users1902 “Mozilla/5.0 (compatible; SemrushBot/7~bl; +
http://www.semrush.com/bot.html )” Set Crawl Delay743 “Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/125
618 “Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like
423 “Mozilla/5.0 (iPhone; CPU iPhone OS 17_5_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, lik
354 “Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/1
307 “Python/3.10 aiohttp/3.9.3”
281 “Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like
262 “Mozilla/5.0 (compatible; DotBot/1.2; +
https://opensiteexplorer.org/dotbot ;help@moz.com )” Set Crawl DelayTop IPs
Hits IP Reverse DNS Potential Solution
646 66.249.68.3 crawl-66-249-68-3.googlebot.com. Rate Limit Google Bots
460 51.222.253.4 proxy-ca003-ext2.a.ahrefs.com.
457 51.222.253.10 proxy-ca009-ext2.a.ahrefs.com.
441 51.222.253.11 proxy-ca010-ext2.a.ahrefs.com.
439 51.222.253.20 proxy-ca019-ext2.a.ahrefs.com.
435 51.222.253.14 proxy-ca013-ext2.a.ahrefs.com.
434 51.222.253.1 proxy-ca000-ext2.a.ahrefs.com.
432 51.222.253.7 proxy-ca006-ext2.a.ahrefs.com.
431 51.222.253.17 proxy-ca016-ext2.a.ahrefs.com.
429 51.222.253.5 proxy-ca004-ext2.a.ahrefs.com.
Best regards,
InMotion Hosting
https://www.inmotionhosting.com/contact Available 24/7 via phone, chat, or email, and check out our customer-exclusive knowledge base:
https://secure1.inmotionhosting.com/amp/support Additionally, our Support Center contains thousands of helpful articles and guides:
Thanks Roy. I mentioned in the email (and posting here for everyone’s benefit)…
Have you ever logged into StayLDS and it took a very long time to load a page? That’s happened to me a lot. I jumped on inmotion’s portal and found one bot in particular (
AhrefsBotI blocked that bot and SemrushBot, which I also saw in the logs. [nerd]I used the robots.txt method, which both bots report that they honor.[/nerd]
I can’t say for sure that was the only problem.
I see in the logs inmotion sent that the main offender is “facebookexternalhit/1.1”. I’ve read that they don’t honor robots.txt blocks, so I’ll have to find another way.
I also banned user agent “facebookexternalhit/1.1 (+ http://www.facebook.com/externalhit_uatext.php )” using .htaccess.That was a new thing to me, so not sure if it worked.
I’ll try to monitor the logs from time to time to see if traffic from those bad acting agents has gone away.
[attachment=0]bot.PNG[/attachment] To give you an idea of how much traffic bots are soaking up. I tried to address the top three offenders.
facebookexternalhit (.htaccess)
AhrefsBot (robots.txt)
SemrushBot (robots.txt)
Immediately after setting up the blocks, the site statistics are still consistently showing a very high CPU usage, which is troubling.
I hear it can take several hours for the changes to take effect. The one that absolutely has to go is facebookexternalhit and unfortunately it has to be taken care of using a method I’m not very familiar with.
I’ll give it some time for the new rules to settle in and see where we’re at.
From everything I’ve read, the facebook bot is particularly aggressive. It doesn’t appear to honor robots.txt and finds a way to circumvent .htaccess. I’m going to ban their IP ranges. The trouble is that it may inadvertently sweep up others in the process, so I’m posting the ranges here so I’ll know to go back and remove them once the facebook bot takes a hint.
173.252.64.0-173.252.127.255
69.171.224.0-69.171.255.255
51.222.253.
I’m sure there will be more. These buggers are pervasive.
See also: viewtopic.php?t=10345viewtopic.php?t=10345” class=”bbcode_url”> Thanks for all of this. I do occasionally have some issues with very slow loading. I had chalked it up to just slow connection on my end even though my fiber is usually quick. I don’t recall which day but I did have some trouble one day this week. It looks like you might have things in hand, but I would suspect that InMotion may have encountered something like this in the past. Do you think they might assist with this particularly pernicious bot?
Thanks, Roy and Nibbler. I am WAY out of my league with this stuff. I hate that this sort of stuff is part of our modern world, but I am grateful for your much better knowledge than mine. Thank you Nibbler! The efforts appear to have worked. Here’s a graph of the bandwidth usage. [attachment=0]bandwidth.PNG[/attachment] The challenge is that these bots jump from IP to IP so we’ll have to be vigilant. That and there are a few more bots out there that I could look into banning.
Great job! Thanks again. What would we do without you? I also wonder if this is related to the 2-3 new member requests we’ve been getting per week where the justification is always the copied “*Your activation is contingent….”? If it’s AI, it’s not learning.
Yes, I’ve seen lots of those lately. I’ve taken to deleting them without investigating. They all have strange email addresses too. I just delete them as well. I’ve done blocking IP address here and there, but my guess is the IPs are spoofed or some sort of VPN. They seem to generally have believable usernames, but beyond that they’re universally off. Thanks, o tech magician.
- You must be logged in to reply to this topic.