Archives

Whitelist Google, Bing, Yahoo, Yandex, Baidu bots in csf and mod_security

ConfigServer Security & Firewall or csf for short is a popular firewall solution for cPanel servers. Combined with some good rules for mod_security, it does a great job.
To prevent csf temporary/permanently blocking the IPs of good bots you should edit the file /etc/csf/csf.rignore

####################### ##########################
# Copyright 2006-2017, Way to the Web Limited
# URL: http://www.configserver.com
# Email: sales@waytotheweb.com
####################### ##########################
# The following is a list of domains and partial domain that lfd process
# tracking will ignore based on reverse and forward DNS lookups. An example of
# its use is to prevent web crawlers from being blocked by lfd, e.g.
# .googlebot.com and .crawl.yahoo.net
#
# You must use either a Fully Qualified Domain Name (FQDN) or a unique ending
# subset of the domain name which must begin with a dot (wildcards are NOT
# otherwise permitted)
#
# For example, the following are all valid entries:
# www.configserver.com
# .configserver.com
# .configserver.co.uk
# .googlebot.com
# .crawl.yahoo.net
# .search.msn.com
#
# The following are NOT valid entries:
# *.configserver.com
# *google.com
# google.com (unless the lookup is EXACTLY google.com with no subdomain
#
# When a candidate IP address is inspected a reverse DNS lookup is performed on
# the IP address. A forward DNS lookup is then performed on the result from the
# reverse DNS lookup. The IP address will only be ignored if:
#
# 1. The results of the final lookup matches the original IP address
# AND
# 2a. The results of the rDNS lookup matches the FQDN
# OR
# 2b. The results of the rDNS lookup matches the partial subset of the domain
#
# Note: If the DNS lookups are too slow or do not return the expected results
# the IP address will be counted towards the blocking trigger as normal
#

Add the following lines to /etc/csf/csf.rignore file:

.googlebot.com
.crawl.yahoo.net
.search.msn.com
.google.com
.yandex.ru
.yandex.net
.yandex.com
.crawl.baidu.com
.crawl.baidu.jp

csf is blocking IPs when a host is blocked for a number of times by a mod_security rule. So, we must go to the root of the problem – we will create mod_security rules to allow good bots.
For this, we will edit the mod_security .conf files. If you are using cPanel EasyApache 4, add the following lines to the file /etc/apache2/conf.d/modsec/ modsec2.user.conf

HostnameLookups On
SecRule REMOTE_HOST "@endsWith .googlebot.com" "allow,log,id:5000001,msg:'googlebot'"
SecRule REMOTE_HOST "@endsWith .google.com" "allow,log,id:5000002,msg:'googlebot'"
SecRule REMOTE_HOST "@endsWith .search.msn.com" "allow,log,id:5000003,msg:'msn bot'"
SecRule REMOTE_HOST "@endsWith .crawl.yahoo.net" "allow,log,id:5000004,msg:'yahoo bot'"
SecRule REMOTE_HOST "@endsWith .yandex.ru" "allow,log,id:5000005,msg:'yandex bot'"
SecRule REMOTE_HOST "@endsWith .yandex.net" "allow,log,id:5000006,msg:'yandex bot'"
SecRule REMOTE_HOST "@endsWith .yandex.com" "allow,log,id:5000007,msg:'yandex bot'"
SecRule REMOTE_HOST "@endsWith .crawl.baidu.com" "allow,log,id:5000008,msg:'baidu bot'"
SecRule REMOTE_HOST "@endsWith .crawl.baidu.jp" "allow,log,id:5000009,msg:'baidu bot'"

After adding these lines, please restart the Apache Web Server. After some time, you will see entries in the server logs. Just go to WHM->Security Center->ModSecurity™ Tools->Hits List or from the command line:

root@web [/]# grep "500000" /usr/local/apache/logs/error_log | tail -30

 

cPanel Mod_Security Logs
cPanel Mod_Security Logs

Resources:
https://webmasters.googleblog.com/2006/09/how-to-verify-googlebot.html
https://yandex.com/support/webmaster/robot-workings/check-yandex-robots.xml
https://www.bing.com/webmaster/help/how-to-verify-bingbot-3905dc26
https://github.com/SpiderLabs/ModSecurity/wiki/

Share this post:

International Women’s Day 2017 Google doodle

Today is the International Women’s Day. For this year Google presents 13 of the past doodles honoring great women in the history.
“This International Women’s Day, we’ve chosen to look to the past to celebrate some of the female pioneers who paved the way to where we are today.” says the Google Doodle post.

You can see below the carousel that appears when you search for “International Women’s Day” on google.com website, as of March 8, 2017.

Here are the thirteen images shared by Google (we linked each woman name with her corresponding Wikipedia page):

Ida Wells

Lotfia El Nadi

Frida Kahlo

Lina Bo Bardi

Olga Skorokhodova

Miriam Makeba

Sally Ride

Halet Çambel

Ada Lovelace

Rukmini Devi

Cecilia Grierson

Lee Tai-young

Suzanne Lenglen

You can read more details at the International Women’s Day 2017.

We wish you a Happy Women’s Day!

Share this post: