CleanTalk's blog

    Sign up

    Category: CleanTalk

    • CleanTalk Malware Scanner — heuristic code analysis

      CleanTalk Malware Scanner — heuristic code analysis

      We have already talked about the launch of security service for WordPress in the previous article. Today we want to talk about the launch of heuristic analysis to detect malicious code.

      The very presence of malicious code can lead to a ban in search results or a warning in the search for that the site is infected, to protect users from potentially dangerous content.

      You can find malicious code on your own, but it’s a lot of work and most WordPress users do not have the necessary skills to find and remove unnecessary lines of code.

      Often, the authors of malicious code disguise it, which makes it difficult to determine by its signatures. The malicious code itself can be located anywhere on the site, for example the obfuscated PHP-code in the logo.png file, and the code itself is called by one inconspicuous line in index.php. Therefore, the use of plugins to search for malicious code is preferable.

      CleanTalk on the first scan scans all WordPress kernel files, plugins and themes. When rescanning, only those files that have changed since the last scan were scanned. This saves resources and increases scanning speed.

      How heuristic analysis works

      One of the main disadvantages of heuristic analysis is that it is quite slow, so we use it only when it is really necessary. First of all, we divide the source code into lexemes (the minimal language construct) and remove all unnecessary:

      1. Space symbols.
      2. Comment of different types.
      3. Not PHP code (outside of tags <?php ?> )

      Next, we recursively simplify the code until there are no “complex constructs”:

      1. Perform concatenation of strings.
      2. Substitution of variables into variables.
      3. and other

      Also, in the process of simplifying the code, we monitor the origin of the variables and many others.

      In the end, we get a clean code that can be analyzed. It is very important that we get the code not in the form of a string, but in the form of lexemes. Thus, we know where the lexeme is a string with the desired text, and where the lexeme function is.

      In the sense of finding “bad constructs” eval for us there is a difference:

      <?php echo 'eval("echo \"some\"")'; ?>

       

      — in this case there will be no lexeme T_EVAL,

      there is a lexeme T_CONSTANT_ENCAPSED_STRING ‘eval (“echo \” eval\”)’

      <?php eval('echo "some"'); ?>

      – and here it is. And this is the version we will find.

      We look for such constructs, we break them down into degrees of criticality:

      Critical:

      • eval
      • include* и require*
        • with bad file extension
        • non-existent files (will be deleted in the next  versions)
        • connecting deleted files

      Dangerous

      • system
      • passthru
      • proc_open
      • exec
      • include* и require*
        • with the error suppression operator (will be deleted in the next versions)
        • with variables depending on POST or GET.

      Suspicious

      • base64_encode
      • str_rot13
      • syslog

      And other.

      We are constantly improving this analysis: adding new constructions to search, reducing the number of false alarm, optimize the simplification of the code.

      In the plans to teach it to detect and decode strings encoded in URL and BASE64 and others.

      The plugin itself is available in the WordPress directory.

    • Anti-Spam Filter for IP Networks

      Dear customers!

      CleanTalk has expanded the functions of personal blacklists.

      We’ve just added a support in your private blacklists to block separate IP networks.

      This will allow you to use the service to block IP networks that use spammers. Very often spammers buy servers from hosting companies or virtual servers and use them to send a spam. So, your website hasn’t to receive a comments/registrations from hosting IPs because a real people never use their IPs. There can be only XML-PRC requests but it is not important because this protocol is using for other attacks such as brute-force and DDoS via XML-RPC pingbacks.

      The instruction of how to add entries to your personal blacklists can be found here:
      https://cleantalk.org/help/I-want-to-block-ip

    • Additional factors for estimating spam activity of IP/Email addresses in Anti-Spam/Anti-Fraud API

      Cloud Anti-Spam by CleanTalk in the process collects data on spam activity of IP/Email addresses. On the basis of these data, a database of spam IP/email is formed. CleanTalk provides several API methods for working with the data we have.

      We have recently added new parameters to the spam activity test method and launched two new API methods:

      • the first is getting the country code letter by IP address
      • the second – checking domain for participation in spam mailings

      But about everything in order.

      Get the country code letter by IP address.

      This API method returns a two-letter country code (US, UK, CN, etc.) or the full name of Russia by IP address. Read more on the use of the method ip_info.

      Checking the domain to participate in spam mailings.

      Allows you to check whether this domain has been used in spam mailings. At this point in CleanTalk database contains records of 383 1 062 domains.

      For example, you have a comment, it is on the topic of the article, with a very meaningful text and does not raise suspicions, but contains a link to a third-party site. We check the links and get that the domain is listed in the database, the links were posted on other websites and the date of their placement approximately coincides with the date of the comment. It turns out that the comment was used to place a link to a third-party resource.

      Using this method is useful when recognizing manual spam when all other checks are completed.

      You can check the blacklist manually on the website

      Read more on the use of the method backlinks_check.

      Update the spam_check method.

      Method allows you to bulk check IP/email against a database of blacklists CleanTalk. At the moment the database contains records about 2 808 344 IP and 990 835 Email. Also, the method allows you to make an IP check for a specific date.

      In addition to the main parameter, as a result of the issuance, showing the presence or absence in the database, additional parameters have been added:

      spam_rate – spam activity rating from 0 to 100%. The parameter is calculated for each IP or email record as the ratio of blocked requests to the total number of requests from a given IP or email. As an example, IP has a total of 100 requests, of which the service blocked 97 requests as spam, so spam_rate will be 97%.

      Why this is important, it gives you the ability to set your own request blocking logic. As the spammers change the IP and at the moment it can be a regular user.

      Following parameter:

      Each parameter shows the activity of the data being checked for the last 10 minutes, 1 hour and 24 hours, respectively. That is, it shows the number of requests for a specific time from a specific IP or email. This is useful in cases where IP or email does not yet have BlackListed status in the database, but has a fairly high activity in a short time. About what other parameters you can use to protect against spam, you can read in our previous articles:

      Non-visual methods of protecting the site from spam Part 1

      Non-visual methods of protecting the site from spam Part 2

      Non-visual methods of protecting the site from spam Part 3

      Example of output of API data for test email *@*******lk.org. This email is for tests, so the “updated” parameter: “2019-03-28 22:07:19” is the date.

      {"data":
            {"*@*******lk.org":
                      {"appears":1,
                      "frequency_time_10m":null,
                      "spam_rate":"0",
                      "frequency":"999",
                      "frequency_time_24h":null,
                      "updated":"2019-03-28 22:07:19",
                      "frequency_time_1h":null}
            }
}

      Learn more about using the spam_check method.

    • Automating CleanTalk Anti-Spam Updates for WordPress

      Automating CleanTalk Anti-Spam Updates for WordPress

      If you serve a couple of sites, then updating the plugins does not cause any difficulties. Difficulties appear if you serve a few dozen, or even hundreds of sites.

      CleanTalk Anti-spam requires frequent updates (we have to release a new version every 1-2 weeks), there are many reasons for this.

      WordPress, as a designer, has a huge number of plugins, themes, widgets, etc. which are not always designed with WordPress Codex, have different architecture and event handling. Therefore, CleanTalk integration errors can occur with different components, especially rare ones.

      Each complex service that uses a large amount of data, changes backend, changes in logic and a lot of the rest, require changes in the plugin.

      At our update rates for the plugin, the auto-update option is required. If this option is done in the plugin’s settings, then the user will still need to go to each site in the plugin’s settings and install it. But what to do in a situation where the user does not want to include auto-update, but you need to update the plug-in immediately on one hundred sites?

      For the convenience of service management, the auto-update option was implemented in the Service Control Panel.

      Auto-update allows you to update the plugin one time at a single site, a group of sites, or enable auto-update on all sites.

      How it works

      Historically, we are monitoring the client version of the plugin, and when clients are contacted and feedback analysis, we need to know the version of the plugin that is being used. Each anti-spam plugin, with each request, sends its version in the parameters. The version number is compared with the number in the repository, and if the versions are different, then in the Control Panel we show a warning about the need for an update.

      Next, when clicking on a link, a modal window opens with options for updates.

      When the option is selected, the server makes a special request to the plugin on the client’s site.

      The plugin, having received the call, checks the parameters of the call for authenticity and starts work. Auto-update is implemented through a modified class of auto-update plugins WordPress. After the class is finished, the plugin checks the version of the updated files and makes a request to the site (itself). If the HTTP response code is 200, it reports this to the server by displaying it on the “OK” page and makes a special API call, reporting on the new version. If the response code is different from 200, the plugin does a rollback of the files to the previous version and responds north with a string with an error code and technical parameters.

      After a successful update, the status in the Service Control Panel changes to “App has been updated”.

      How to set up an auto-update

      Please, go to your CleanTalk Dashboard.

      • Choose a website that needs to update the plugin.
      • Click on the link Update app.
      • Next step, in pop-up you can choose:
        do a manual update and the plugin will be updated immediately. You can do this action for all websites
        or
        set auto-update, in the next time plugin will be updated automatically.

    • CleanTalk, the launch of WordPress security

      While developing the Anti-Spam service, we often encounter other issues related to the security of websites. The most common questions were about brute force attacks. In addition to problems with the selection of passwords for the administrator account, often brute force attacks cause a high load on the server, and users receive notification from the hosting about exceeding the allowed load values for the processor.

      We thought if we are receiving such requests, why don’t we solve them? Since tasks relate to security functions, the decision to launch a separate security service was obvious.

      At the moment, the Security service is developed only under WordPress, there are several reasons for this: the greatest demand, a large number of websites use this particular CMS, the complexity of the development of several CMS.

      Despite the fact that anti-spam protection is a part of security, we decided to split these two services. There are several reasons for this:

      1. Complication of the plugin, which leads to increased errors, compatibility issues with other plugins/themes
      2. Promotion by search queries
      3. Easier development and independent release of updates
      4. The interface of the plugin is not complicated by a bunch of additional options that are not needed if the user uses only one function
      5. A separate management interface and logging in the control panel CleanTalk

      We decided to start with the implementation of protection against brute force attacks and further gradually expand the functionality.

      Protection from brute force attacks – implemented by adding delays between incorrect authorization attempts. A delay of 3 seconds is set for the first attempts, for a subsequent one in 10 seconds. If there were 10 unsuccessful attempts of authorization within an hour, the IP address will be added to the FireWall database for 24 hours. To protect against hackers trying to find a password for your account, this is enough, since they significantly increase the time between attempts, and they can be tens or hundreds of thousands. All logs of access attempts are available in the weekly report and in the service control panel, which allows you to quickly add IP addresses to the FireWall blacklist. Protection against brute force attacks extends only to users with administrator rights.

      Traffic control – allows you to view information about visitors, such as:

      • IP
      • Country
      • Date/time of the last query
      • The number of allowed/blocked HTTP requests
      • Status-banned or allowed
      • The URL of the page visit
      • User Agent

      Another option in traffic Control — “Block visitor if the number of requests is greater than” – blocks access to the site for any IP that exceeds the number of HTTP requests per hour. The number of requests can be set in the settings, the default is 1000. If the IP is exceeded, the Firewall will be added to the Blacklist for 24 hours.

      This will help solve the problem of DoS attacks on the site when a large number of HTTP requests are sent to the site, because of which it stops responding or starts to work very slowly. This situation is possible because of a massive brute force attack.

      Audit log – allows you to monitor the actions of users in the admin WordPress, keeps a log of visits to pages with the date/time and length of stay. Allows you to monitor the actions of administrators and unauthorized access and in case of problems to understand where by whom and what changes have been made.

      Malware Scanner – scans WordPress files, plugins and themes for malicious code and changes. If the changes in the files were made illegally, it allows you to restore the original files.

      Automatic scanning takes place every 24 hours, and you can also start it manually.

      Security FireWall – blocks access to the site for POST/GET requests by IP addresses. Base IP addresses for the FireWall is generated from our database of blacklists CleanTalk. It is possible to get IP addresses that have a high spam activity or was seen in attempts brute force attacks. It is possible to use their own blacklists, both for individual IP addresses/subnets and by country. Due to this, it is possible to reduce the load on the website or to block a DOS attack.

      Ready to release:

      • outbound link scanner
      • checking links against a database of domains that are promoted with spam
      • protection from XSS and SQL injections

      Development notes

      Everything was written from scratch, not peeking at other solutions. This was done specifically to not to pick up other people’s mistakes and to develop your own vision for the application.

      Further development for other CMS is planned, so it was decided to develop a modular design. Use an object-oriented approach and everything like that. Of course, in the process had to solve various problems that do not fit into this concept and did not do without a workaround.

      As a result, there are several classes that without significant improvements can be used on other CMS (including self-recording), using a couple of wraps, for example for the database.

      Was written our own class Cron is not dependent on Cron WordPress. Still, the application for security and should not rely on functionality that may or may not work, or which may interfere with the work of third-party developers.

      To implement heuristic code analysis, we have written our own code minimizer parser, which will continue to develop. With it, you can track dangerous variables, functions, constructions. Not sure if other plugins/anti-viruses/applications use similar solutions (probably not), but this pros and cons of independent development, our approach may have turned out unique.

      Example of the “minimizer”:

      Source code:

      <?php
	//$some = 'n'.'o'.'t'
	$some = 's'.'o'.'m'.'e'; // String concatenation
	$stuff = 'stuff';
	
	$first = 'first';
	$func = 'func';
	
	$first_func = $some."$first$func"; // Variable replacement
?>
$some = 'n'.'o'.'t';
<?php
	// Variable replacement
	$i = 'i';
	$c = 'c';
	$o = 'o';
	$co = $c.
	// some obfuscating comment
	$o;
	$ico = $i/* some obfuscating comment */.$co;
	
	require($some.'_'.$stuff.'.'.$ico);
	require($some.'_'.$stuff.'.php');
	require($some.'_'.$stuff.'.p'.$ico);

	$first_func();
?>

      Result:

      <?php $some='some';$stuff='stuff';$first='first';$func='func';$first_func='somefirstfunc';$i='i';$c='c';$o='o';$co='co';$ico='ico';require'some_stuff.ico';require'some_stuff.php';require'some_stuff.pico';somefirstfunc();?>

      If you bring in a more understandable form:

      <?php 
	$some='some';
	$stuff='stuff';
	$first='first';
	$func='func';
	$first_func='somefirstfunc';
	$i='i';$c='c';$o='o';
	$co='co';
	$ico='ico';
	require'some_stuff.ico';
	require'some_stuff.php';
	require'some_stuff.pico';
	somefirstfunc();
?>

      Some things that it can do: do concatenation, substitute variables, track the origin of variables (let’s say if they use unreliable $ _POST and $ _GET), track and check the file connections (include, require) for various parameters and much more. We can say that this is the basis on which the functional will be added.

      Especially I did not like to support WPMS, because for each functional I had to make exceptions taking into account whether the main site is this, whether the user of the secondary site inherits the key from the main site or enters his own access key, whether the secondary site allowed to activate plug-ins and the like. Unfortunately, we had to remove part of the functionality for WPMS and secondary sites due to non-compatibility.

      In general, it turned out a beautiful application in places from the point of view of the code, which we will develop in the future.

      The plugin itself can be found in the directory.

    • 5 new anti-spam plugins from CleanTalk

      We continue blogging about our work and will talk about our work in it. To begin with, that will tell you about what we have done in 2017. Over the past year, we have developed several anti-spam modules for CMS, which I will describe in more detail.

      Why modules and not the API. First, they allow users to quickly and easily connect to the service. Second, not all users have the knowledge to connect the API. Third, the modules have a management interface, which makes it easier to use.

      A little about the service itself: CleanTalk is a cloud-based service to protect websites from spam, provides a simple and convenient form of comment/registration for visitors, which will not require the visitor to prove that he/she is a person that saves time and resources spent on moderation and verification of questionable users or comments. All requests are stored in the cloud, including blocked ones, which helps to prevent data loss.

      Additional features: SpamFireWall – “soft” blocking of POST and GET requests over IP and subnet masks (soft – if the user was mistakenly added, then after 1 second will be redirected to the page of the site).

      How it works: The anti-spam module installed on the website transmits the behavior parameters of the visitor, browser, IP / email and message text. These parameters are evaluated and the service decides whether to post the message or define it as spam and reject it. Based on these checks, the service creates its own list of IP / Email addresses used by spambots. In the blacklist are added not only to IP / Email, but also domains sites promoted through spam. All this happens automatically and does not require any action from the site administration.

      MODX Anti-Spam Plugin

      The module was developed at the request of several clients and provides protection from spam for registrations, comments, feedback forms.

      For the development of MODX, there is a fairly good documentation. For those who start developing and getting acquainted with MODX for the first time, it would be useful to add an example of creating the first simplest plugin (build your own first plugin) to the documentation, which greatly simplifies the process. The development process itself took 3-4 days, together with related tasks.

      Adding the module to the official catalog did not cause any difficulties, everything was quite simple and understandable. Moderation took about a week after sending the module, waited 5-6 days and wrote to the technical support to find out at what stage and how long to wait, and the next day the module was published. It is not known whether this is due to treatment or not.

      MyBB Anti-Spam Plugin

      There are no problems with the documentation, everything is clear and there are no questions. The same with the development.

      With the placement in the catalog, it is more difficult, it is necessary to understand the interface – it is not very convenient, but the worst thing is moderation of new plugins. Having sent the module for moderation in June 2017, we are still waiting for it to be published in the catalog. In general, the situation is similar to the directory phpBB, there also have to wait for months.

      We decided to follow the advice of one of the users and create a topic on the forum, in the plugins section, added a description, links to the module.

      OpenCart Anti-Spam Plugin

      With documentation for development under this CMS, there are problems, it almost is not present. Good documentation was found here, for which many thanks to the compilers. To develop, you need vQmod and understanding MVC. In the rest, there is nothing complicated.

      Quite a convenient interface for the marketplace, everything is clear and fast. Complexities with the addition did not arise.

      XenForo 2 Anti-Spam Plugin

      As for the developer documentation, even though XF2 is still a beta version, the documentation for it is one of the best. There were no difficulties with the development. The only thing is not entirely clear why automatic hash file creation (hashes.json) is done when the plugin is loaded and in the end each time you have to manually do this by the command.

      The interface of the marketplace is convenient. There is no plugin moderation before publishing, plugins are moderated after you publish the plugin. This is probably not very convenient, as the version may contain errors, while the plugin will have time to download. By our first version, we have received code comments and a notification that if we do not resolve, the plugin will be removed from the directory.

      Universal Anti-Spam Plugin

      Since the number of requests to connect the service to not very popular CMS is stable, besides, we regularly receive requests using the API integration. Since it will be expensive to develop each time a separate module, it was decided to make a universal plugin. Universal Anti-Spam plugin can be installed on any custom websites, content management systems and frameworks. If the user has no programming experience to connect the API to the site, this will be the best solution for protecting the site from spam.

      How it works?

      The CleanTalk installer adds its code to the index.php file. When the visitor fills in and submits the form, the plugin intercepts the form data and finds the email, the message itself and adds some other parameters to them and sends them to the CleanTalk cloud, except when the form has been found banned for sending data (they are stitched into the plugin and can not changed). After receiving a response from the server, the plugin skips or forbids further execution (displaying a message about the reasons for the lock).

      After analyzing the parameters sent, CleanTalk servers decide whether the request should be blocked or allowed. Since the universal CleanTalk libraries were used in writing, it was necessary to organize only the installation and interception of forms. To be honest, we had to rewrite the libraries so that they worked on pure PHP and add exceptions for some fields, such as registering or recovering passwords or paying with cards.

      For each client’s request, we do a test with its CMS, complementing the plugin to work specifically with this CMS. Therefore, at the moment we do not want to make new plugins, as it entails overhead costs in the form of loss of time to support versions up to date.

      The plugins themselves:

      MODX

      MyBB

      OpenCart

      XenForo2

      Universal plugin

      At the moment, we do not plan to expand the range of plugins, only support and development of the current functionality. We hope that the universal plugin will be able to close these gaps, as it is easier to modify one plugin than to do each time a new one.

    • Spam attack on the CleanTalk Service

      Good day!

      We must inform you that some days ago we have been attacked by spam on our service. Spammers used species emails xx**@*******lk.org to send spam comments/registrations / subscriptions. At the moment, the attack affected about 4,000 websites. This is not the first such attack on our service, just a couple of weeks ago we faced with a DDoS attack, which we are pretty well blocked.

      PS

      A little add the message to avoid misunderstandings.

      It was not a spam via SMTP and the substitution of the sender’s address. This spam is caused by the fact that spammers use your public email for subscriptions to news and stuff on other websites.

      The problem is actually much wider, we are a small company and we have not caused any difficulties. But for large companies, which business is tied to e-mail, receiving orders for example?

      Let’s say John Doe took for the example online shop, took them the email address for type orders or***@*****le.com and signed up with this address on 10,000 newsletters at different sites. Every day, this address will be sent news and other mailings, the work of the sales department in this store can be significantly slowed down and cause losses for business, including reputation ones.

      There are several problems here:

      1. Websites are poorly protected from spam.
      2. The company may have problems because it starts receiving a lot of messages.
      3. Reputation losses. Site owners see spam from you, the visitors of websites can see spam from you.

      What do you think, how we can avoid it?

    • Auto-Update for CleanTalk Anti-Spam Plugin

      Updating plugins and themes on the site can be a problem for website owners. This is especially true for webmasters who support several websites, you have to go to every website and make an update, and it takes your time. We have released an update that will perform this routine task and will update CleanTalk Anti-Spam on all sites at once.

      CleanTalk Dashboard allows you to select several websites and update the plugin at once on all sites one click or you can setup auto-update for all websites or separate websites.
      Note: there is 24 hours delay before auto-update will do. This delay allows needing to avoid any issues. All updates that made through CleanTalk Dashboard manually will do immediately.

      How does it work?

      • Manual update on all or selected websites at once.
      • Auto-update on all or selected websites at once.

      Please, go to your CleanTalk Dashboard.

      • Choose a website that needs to update the plugin.
      • Click on the link Update app.
      • Next step, in pop-up you can choose:
        do a manual update and the plugin will be updated immediately. You can do this action for all websites
        or
        set auto-update, in the next time plugin will be updated automatically.

      In the end, enjoy you saved your time.

      Auto-updating system will work from CleanTalk Anti-Spam version 5.88

    • Changing SSL Policy Requirements

      Changing SSL Policy Requirements

      Hello,

      We want to inform you about changes in our SSL service providing.

      The Comodo Certificate Authority will make a change in Certification Authorities’ policy requirements, reducing the maximum lifetime for all TLS/SSL certificates to 825 days, which is just over 27 months. This is a reduction from the existing maximum term of 39 months, meaning the maximum validity for an SSL certificate is now two years, and not three.

      As a result, Comodo CA will adopt these new lifetime requirements effective March 1, 2018 and three-year certificates will be removed as options from all prices.

      So, you don’t have a much time left to buy SSL for 3 years and forget about re-insurance your SSL every year.

      Look at our SSL offer.

      Thank you!

    • CleanTalk Released a MOD for the Protection of XenForo 2 Against Spam and Spam Bots

      CleanTalk completed the development MOD for the XenForo2, to protect your online store against spam and spambots.

      The anti-spam MOD is compatible with XenForo versions v2.x. and approved on the XenForo official catalog.

      This extension allows users to quickly and easily connect to CleanTalk anti-spam service. Its service provides 100% protection from spam bots and provides a simple and convenient commentary/sign-up form for visitors.

      This MOD allows protection from automatically distributed spam, as well as from spam bots registrations. The protection method offered by CleanTalk allows users to switch from the methods that trouble the communication (CAPTCHA, question-answer etc.) to a more convenient one.

      CleanTalk saves time and resources spent on moderation and verification of questionable users or comments. Unnoticeable Protection from spam and registration forms, which does not require the visitor to prove who they are, increases the conversion in registration and is not annoying.

      CleanTalk is a cloud service, security module, installed on a website, which sends the behavior parameters of the visitor or spambot. These parameters are estimated, and the service makes a decision – to post a message or to define it as spam and reject.

      On the basis of such checks, the service forms its own list of email addresses used by spambots. Likewise, the registration of visitors is checked. The service adds not only email addresses to the blacklist but also IP addresses and domains of the websites that are promoted through spam mailout. The CleanTalk company developed unique algorithms to assess the behavior of the visitor, and with 100% accuracy, we keep spam messages at bay. All of this happens automatically and requires no action from the administration of the website.

      You can now download CleanTalk Anti-Spam from XenForo official catalog.

      The Company provides automatic spam protection services for popular CMS: WordPress, Joomla, phpBB, Drupal, DataLife Engine, IP Board, vBulletin, MediaWiki, SimpleMachinesForum and API for PHP, Perl, Python, NET.