Category: CleanTalk

As you know DDoS attacks on the site are of different intensity, it is important to the number of hosts involved in the attack, the number of network packets and the amount of data transmitted. In the most severe cases, it is possible to repel the attack only using specialized equipment and services.

If the volume of the attack is less than the bandwidth of the network equipment and the computing power of the server (server pool) serving the site, you can try to “block” the attack without resorting to third-party services, namely to include a program filter of traffic coming to the site. This filter will filter out the traffic of bots participating in the attack, while skipping legitimate traffic of “live” site visitors.

Scheme of software filter from DDoS attacks on the site

The filter is based on the fact that bots participating in DDoS attacks are not able to execute JavaScript code, so bots will not go beyond the stop page of the filter, which will significantly unload the frontend/backend and the site database. Because to process each GET/POST DDoS attack request, you will need to execute no more than 20 lines of code in the backend of the site and give the page-stub of less than 2KB of data.

1. The filter is called by the first line of the web application, before calling the rest of the application code. So it is possible maximally to unload the “hardware” of the server and reduce the amount of traffic sent to the bots.
2. If the visitor falls under the filter conditions, then we give the visitor a special page-stub. On the page,

• We are reporting the reasons for issuing a special page instead of the requested one
• We set the special cookie in the user’s browser through JavaScript
• Run JavaScript redirect code to the source page

1. If the visitor has a special cookie, the filter transparently passes the visitor to the requested page of the site.
2. If the visitor’s IP address belongs to an autonomous system from the list of exceptions, then the traffic is also transparently skipped. This condition is necessary to exclude the filtering of search engine bots.

Project filter on github.com.

Synthetic tests of the filter

We tested ab utility from Apache Foundation on the main page of the combat site, previously removing the load from one of the nodes.

Results with a disabled filter,

ab -c 100 -n 1000 https://cleantalk.org/

Total transferred: 27615000 bytes
HTML transferred: 27148000 bytes
Requests per second: 40.75 [#/sec] (mean)
Time per request: 2454.211 [ms] (mean)
Time per request: 24.542 [ms] (mean, across all concurrent requests)
Transfer rate: 1098.84 [Kbytes/sec] received

Now the same thing with the filter on,

Total transferred: 2921000 bytes
HTML transferred: 2783000 bytes
Requests per second: 294.70 [#/sec] (mean)
Time per request: 339.332 [ms] (mean)
Time per request: 3.393 [ms] (mean, across all concurrent requests)
Transfer rate: 840.63 [Kbytes/sec] received

As you can see from the test results, enabling the filter allows the web server to process almost an order of magnitude more requests than without the filter. Naturally, we are talking only about requests from visitors without JavaScript support.

Application of the filter in practice, the history of saving the site from one small DDoS attack

Periodically, we are faced with DDoS attacks on our own corporate site https://cleantalk.org. Actually during the last of the attacks, we applied a filter from DDoS at the level of the website applications.

The start of the attack

The attack started at 18:10 UTC + 5 on January 18, 2018, attacked the GET with requests to the URL https://cleantalk.org/blacklists. On the network interfaces of the Front-end servers, there appeared an additional 1000-1200 kbit/second of incoming traffic, i.e. received a load of 150/second GET requests to each server, which is higher than the nominal load 5 times. As a consequence, the Load average of Front-end servers and database servers has grown dramatically. As a result, the site began to issue error 502 due to the lack of free processes php-fpm.

• Attack analysis
• After spending some time studying the logs, it became clear that this is the DDoS attack, because,
• 5/6 requests were for the same URL.
• There was no clearly defined group of IP addresses creating a load on the URL from item 1.

CPU front-end servers were loaded an order of magnitude higher than the surge load on network interfaces.

Accordingly, it was decided to include a filter of visitors to the site using the algorithm described above, additionally including in it the checking of incoming traffic through our database of blacklists, thereby reducing the probability of issuing a stop page to legitimate visitors to the site.

Enabling the filter

After spending some more time preparing the filter, it was switched on at 19:15-19:20.

After a few minutes, we got the first positive results, first Load average returned to normal, then the load on the network interfaces fell. A few hours later, the attack was repeated twice, but its consequences were almost invisible, the frontends worked without errors 502.

Conclusion

As a result, by using the simplest JavaScript code, we solved the problem of filtering traffic from bots, thereby extinguishing the DDoS attack and returning the availability of the site to its regular state.

Honestly, this bot filtering algorithm was not invented on the day of the attack described above. A few years ago, we implemented the additional function SpamFireWall to our Anti-Spam service, SpamFireWall uses more than 10 thousand websites and there is a separate article about it.

SpamFireWall was developed primarily to deal with spam bots, but since the lists of spambots intersect with the lists of other bots used for questionable purposes, the use of SFW is quite effective, including for stopping small DDoS attacks on the site.

About CleanTalk service

CleanTalk is a cloud-based service for protecting websites from spambots. CleanTalk uses protection methods that are invisible to the visitors of the website. This allows you to opt out of methods of protection that require the user to prove that he is human (captcha, question-answer, etc.).

E-mail is still one of the most important and effective elements of online business and marketing and is the most effective channel for generating revenue. Therefore, for any online business and website owners, it is important to be sure that the email owner used it for registration/subscription so that the user used his real email address.

There are several important reasons for this.

First, it is important to recover the forgotten password, for example: the user has misspelled his email address, after a while used the password recovery function and cannot get a new password.

Second, this user will not receive your email notifications.

Third, the user whose email was used by spammer for registration/subscription will send your newsletter to spam. Further, spammers can use this email for sending spam, brute force, etc.

The use of such emails for marketing mailings and other mailings may carry certain risks for the owner of the site and increase the risks of adding to the lists DNSBL, various blacklist’s, increase the likelihood of getting into the spam folder and your other emails. Increase your budget costs for mailing to fake addresses or addresses whose owners have been compromised by spammers.

To avoid or minimize risks to a minimum, the list of email addresses must be checked before sending.

Recently, we started checking the existence of email addresses, which checks if such an email address exists or not. It uses the MX records of the domain of the email address and connects to the mail servers (via SMTP, and also simulates the sending of the message) to ensure that the mailbox does exist for that user/address and it can receive mail.

This test solves several tasks at once, reducing the likelihood of skipping spam and allowing users to check the emails of their users. To test users, you can use the API method https://cleantalk.org/help/api-spam-check.

An example of the result of the API response for https://cleantalk.org/blacklists?record=mattressfg%40gmail.com

{"data":
            {"ma********@gm***.com":
                      {"appears":1,
                      "frequency_time_10m":0,
                      "spam_rate":"1",
                      "frequency":"8",
                      "frequency_time_24h":1,
                      "updated":"2018-03-26 00:26:48",
                                                  "exists": 0, 
                      "frequency_time_1h":1}
            }
}

At the moment, we use this tool only to improve spam detection and check all email addresses that have had spam activity in the last 10 minutes. For checked addresses, a flag is set in the database, which is taken into account in the further spam filtering and API response.

“exists”: “1” – exists

“exists”: “0” – does not exist

You can sum up the interim results; the percentage of non-existent email addresses in the spam mailing is 25.34%.

We collect information about spam IP/email addresses from more than 237,000 websites. The email database contains more than 10,095,239 email addresses. And in our immediate plans, check all emails in the database and run a scan of all the email addresses that are processed by the service.