Stop spam without frustrating your visitors

Create your CleanTalk account and start blocking spam — no CAPTCHA challenges and no impact on visitors.

Security Block Lists

CleanTalk Account

No credit card required • Setup takes less than a minute • Your temporary password will be sent by email.

Author: Vitalii

  • How AI Helps Speed Up Web Application Development

    How AI Helps Speed Up Web Application Development

    Web application development is not only about writing new code. A large part of daily work involves understanding existing logic, analyzing bugs, preparing SQL queries, refactoring code, reviewing implementation options, and improving interface messages.

    In these tasks, AI can be a practical development assistant. It does not replace the developer, but it helps reduce the time spent on routine analysis, first drafts, and alternative solutions.

    Understanding Existing Code Faster

    In real projects, developers often work with code that was written long ago, changed many times, and contains complex conditions or business logic.

    Before making changes, it is important to understand which scenarios are already covered, which checks are critical, and where weak points may exist. AI can help explain the general logic, highlight important conditions, and suggest cases that should be tested separately.

    This is especially useful when working with legacy code, where the goal is not always to rewrite everything from scratch, but to make a safe and controlled change.

    Formatting Code and SQL Queries

    One of the simplest but most common use cases is making code easier to read.

    A long PHP condition, a complex SQL query, or a large JavaScript block may work correctly but still be difficult to analyze. AI can quickly format such fragments, split them into readable blocks, align the structure, and prepare the code for review.

    This saves time because the faster the code becomes readable, the faster the developer can focus on the actual problem.

    Refactoring Without Rewriting Everything

    In production projects, there is not always time or reason to rebuild a module from scratch. More often, the task is to improve a specific part of the code within the existing architecture.

    AI can suggest several refactoring options: a minimal and safe change, a cleaner but larger update, or a balanced compromise. For example, it can help remove duplication, simplify conditions, extract repeated logic, or make a function easier to understand.

    The main value is not that AI “rewrites the code,” but that it speeds up the preparation of options and helps identify where the solution can be simplified.

    Working with SQL and Databases

    SQL often requires separate analysis. Developers may need to select data, find duplicates, check table relations, understand why a query is slow, or prepare a diagnostic query.

    AI can help create an initial query, format existing SQL, suggest possible reasons for duplicate rows, propose additional checks, or help interpret an EXPLAIN result.

    The final validation must still be done on the real database, because AI does not know the actual data distribution, indexes, and production load. However, it helps form hypotheses faster and gives the developer a better starting point.

    Reviewing Logic and Finding Weak Points

    AI can also be used as an additional review layer before code review or release. It can help check whether some conditions are missing, whether edge cases were overlooked, or whether the behavior may differ from what was expected.

    For example, AI can point out what may happen with an empty value, a missing array key, a repeated request from the interface, or a conflict between old and new logic.

    This does not replace a proper code review, but it helps catch part of the issues earlier.

    Comparing Implementation Options

    Many tasks can be solved in more than one way. A check can be added on the frontend or backend. A SQL query can be changed, or an index can be added. An existing method can be extended, or a separate one can be created.

    AI helps compare these options quickly. It can outline the pros and cons, show potential risks, and suggest a simpler approach when the task does not require a complex solution.

    This is especially useful during the discussion stage, before the final implementation decision is made.

    Interface Messages, Errors, and Translations

    In web applications, it is important not only to handle errors correctly but also to explain them clearly to the user. A message that is too technical may confuse the user, while a message that is too generic may not help solve the problem.

    AI can help prepare clearer wording for errors, notifications, hints, titles, and translations. It can also adapt the same message for different languages while preserving the meaning and product tone.

    The team still chooses the final version, but the first draft appears much faster.

    Turning a Problem into a Task

    A development task often starts not with a clear technical description, but with an observation: “something looks wrong in the interface,” “the user does not understand the result,” or “the query is too slow.”

    AI can help structure such input into a clear task: what happens now, why it is a problem, what result is expected, and which scenarios should be checked.

    This improves communication between developers, managers, and support teams, and reduces the number of clarifications before implementation.

    Where the Developer Is Still Essential

    AI can suggest solutions, but it does not know the full project context: production load, business rules, change history, hidden dependencies, and long-term architectural goals.

    That is why final decisions must remain with the developer. This is especially important for security, access control, payments, user data, architecture, and critical business logic.

    AI is most useful as a tool for faster analysis, drafts, and comparison of options. Responsibility for the final result stays with the team.

    Conclusion

    AI speeds up web application development by reducing the time spent on routine and analytical work. It helps developers understand existing code, format complex fragments, prepare SQL queries, review logic, improve interface messages, and compare implementation options.

    Used correctly, AI becomes not a replacement for developers, but a practical tool that helps the team move from problem to solution faster.

  • How to Reduce Server Load by Simply Filtering Bad Traffic

    How to Reduce Server Load by Simply Filtering Bad Traffic

    When a website starts slowing down, many teams immediately think about scaling infrastructure: adding CPU, RAM, more servers, or optimizing the database. In reality, a significant part of server load is often caused not by real users, but by automated traffic — bots, scrapers, vulnerability scanners, spam robots, and aggressive crawlers.

    These requests may continuously scan pages, submit forms, probe URLs, overload search, login, registration, and API endpoints. As a result, your server wastes resources processing useless traffic: PHP workers are occupied, database connections increase, memory is consumed, and response times get worse for legitimate visitors.

    How to Reduce Server Load
    How to Reduce Server Load

    Why Blocking Traffic Early Matters

    Once a malicious or unwanted request reaches backend logic, some resources have already been spent. That is why one of the most effective ways to reduce load is to block suspicious traffic as early as possible, before expensive application code runs.

    Even basic filtering can provide immediate benefits:

    • lower CPU usage;
    • fewer PHP worker bottlenecks;
    • reduced MySQL load;
    • faster page responses for real users;
    • better stability during traffic spikes;
    • cleaner analytics data.

    A Practical Solution for PHP Projects — Anti-Crawler PHP Library

    For PHP-based websites and services, a useful option is CleanTalk php-anticrawler — an open-source Anti-Crawler PHP Library designed to detect and filter unwanted bot traffic.

    It can be integrated into PHP applications as an additional protection layer without requiring a major architecture rebuild.

    Real Usage Example: CleanTalk.org

    The library has already been successfully connected to the CleanTalk website in the Blacklists section. Over the last 60 days, the system processed a large volume of traffic and showed clear filtering results:

    • BLOCKED: 1,791,250 requests
    • LEGITIMATE: 460,502 requests

    This means a significant amount of unwanted automated traffic was stopped before consuming backend resources. A traffic chart for this period can clearly demonstrate how early filtering helps reduce unnecessary server load.

    What the Anti-Crawler PHP Library Can Help With

    The library helps detect and limit suspicious requests using signals such as:

    • IP address reputation;
    • abnormal request frequency;
    • bot-like behavior patterns;
    • technical signs of automated clients;
    • aggressive crawling activity.

    It is especially useful for protecting:

    • login pages;
    • registration forms;
    • contact forms;
    • search pages;
    • REST/API endpoints;
    • resource-heavy pages.

    Business Benefits

    Many companies try to solve load issues by upgrading servers or paying for more infrastructure. But if a large share of requests has no business value, reducing useless traffic is often the smarter first step.

    Filtering bad traffic can help:

    • lower hosting and infrastructure costs;
    • reduce downtime and overload incidents;
    • improve website speed and uptime;
    • increase conversion rates through faster UX;
    • clean traffic reports and analytics.

    Best Use Cases

    This approach is highly effective for:

    • eCommerce stores;
    • SaaS platforms;
    • WordPress and other PHP CMS websites;
    • lead generation websites;
    • public API services.

    Final Thoughts

    Not every performance issue requires more servers. In many cases, the first step should be identifying how much of your resources are wasted on useless automated traffic.

    For PHP projects, the Anti-Crawler PHP Library by CleanTalk can be a practical way to reduce backend load, improve performance, and protect your website from unwanted traffic.

  • Reducing Disk Load in High-Traffic PHP Applications: Switching from SQLite to Redis for Anti-Crawler Storage

    Reducing Disk Load in High-Traffic PHP Applications: Switching from SQLite to Redis for Anti-Crawler Storage

    Automated crawlers and scraping bots are a growing problem for modern websites. While search engine bots are useful, many other crawlers generate excessive traffic, scrape content, or overload servers.

    To help website owners control this type of traffic, we recently released the Anti-Crawler PHP Library by CleanTalk, an open-source tool designed to detect and limit aggressive crawlers before they cause performance problems.

    GitHub repository: https://github.com/CleanTalk/php-anticrawler

    The library analyzes incoming requests and applies rate-limiting logic to detect crawler-like behavior. Once a bot exceeds defined limits, the system blocks or restricts further requests.

    In the first version of the library we chose SQLite as the storage backend. SQLite allowed the library to work immediately after installation without requiring additional infrastructure such as Redis or Memcached.

    However, after deploying the library on our own high-traffic website cleantalk.org, we encountered an unexpected performance issue: disk load increased significantly.

    The result was a simple architectural change that completely removed the disk load increase while improving scalability.

    The First Version of the Anti-Crawler Library

    The goal of the library was to provide a simple crawler protection mechanism for PHP applications. Typical anti-crawler logic requires storing temporary request data. Each request updates this data so the system can determine whether a visitor behaves like a normal user or an automated crawler. Because the data must be updated frequently, the storage backend plays a critical role in overall performance.

    Why SQLite Was Chosen

    For the initial release we selected SQLite for several reasons:

    1. Zero configuration. SQLite is included in most PHP environments and does not require running an additional service.
    2. Single-file storage. All data is stored in a single database file, making installation extremely simple.
    3. Good performance for moderate workloads. SQLite performs very well for many typical web applications.
    4. Easy deployment. Users could install the library without modifying their infrastructure.

    This approach allowed the library to work immediately after installation and made it suitable for shared hosting environments. For many websites this configuration works perfectly. However, high-traffic environments behave differently.

    Deploying the Library on a High-Traffic Website

    After releasing the first version of the library, we deployed it on our own website https://cleantalk.org Our infrastructure handles a large volume of traffic, including both legitimate users and automated bots. Shortly after enabling the library, our monitoring systems detected something unusual. Disk Activity Increased. Server monitoring showed a noticeable increase in disk activity. After analyzing the metrics we observed: Disk load increased by approximately 30%.

    This was unexpected because the library itself performs only lightweight operations. The problem was not CPU usage or memory consumption. Instead, the issue was directly related to disk I/O. Further investigation showed that the additional disk operations were coming from the SQLite database used by the anti-crawler system.

    Why SQLite Became a Bottleneck

    SQLite is a reliable and efficient embedded database, but its design has limitations under certain workloads. The anti-crawler system generates a very specific traffic pattern. For each HTTP request the library needs to:

    • read crawler counters
    • update request statistics
    • write the updated data back to storage

    This means the database receives frequent write operations.

    Because SQLite stores data on disk, every update results in disk activity. Under high traffic this leads to a large number of disk writes. SQLite also uses file-level locking to ensure consistency. When many requests attempt to update the database simultaneously, additional locking overhead appears.

    As a result, frequent writes combined with locking increased disk activity on our production servers.

    Moving the Storage Layer to Redis / KeyDB

    To eliminate disk operations we needed a storage system optimized for frequent updates. The natural solution was an in-memory data store, so we added support for: Redis and KeyDB. Both systems keep data in memory and provide extremely fast read and write operations. This approach removes disk I/O and allows the crawler detection logic to update counters much more efficiently.

    The Anti-Crawler PHP Library was updated to support multiple storage backends. Users can now choose between:

    • SQLite (default)
    • Redis
    • KeyDB

    SQLite remains useful for simple deployments, while Redis or KeyDB can be enabled for high-traffic environments. The crawler detection logic itself remains unchanged — only the storage backend is replaced.

    Results After Switching to Redis

    After switching the storage backend to Redis on our production servers we immediately saw improvements. Disk activity returned to normal because the crawler counters were now stored in memory instead of on disk. The previous 30% increase in disk load disappeared, and request processing became faster. The Redis-based architecture also scales better under heavy traffic and avoids locking issues associated with file-based databases.

    disk io
    disk io

    When to Use SQLite vs Redis

    Both storage options remain available because they fit different environments.

    SQLite works well for:

    • small and medium websites
    • environments without Redis
    • simple installations

    Redis or KeyDB is recommended for:

    • high-traffic websites
    • infrastructure already using Redis
    • environments with heavy bot traffic

    How to Use the Anti-Crawler PHP Library

    The library is open source and available on GitHub: https://github.com/CleanTalk/php-anticrawler It can be integrated into any PHP application to detect aggressive crawlers and limit automated traffic.

    Installation

    composer require cleantalk/php-anticrawler

    Quick starthttps://github.com/CleanTalk/php-anticrawler?tab=readme-ov-file#anti-crawler-php-library-by-cleantalk

    Conclusion

    Switching the storage backend of our Anti-Crawler PHP Library from SQLite to Redis/KeyDB allowed us to eliminate the disk I/O overhead that appeared under high traffic. This small architectural change removed the 30% disk load increase and made the crawler detection system faster and more scalable for busy websites.

    On cleantalk.org Anti-Crawler PHP Library serves about 20k sessions weekly, wich gives roughly 500k hits weekl.

    Anti-Crawler PHP Library by CleanTalk

    Protect your website from aggressive crawlers, automated scraping, and unwanted bot traffic using the CleanTalk Anti-Crawler PHP library.