Crawlers on Sierra Systems

Overview

The recent substantial increase in automated deep scans of Sierra, Encore and INN-Reach systems has resulted in significant extra load being placed on systems, to the point of denying service. We have identified user agents that are placing unacceptable levels of stress on systems and are implementing measures to block these agents from accessing Clarivate hosted systems at the datacenter firewalls.

We continue to monitor the situation closely. In particular, the list of user agents to block changes over time, and we will provide updates when we add agents to the list. We will continue to implement further measures as necessary to protect and preserve service levels for all our products.

Our current actions are listed below. Please open a support case, contact your regional support management contact, or contact our email escalation alias at iiisupportescalation@clarivate.com if you have questions and would like a conversation.

For all Sierra, Encore & INN-Reach systems, per system:

• Proactively push Clarivate-maintained list of forbidden User-Agents based on collectively observed disruptive deep scan behavior, so Sierra and Encore respond with 403 Forbidden results for requests from those agents
• Update robots.txt to include Crawl-delay directive for all User-Agents on all systems
• Individually block ip addresses, or short lists of ip addresses, which are seen during our response to an individual library's event
• Individually block per library User-Agents which contribute to disruptive deep scan behavior on a particular system, as part of our response to a reported event

Clarivate Hosted Sierra, Encore & INN-Reach systems, additionally:

• Support staff use automated log-scanning tools to identify addresses associated with antisocial behavior and, after manual verification, add them to a curated block list of addresses to protect all Hosted Sierras and Encores.
• On each Sierra application and Encore server, automated blocking of participating IP addresses based on log analysis detecting very high volumes (bursts) connections from the same IP address, as well as matching rogue User-Agent and URL/URI pattern requests. Identified IP addresses are blocked for a period of time before being allowed through again to avoid permanently banning users of dynamically-allocated addresses (e.g. home users with broadband).
• Geo-block requests from countries from which disruptive deep scanning has been frequently collectively observed on Sierra and Encore systems may be configured
• Continual monitoring of resource consumption, downtime and response time, to enable rapid response to a disruptive scan

Software Only Systems (Self-hosted):

• We recommend libraries work with their local IT or hosting provider to block requests from ip addresses which are observed to be disruptive, and to geo-block as appropriate, at the network perimeter
• We recommend libraries work with their local IT or hosting provider to implement request rate limit protections at the network perimeter
• We recommend libraries work with their local IT or hosting provider to analyze Sierra and Encore request logs as necessary to provide actionable block lists based on high volume of same User-Agent, same URL/URI pattern requests, originating from a large number of distributed ip addresses, and to block those ip addresses at the network perimeter