Crawlers on Sierra Systems

Overview

The recent substantial increase in automated deep scans of Sierra, Encore and INN-Reach systems has resulted in significant extra load being placed on systems, to the point of denying service. We have identified user agents that are placing unacceptable levels of stress on systems and are implementing measures to block these agents from accessing Clarivate hosted systems at the datacenter firewalls.

We continue to monitor the situation closely. In particular, the list of user agents to block changes over time, and we will provide updates when we add agents to the list. We will continue to implement further measures as necessary to protect and preserve service levels for all our products.

Our current actions are listed below. Please open a support case, contact your regional support management contact, or contact our email escalation alias at iiisupportescalation@clarivate.com if you have questions and would like a conversation.

For all Sierra, Encore & INN-Reach systems, per system:

• Proactively push Clarivate-maintained list of forbidden User-Agents based on collectively observed disruptive deep scan behavior, so Sierra and Encore respond with 403 Forbidden results for requests from those agents
• Update robots.txt to include Crawl-delay directive for all User-Agents on all systems
• Individually block ip addresses, or short lists of ip addresses, which are seen during our response to an individual library's event
• Individually block per library User-Agents which contribute to disruptive deep scan behavior on a particular system, as part of our response to a reported event

Clarivate Hosted Sierra, Encore & INN-Reach systems, additionally:

• Autoblock of participating ip addresses based on log analysis detecting high volume of same User-Agent, same URL/URI pattern requests, originating from a large number of distributed ip addresses, indicating a centrally controlled botnet scan. Participating ip addresses are blocked for a period of time.
• Geo-block requests from countries from which disruptive deep scanning has been frequently collectively observed on Sierra and Encore systems
• Continual monitoring of resource consumption, downtime and response time, to enable rapid response to a disruptive scan

Software Only Systems (Self-hosted):

• We recommend libraries work with their local IT or hosting provider to block requests from ip addresses which are observed to be disruptive, and to geo-block as appropriate, at the network perimeter
• We recommend libraries work with their local IT or hosting provider to implement request rate limit protections at the network perimeter
• We recommend libraries work with their local IT or hosting provider to analyze Sierra and Encore request logs as necessary to provide actionable block lists based on high volume of same User-Agent, same URL/URI pattern requests, originating from a large number of distributed ip addresses, and to block those ip addresses at the network perimeter