Rate Limits

Overview

There are many forms of web attack that are built upon requests which otherwise seem benign: brute-forcing login forms, enumeration, payment card validation, coupon/gift code discovery, and more. Rate limiting is a crucial defense against this malicious activity.

Rate Limits are rules which define the number of requests with certain characteristics that are allowed within defined time frames. When a request is received that exceeds a Rate Limit, a specified action is performed.

The input controls at the top of this page are described here: Policies & Rules Entry Administration. Specific editing of a Rate Limit is described below.

Rate Limit concepts

Definitions

A Rate Limit defines the number of times (specified as the Limit) that requests are allowed to match certain conditions within a certain Time Frame. Once that limit has been reached, subsequent requests matching those conditions within the same Time Frame will trigger an Action.

The matching conditions are specified with the Count by parameters and the Event. See Matching Conditions below.

Tiered Rate Limiting

A Rate Limit must contain at least one Threshold: a combination of Limit and Action.

For a single-Threshold Rate Limit, incoming requests that meet the Matching Conditions are processed as follows:

Allow the requests until the Limit is reached.
Each subsequent request within the same Time Frame will trigger the Action.

A Rate Limit can contain more than one Threshold, with successively increasing Limits. This creates a 'multi-tier' Rate Limit. Incoming requests that meet the Matching Conditions are processed as follows:

Allow the requests (and take no Action) until the first Limit is reached.
Each subsequent request will trigger the first tier's Action, until the second Limit is reached.
Each subsequent request will trigger the second tier's Action, until the third Limit is reached.
And so on.

This allows for increasingly restrictive actions to be taken as the number of incoming requests within the same Time Frame increases.

For an example of a multi-tiered Rate Limit, see the first example of using the Ban Action below.

Enforcement

By default, a Rate Limit will be enforced for all requests for the URL(s) to which it is assigned.

After a Rate Limit is defined on this page, it will not be active until it is assigned to one or more URLs via a Security Policy. This is done at the bottom of the page in the Connections to Security Policies, or alternately in the Security Policies Document.

Enforcement can be further limited to a subset of these requests: see Limiting the scope of a Rate Limit below.

Meta Parameters

Field

Value

Name

A name that will be used within the Curiefense interface, and will also be used to create a tag that will be assigned to requests which trigger the Rate Limit. It is recommended that the name summarizes the rule; for example, a rule with a Threshold of 5 and a TTL of 60 could be named "Rate limit rule 5/60".

Description

A description that will be used within the Curiefense interface.

Rate Limit

Field

Value

Time Frame

The period (specified in seconds) within which each Limit is enforced.

Limit

The maximum number of allowable requests within the specified Time Frame. Subsequent requests within this period will trigger the specified Action (discussed below). Each Rate Limit tier has its own Limit and an associated Action.

Matching Conditions (the "Count by" input controls)

A condition consists of a field and a value. Within a Rate Limit, they play a role like this:

"More than <LIMIT> requests with the same <CONDITION-VALUE> <CONDITION-FIELD> sent to <ASSIGNED-LOCATION> within <TIME-FRAME> will cause <ACTION>."

Example: "More than three requests with the same username argument sent to the login form within one hour will cause the requestor to be banned for six hours."

Note that the <ASSIGNED-LOCATION> mentioned above is included for illustration purposes only. This value is not part of a Rate Limit definition—it is specified at the bottom of the page in the Connections to Security Policies, or alternately in the Security Policies Document.

Condition Definition

A condition can be built upon any one of these four categories:

Field

Result

Header

All requests with the same value for the specified header will be counted together toward the Limit.

Cookie

All requests with the same value for the specified cookie will be counted together toward the Limit.

Argument

All requests with the same value for the specified argument will be counted together toward the Limit.

Attribute

All requests with the same value for the specified attribute will be counted together toward the Limit.

Multiple Conditions

Multiple conditions can be defined within the same Rate Limit. To create a new condition and open it for editing, select "New entry" below the Count by input controls.

If multiple conditions are defined, they are evaluated by combining them together with a logical AND. In other words, the cumulative count toward the Limit will be incremented whenever a request is seen that matches all of the conditions simultaneously. Different combinations of conditions will have separate Limit counts maintained for them.

Example. A Rate Limit contains two conditions: "Attribute / IP Address" and "Argument / Username". When the first request is received, an internal counter is created (set to a value of one) for this unique combination of IP Address and Username. A second request is then received, originating from the same IP and for the same Username; this causes the internal counter to be incremented up to two. A third request is then received from the same IP but with a different Username; this causes a new internal counter to be created (and set to a value of one) for this combination.

The "Event" option: Changing the meaning of the Rate Limit

Below the list of condition(s), there is another condition named "Event."

By default, this is set to "HTTP request,", which simply means to increment a counter each time a request is received that matches the conditions.

However, if the Event condition is changed to a different value, then the following applies.

In the discussion below, "Count Condition" will refer to the condition (or combination of conditions) defined by the Count by input controls.

"Event Condition" will refer to the optional, additional condition defined by the Event input controls.

Adding an Event Condition changes the evaluation process. An Event Condition is not logically combined with the preceding Count Condition; it is always evaluated separately.

More importantly, adding an Event Condition changes the meaning of the Rate Limit.

If an Event Condition is not defined—in other words, if "HTTP request" is selected—then as discussed above, an internal counter is maintained for each Count Condition value, and incremented each time that value is encountered in a request.
If an Event Condition is defined—in other words, if something other than "HTTP request" is selected—an internal counter is maintained for each Count Condition Value, and incremented each time a new, previously unobserved Event Condition value is encountered in a request.

Therefore, if an Event Condition is defined, the Rate Limit constrains the number of allowable Event Condition values for any given Count Condition value.

So, the evaluation becomes something like this:

"More than <LIMIT> <EVENT-CONDITION-VALUE> <EVENT-CONDITION-FIELD>s per any one <COUNT-CONDITION-VALUE><COUNT-CONDITION-FIELD> sent to <ASSIGNED-LOCATION> within <TIME-FRAME> will cause <ACTION>."

Note that the number of Count Condition values is not being limited here. The limit is on the number of Event Condition values that each Count Condition value is allowed.

Example: Let's say we want to allow an individual user to login from a maximum of two ASNs within one hour. (Perhaps the user is accessing our web application from a coffee shop's WiFi, and then a few minutes later, leaves the coffee shop and begins using the cell network instead.) We want to allow this possibility; however, if we receive requests from the same user originating from three or more ASNs within an hour, we want to treat this traffic more suspiciously. This is not possible merely by specifying two Count Conditions, as described earlier in the "Multiple Conditions" section. If we set up two conditions ("Argument / Username" and "Attribute / Organization") with a Limit of 2, and assign it to our login form, then this will merely limit the number of times that the user can login from each ASN within an hour. Instead, we can set up one Count Condition ("Argument / Username") and then set up the Event Condition ("Attribute / Organization"). Now the Limit will apply to the number of Organizations that are observed for each specific Username.

Action

When an incoming request exceeds the Limit, the specified Action will occur.

Action

Meaning

503 Service Unavailable

The request will be blocked and the requestor will receive a response of "503 Service Unavailable".

Challenge

For a browser-based web application, a bot challenge will be issued to verify that the requestor is a human using a browser, and not a bot using a headless browser or emulator. If the challenge is failed, the request is blocked.

Tag Only

The request will not be blocked; it will merely be tagged with the Rate Limit's name, for subsequent viewing in the Access Log and other places. This Action is useful for testing new Rate Limit rules without actually affecting incoming traffic.

Response

Blocks the request, and responds with a custom error code (0-999) and response body.

Redirect

Blocks the request with a custom error code, and redirects the requestor to a specified URL. For example, the URL might be a page that says, "Your activity appears suspicious, and your access has been restricted. Contact support if you think this decision was made in error."

Ban

Blocks the requestor for the specified amount of time. See further discussion below.

Header

Does not block the request, but adds headers to it (indicating the Rate Limit rule name and the Limit) for receipt and evaluation by the user's backend.

The Ban action

Most of the Actions listed above will not, by themselves, fully exclude an attacker that continues pressing the attack.

Example: Access to a login form is rate-limited to four requests per minute. An attacker tries to brute-force the login, and sends 60 requests per minute. The Rate Limit allows the first four requests, but then blocks the next 56 requests with a 503 error. However, after the minute has passed, the Rate Limit resets. The attacker is allowed another four attempts before being temporarily blocked again. This cycle can continue for as long as the attacker wishes. In effect, the Rate Limit is not preventing the attack; it is merely slowing it from 60 attempts per minute down to four attempts per minute.

The Ban action can be used to block (or take some other Action in response to) a Rate Limit violator for an extended period of time.

A Rate Limit is created for a login form, allowing four requests per minute. After that, the user is redirected to a page that warns them that their activity is suspicious. An additional tier is also defined: a Limit of 15, with an Action of Ban for one hour.

Now an attacker tries to brute-force the login form, attempting to send 60 requests per minute. The first four requests are allowed. The next eleven requests are redirected to the warning page. The sixteenth request triggers the Ban. For the next hour, the attacker's requests will be blocked with a 503 error.

Bans are commonly used as the final tier in a multi-tier Rate Limit, as in the example above.

In earlier versions of Curiefense, each Rate Limit contained one Action. To create the equivalent of a multi-tiered Rate Limit—for example, a Challenge and then a Ban—admins had to create separate Rate Limits and then assign them all to the same URL in Security Policies.

This technique is still supported. However, now that it is possible to create multiple tiers within the same Rate Limit, this will be the best approach in most situations.

Example: Using Ban to increase performance when filtering hostile bots

When a traffic source fails a bot challenge, Curiefense will block the request. If the bot keeps re-submitting its request, it will continue to fail the challenges. However, each time the bot tries again, Curiefense has to issue a new challenge . To solve this problem, a Rate Limit is created with a Ban action. Now a persistent bot will simply be Banned, saving Curiefense the overhead of issuing continuous challenges.

Ban Actions

Note that when setting up a Ban, the most common choices for its Action are to deny the violator's requests (via 503 Service Unavailable, Response, or Redirect). However, you can also choose Tag Only (to observe the violator's actions during the ban period), Challenge (to verify that the violating activity is not being done by bots), or Header (to mark the requests for further scrutiny by the backend).

Connections to Security Policies

In this section of the interface, Rate Limits can be attached to one or more Security Policies. This attachment can also be done, and the Security Policies themselves are defined, on the Security Policies page.

Limiting the scope of a Rate Limit

By default, an active Rate Limit rule will be enforced upon all incoming requests targeting URLs to which this rule has been assigned via a Security Policy.

To change this behavior, you can add Include and/or Exclude tags (as defined in Global Filters). These define the portion of the incoming traffic stream that will be evaluated for possible violation of the Rate Limit. In other words, they limit the scope of the Rate Limit's enforcement:

The Include filter will limit enforcement to requests that have these Tags attached. All other requests in the traffic stream will not have this Rate Limit enforced upon them.
The Exclude filter will exempt requests from enforcement that otherwise would have been subject to it.

(Internally, Curiefense evaluates Exclude parameters first, and then Include parameters.)

To add one or more filters, select New entry, define the parameters, then select the "+" button. If more than one Include filter is specified, they are combined with a logical AND.