There are many forms of web attack that are built upon requests which otherwise seem benign: brute-forcing login forms, enumeration, payment card validation, coupon/gift code discovery, and more. Rate limiting is a crucial defense against this malicious activity.
Rate Limits are rules which define the number of requests with certain characteristics that are allowed within defined time frames. When a request is received that exceeds a Rate Limit, a specified action is performed.
A Rate Limit defines the number of times (specified as the Limit) that requests are allowed to match certain conditions within a certain Time Frame. Once that limit has been reached, subsequent requests matching those conditions within the same Time Frame will trigger an Action.
The matching conditions are specified with the Count by parameters and the Event. See Matching Conditions below.
A Rate Limit must contain at least one Threshold: a combination of Limit and Action.
For a single-Threshold Rate Limit, incoming requests that meet the Matching Conditions are processed as follows:
- Allow the requests until the Limit is reached.
- Each subsequent request within the same Time Frame will trigger the Action.
A Rate Limit can contain more than one Threshold, with successively increasing Limits. This creates a 'multi-tier' Rate Limit. Incoming requests that meet the Matching Conditions are processed as follows:
- Allow the requests (and take no Action) until the first Limit is reached.
- Each subsequent request will trigger the first tier's Action, until the second Limit is reached.
- Each subsequent request will trigger the second tier's Action, until the third Limit is reached.
- And so on.
This allows for increasingly restrictive actions to be taken as the number of incoming requests within the same Time Frame increases.
By default, a Rate Limit will be enforced for all requests for the URL(s) to which it is assigned.
Enforcement can be further limited to a subset of these requests: see Limiting the scope of a Rate Limit below.
A condition consists of a field and a value. Within a Rate Limit, they play a role like this:
"More than <LIMIT> requests with the same <CONDITION-VALUE> <CONDITION-FIELD> sent to <ASSIGNED-LOCATION> within <TIME-FRAME> will cause <ACTION>."
A condition can be built upon any one of these four categories:
Multiple conditions can be defined within the same Rate Limit. To create a new condition and open it for editing, select "New entry" below the Count by input controls.
If multiple conditions are defined, they are evaluated by combining them together with a logical AND. In other words, the cumulative count toward the Limit will be incremented whenever a request is seen that matches all of the conditions simultaneously. Different combinations of conditions will have separate Limit counts maintained for them.
Below the list of condition(s), there is another condition named "Event."
By default, this is set to "HTTP request,", which simply means to increment a counter each time a request is received that matches the conditions.
However, if the Event condition is changed to a different value, then the following applies.
Adding an Event Condition changes the evaluation process. An Event Condition is not logically combined with the preceding Count Condition; it is always evaluated separately.
More importantly, adding an Event Condition changes the meaning of the Rate Limit.
- If an Event Condition is not defined—in other words, if "HTTP request" is selected—then as discussed above, an internal counter is maintained for each Count Condition value, and incremented each time that value is encountered in a request.
- If an Event Condition is defined—in other words, if something other than "HTTP request" is selected—an internal counter is maintained for each Count Condition Value, and incremented each time a new, previously unobserved Event Condition value is encountered in a request.
Therefore, if an Event Condition is defined, the Rate Limit constrains the number of allowable Event Condition values for any given Count Condition value.
So, the evaluation becomes something like this:
"More than <LIMIT> <EVENT-CONDITION-VALUE> <EVENT-CONDITION-FIELD>s per any one <COUNT-CONDITION-VALUE><COUNT-CONDITION-FIELD> sent to <ASSIGNED-LOCATION> within <TIME-FRAME> will cause <ACTION>."
Note that the number of Count Condition values is not being limited here. The limit is on the number of Event Condition values that each Count Condition value is allowed.
When an incoming request exceeds the Limit, the specified Action will occur.
Most of the Actions listed above will not, by themselves, fully exclude an attacker that continues pressing the attack.
The Ban action can be used to block (or take some other Action in response to) a Rate Limit violator for an extended period of time.
A Rate Limit is created for a login form, allowing four requests per minute. After that, the user is redirected to a page that warns them that their activity is suspicious. An additional tier is also defined: a Limit of 15, with an Action of Ban for one hour.
Now an attacker tries to brute-force the login form, attempting to send 60 requests per minute. The first four requests are allowed. The next eleven requests are redirected to the warning page. The sixteenth request triggers the Ban. For the next hour, the attacker's requests will be blocked with a 503 error.
When a traffic source fails a bot challenge, Curiefense will block the request. If the bot keeps re-submitting its request, it will continue to fail the challenges. However, each time the bot tries again, Curiefense has to issue a new challenge . To solve this problem, a Rate Limit is created with a Ban action. Now a persistent bot will simply be Banned, saving Curiefense the overhead of issuing continuous challenges.
Note that when setting up a Ban, the most common choices for its Action are to deny the violator's requests (via 503 Service Unavailable, Response, or Redirect). However, you can also choose Tag Only (to observe the violator's actions during the ban period), Challenge (to verify that the violating activity is not being done by bots), or Header (to mark the requests for further scrutiny by the backend).
In this section of the interface, Rate Limits can be attached to one or more Security Policies. This attachment can also be done, and the Security Policies themselves are defined, on the Security Policies page.
By default, an active Rate Limit rule will be enforced upon all incoming requests targeting URLs to which this rule has been assigned via a Security Policy.
To change this behavior, you can add Include and/or Exclude tags (as defined in Global Filters). These define the portion of the incoming traffic stream that will be evaluated for possible violation of the Rate Limit. In other words, they limit the scope of the Rate Limit's enforcement:
- The Include filter will limit enforcement to requests that have these Tags attached. All other requests in the traffic stream will not have this Rate Limit enforced upon them.
- The Exclude filter will exempt requests from enforcement that otherwise would have been subject to it.
(Internally, Curiefense evaluates Exclude parameters first, and then Include parameters.)
To add one or more filters, select New entry, define the parameters, then select the "+" button. If more than one Include filter is specified, they are combined with a logical AND.