Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Curiefense maintains its security parameters in Documents, which contain Entries. (Read more about Curiefense's data structures.)
The Document Editor is where Documents and their Entries are created, defined, and administered.
Curiefense processes incoming requests according to this traffic flow:
Each incoming request is inspected, and tags are assigned to it. For example, if the request's IP was found on the Spamhaus DROP list, it could be assigned a "spamhaus" tag. Some tags are generated automatically, while others are user-defined. (Read more about tags.)
Next, Curiefense determines the security ruleset(s) that have been assigned to the request's target URL, and which match the tags. For example, there might be a ruleset defined for the "spamhaus" tag, or for the "devops" tag.
Curiefense then enforces the ruleset(s), and takes the defined Action(s). For example, "block requests from Spamhaus-listed IPs", or "bypass devops requests from further filtering."
This process is based on the Documents as follows:
Profiling Lists is the Document which defines tags for external lists and custom lists.
ACL Profiles, Rate Limits, and WAF Profiles define security rulesets, i.e. the actions to take when specific tags and/or other criteria are observed.
URL Maps assign the security rulesets to internal URLs.
This page is divided into three vertical sections. From top to bottom, they are:
Administration
Entry editing
Versioning
Each is discussed below.
After editing anything on this page, you must save your changes (with the Save button on the upper right) and then publish them.
The top section contains a toolbar with input controls.
Configuration pulldown: Selects the branch/configuration for editing.
Document pulldown: Selects the Document to display for editing.
Entry pulldown: Selects the Entry (the ruleset) that is being displayed for editing.
Duplicate button: Makes a copy of the currently displayed Entry.
Download button: Downloads the currently displayed parameters.
Add button: Creates a new Entry of the currently selected type.
Save button: Saves all changes that were made since the last Save action.
Delete button. Deletes the currently selected Entry.
The UI in this section will vary, depending on the type of Document being edited. Each is discussed in more depth here:
At the bottom of this section, the URL of the current entry is shown in gray. It reflects the structure of the data.
For example, /conf/api/v1/configs/devops/d/urlmaps/e/__default__ shows that:
The current Configuration (or branch) is devops.
The current Document is urlmaps.
The current Entry is __default__.
The bottom section of this page shows a history of versions for this Document. In a later version of Curiefense, this section will allow for reversions. In the current version, reversions are only available in the API and in the Publish Configuration section of the UI.
Rate Limits are rules which define the number of requests with certain characteristics that are allowed within defined time frames. When a request is received that exceeds a Rate Limit, a specified action is performed.
The input controls at the top of this page are described here: Document Editor interface. Specific editing of a Rate Limit is described below.
A Rate Limit defines the number of times that requests can match certain conditions within a certain time frame. Once that limit has been reached, subsequent requests matching those conditions within the same time frame will trigger an action.
The matching conditions are specified with the parameters in the "Count by" section and the optional Event. See Matching Conditions below.
After a Rate Limit is defined on this page, it will not be active until it is assigned to one or more URLs. This is done in the URL Maps Document.
By default, a Rate Limit will be enforced for all requests for the URL(s) to which it is assigned.
Enforcement can be further limited to a subset of these requests: see Including/Excluding Requests below.
A condition consists of a field and a value. Within a Rate Limit, they play a role like this:
"More than <THRESHOLD> requests with the same <CONDITION-VALUE> <CONDITION-FIELD> sent to <ASSIGNED-LOCATION> within <TTL-PERIOD> will cause <ACTION>."
Example: "More than three requests with the same username argument sent to the login form within one hour will cause the requestor to be banned for six hours."
Note that the <ASSIGNED-LOCATION> mentioned above is included for illustration purposes only. This value is not part of a Rate Limit definition—it is defined in a URL Map.
A condition can be built upon any one of these four fields:
Multiple conditions can be defined within the same Rate Limit. To create a new condition and open it for editing, select "New entry" below the list of conditions.
If multiple conditions are defined, they are evaluated by combining them together with a logical AND. In other words, the cumulative count toward the Threshold will be incremented whenever a request is seen that matches all of the conditions simultaneously. Different combinations of conditions will have separate Threshold counts maintained for them.
Example. A Rule contains two conditions: "Attribute / Remote Address" and "Argument / Username". When the first request is received, an internal counter is created (set to a value of one) for this unique combination of Remote Address and Username. A second request is then received, originating from the same Remote Address and for the same Username; this causes the internal counter to be incremented up to two. A third request is then received from the same Remote Address but with a different Username; this causes a new internal counter to be created (and set to a value of one) for this combination.
Below the list of condition(s), there is another condition named "Event."
By default, this is set to "HTTP request,", which simply means to increment a counter each time a request is received that matches the conditions.
However, if the Event condition is changed to a different value, then the following applies.
In the discussion below, "Count Condition" will refer to the condition (or combination of conditions) defined by the "Count by" input controls.
"Event Condition" will refer to the optional, additional condition defined by the "Event" input controls.
Adding an Event Condition changes the evaluation process. An Event Condition is not logically combined with the preceding Count Condition; it is always evaluated separately.
More importantly, adding an Event Condition changes the meaning of the Rate Limit.
If an Event Condition is not defined—in other words, if "HTTP request" is selected—then as discussed above, an internal counter is maintained for each Count Condition value, and incremented each time that value is encountered in a request.
If an Event Condition is defined—in other words, if something other than "HTTP request" is selected—an internal counter is maintained for each Count Condition Value, and incremented each time a new, previously unobserved Event Condition value is encountered in a request.
Therefore, if an Event Condition is defined, the Rate Limit constrains the number of allowable Event Condition values for any given Count Condition value.
So, the evaluation becomes something like this:
"More than <THRESHOLD> <EVENT-CONDITION-VALUE> <EVENT-CONDITION-FIELD>s per any one <COUNT-CONDITION-VALUE><COUNT-CONDITION-FIELD> sent to <ASSIGNED-LOCATION> within <TTL-PERIOD> will cause <ACTION>."
Note that the number of Count Condition values is not being limited here. The limit is on the number of Event Condition values that each Count Condition value is allowed.
Example: Let's say we want to allow an individual user to login from a maximum of two ASNs within one hour. (Perhaps the user is accessing our web application from a coffee shop's WiFi, and then a few minutes later, leaves the coffee shop and begins using the cell network instead.) We want to allow this possibility; however, if we receive requests from the same user originating from three or more ASNs within an hour, we want to treat this traffic more suspiciously. This is not possible merely by specifying two Count Conditions, as described earlier in the "Multiple Conditions" section. If we set up two conditions ("Argument / Username" and "Attribute / Organization") with a Threshold of 2, and assign it to our login form, then this will merely limit the number of times that the user can login from each ASN within an hour. Instead, we can set up one Count Condition ("Argument / Username") and then set up the Event Condition ("Attribute / Organization"). Now the Threshold will apply to the number of Organizations that are observed for each specific Username.
When an incoming request exceeds the Threshold, the Action specified here will occur.
Most of the Actions listed above will not fully exclude an attacker that continues pressing the attack.
Example: Access to a login form is rate-limited to three requests per minute. An attacker tries to brute-force the login, and sends 60 requests per minute. The Rate Limit allows the first three requests, but then blocks the next 57 requests with a 503 error. However, after the minute has passed, the Rate Limit resets. The attacker is allowed another three attempts before being temporarily blocked again. This cycle can continue for as long as the attacker wishes. In effect, the Rate Limit is not preventing the attack; it is merely slowing it from 60 attempts per minute down to three attempts per minute.
The Ban action can be used to block (or take some other Action in response to) a Rate Limit violator for an extended period of time.
Example: As described above, a Rate Limit is created to allow three requests per minute, with an Action of Default (503). However, an additional Rate Limit rule is also defined: nine requests per three minutes, with an Action of Ban. The Ban has an Action of Default (503), and a duration of one hour. URL Maps allow for multiple Rate Limits to be assigned to a single URL. Thus, both of the above rules can be assigned to the login form. Now an attacker tries to brute-force the login form, sending 60 requests per minute. The first three requests are allowed. The next six requests are blocked (and a 503 error is issued) by the first Rate Limit. The tenth request triggers the second Rate Limit, and the Ban occurs. For the next hour, the attacker's requests will be blocked with a 503 error.
Second example: A hostile bot receives a bot challenge, which it fails. Curiefense will block the request. If the bot keeps re-submitting its request, it will continue to fail the challenges. However, each time the bot tries again, Curiefense has to issue a new challenge . To solve this problem, a second Rate Limit is created with a Ban action. Now a persistent bot will simply be Banned, saving Curiefense the overhead of issuing continuous challenges.
Note that when setting up a Ban, the most common choices for its Action are to deny the violator's requests (via Default 503, Response, or Redirect). However, you can also choose Monitor (to observe the violator's actions during the ban period), Challenge (to verify that the violating activity is not being done by bots), or Header (to mark the requests for further scrutiny by the backend).
By default, an active Rate Limit rule will be enforced upon all incoming requests targeting URLs to which this rule has been assigned.
At the bottom of this page, you can add Include and/or Exclude parameters. These define the portion of the incoming traffic stream that will be evaluated for possible violation of the Rate Limit. In other words, they limit the scope of the Rate Limit's enforcement:
The Include filter will limit enforcement to requests matching its parameters. All other requests in the traffic stream will not have this Rate Limit enforced upon them.
The Exclude filter will exempt requests from enforcement that otherwise would have been subject to it.
(Internally, Curiefense evaluates Exclude parameters first, and then Include parameters.)
To add one or more filters, select "New entry", define the parameters, then select the "+" button. If more than one Include filter is specified, they are combined with a logical AND.
This section displays all the WAF Signatures defined within Curiefense. This has various uses; for example, looking up the underlying signature for a tag that appears in the Access Log, or getting the ID of a signature to exclude in a WAF Profile.
Curiefense comes with many WAF Signatures. You can view them by choosing different IDs in the pulldown on the upper right.
Note that this list is for reference only. The entries cannot be edited.
The input controls at the top of this page are described here: . Specific editing of a URL Map is described below.
This page specifies a list of URLs and the security policies assigned to them.
Every incoming HTTP/S request targets a specific URL. Curiefense finds the best match for that URL in the URL Maps, and applies the security policies defined for it.
The "best match" is determined by regex evaluation. (The order in which the URLs are listed in the interface does not matter.) If no matching definition is found, Curiefense applies the rulesets from the default definition.
A URL Map consists of:
Host definition: The (sub)domain(s) within which the Path Maps will be found.
Path Maps: one or more paths, and the security policies which will be applied to them.
To add a new URL Map, use the buttons at the top of the window to duplicate an existing one or create a new one. Then fill in these fields.
To add a new Path Map, select an existing one in the middle of the window, expand it, and select "Fork Profile" at the bottom. The existing one will be cloned, and the new one will be displayed for editing.
Note that the buttons at the top of the window are for administering URL Maps (which generally correspond to domains). Administering Path Maps (for paths and URLs within the specified domain) is done in the middle of the window.
To edit an existing Path Map definition, click on the "expand" button at the end of its listing. The edit window shown below will appear.
In addition to editing the fields discussed above, this window also provides the ability to:
Deactivate the WAF Profile (by unselecting its Active Mode checkbox).
Deactivate the ACL Profile (by unselecting its Active Mode checkbox).
Assign (or remove) Rate Limit rules to this Path Map, via the + button (or remove button). The + button will only be shown if there are unassigned Rate Limit rules available.
Create a copy of this Path Map, and open it for editing, via the Fork Profile button.
Field
Value
Name
A name that will be used within the Curiefense interface, and will also be used to create a tag that will be assigned to requests which trigger the Rate Limit. It is recommended that the name summarizes the rule; for example, a rule with a Threshold of 5 and a TTL of 60 could be named "Rate limit rule 5/60".
Description
A description that will be used within the Curiefense interface.
Field
Value
Threshold
The maximum number of allowable requests within the specified TTL. Subsequent requests within the TTL will trigger the Action.
TTL
Time to Live: the period (specified in seconds) within which the Threshold is enforced.
Field
Result
Header
All requests with the same value for the specified header will be counted together toward the Threshold.
Cookie
All requests with the same value for the specified cookie will be counted together toward the Threshold.
Argument
All requests with the same value for the specified argument will be counted together toward the Threshold.
Attribute
All requests with the same value for the specified attribute will be counted together toward the Threshold.
Action
Meaning
Default (503)
The request will be blocked and the requestor will receive a response of "503 Service Unavailable".
Challenge
For a browser-based web application, a bot challenge will be issued to verify that the requestor is a human using a browser, and not a bot using a headless browser or emulator. If the challenge is failed, the request is blocked.
Monitor
The request will not be blocked; it will merely be tagged with the Rate Limit's name, for subsequent viewing in the Access Log and other places. This Action is useful for testing new Rate Limit rules without actually affecting incoming traffic.
Response
Blocks the request, and responds with a custom error code (0-999) and response body.
Redirect
Blocks the request with a custom error code, and redirects the requestor to a specified URL. For example, the URL might be a page that says, "Your activity appears suspicious, and your access has been restricted. Contact support if you think this decision was made in error."
Ban
Blocks the requestor for the specified amount of time. See further discussion below.
Header
Does not block the request, but adds headers to it (indicating the Rate Limit rule name and the threshold) for receipt and evaluation by the user's backend.
Field | Value |
(unlabeled) | The name of the URL Map is displayed for editing on the upper left. |
Matching Names | A regex for the subdomain(s) and/or domain(s). |
Field | Value |
Name | A descriptive label for use within the interface. |
Match | An expression for the path, expressed as PCRE (Perl Compatible Regular Expressions). |
WAF |
ACL |
RL |
The input controls at the top of this page are described here: . Specific editing of a WAF Profile is described below.
A WAF Profile is a set of security policies that are used by the Curiefense WAF (Web Application Firewall). Every deployment includes a default WAF Profile, and additional Profiles can be created.
At the top of the page, the following values are defined for incoming requests.
However, parameters can be whitelisted and exempted from this filtering. For each parameter, this can be done in two ways:
Full exemption is available by specifying a regex pattern which, if it matches the parameter's value, will exempt that parameter from WAF Signature evaluation.
Partial exemption is available by specifying a list of WAF Signatures which will not be evaluated, even if the regex pattern is not matched.
Along with this content whitelisting, a "positive security" form of content filtering is also available. Curiefense can be configured to require certain content in a specified parameter, and reject requests that do not contain it.
The bottom part of the UI defines Curiefense's behavior for both whitelisting and content filtering for each parameter.
In the following discussion, a constraint refers to the values in the UI input controls (Parameter, Matching Value, Restrict?, and Exclude Sig) that have been specified for one parameter.
Each incoming request is processed like this:
This behavior is defined in the following fields in the UI.
Constraints are defined in the same way for Headers, Cookies, and Arguments within their respective tabs.
An ACL (Access Control List) Profile is a set of sequential criteria by which a request is evaluated. The input controls at the top of this page are described here: . Specific editing of an ACL Profile is described below.
This page defines ACL Profiles. The Profiles are assigned to URLs within the web application via .
Out of the box, Curiefense includes a default ACL Profile. It can (and usually should) be edited, but it cannot be deleted. It is used for all URLs where no other Profile has been assigned.
In the interface, the tags are listed below their associated actions. Evaluation is performed from left to right:
The incoming request is evaluated to see if it matches any of the tags in the far left column.
If a match is found, that column's action is performed, and no further evaluation is done.
If a match is not found, then the next column to the right is evaluated. A match will trigger that column's action; otherwise the next column is evaluated.
This process continues until an action is taken or all columns have been evaluated.
If a request does not match any tags in this Profile, it is passed to the WAF for further inspection.
Be very cautious when adding the "all" tag to an action column. In most situations, that action will be taken for all requests for which a match was not found in any columns to the left.
Another possible use is to place "all" in the Deny column, to create a positive security model. (This will block all requests which did not explicitly meet any of the conditions to be Bypassed or Allowed.) But even this usage can be dangerous. Unless all the conditions for Bypassing or Allowing requests are fully and correctly defined, this can result in False Positive errors, and some legitimate requests will be blocked and filtered.
In other columns, "all" can have serious consequences. For example, placing it in the Enforce Deny column will block all incoming traffic to which this ACL Profile is applied. On the other hand, placing it in the Bypass column will allow all incoming traffic (and exempt it from being scrubbed by the WAF!), except for those requests which matched a tag in the Enforce Deny column.
Summary: before using the "all" tag in an ACL Profile, carefully consider its ramifications.
The input controls at the top of this page are described here: . Specific editing of a Profiling List is described below.
Each Profiling List consists of:
Match conditions: A list of possible characteristics that a request can match (e.g., a list of IP addresses that it might originate from), plus the logical operator to use when evaluating the match.
One or more Tags to assign when a match occurs.
For each incoming request, Curiefense will evaluate all active Profiling Lists. A single request will receive Tags from all Lists which match it.
Profiling Lists can be either Internet-sourced or self-managed.
Internet-sourced Lists are based upon online sources (e.g., Spamhaus DROP lists). The entries for these lists are not editable within the interface, because they are obtained automatically; Curiefense updates them every 24 hours.
Self-managed Lists are created manually. These are fully editable within the interface.
Match conditions consist of two parts:
A list of one or more criteria to match. The list will be entered differently for the two types of Profiling Lists.
The logical condition to apply (either OR or AND), specified in the Entries Relation pulldown
If the list contains more than category--which is unlikely for an Internet-sourced Profile, but not impossible--also choose the appropriate value for Entries Relation, as discussed below.
At this point, the new Profiling List should be complete. Before exiting the page, be sure to save your work.
To add a match criterion, select the "+" button at the top of the criteria list. The following dialog will appear.
For most of the categories, the dialog will appear as it is above. Multiple entries can be made at once, with each entry on a separate line. Each line contains the value, plus a pound sign (#) followed by an annotation (a label for display within the Curiefense interface). Example:
For some categories, one entry can be made at a time, with each entry requiring multiple lines. Annotations are not preceded by a pound sign.
Match criteria are case-insensitive.
Here are some sample entries for the various categories. Notice that the logical operators are available.
Once created, these entries cannot be edited. If one needs to be modified, remove it and re-create it.
This specifies the logical operation used when evaluating a request against the match criteria.
When all the match criteria are in the same category, the operation is OR. For a request x and list of criteria a, b, c , then the evaluation will be (x==a) OR (x==b) OR (x==c) For example, if the match criteria are all IP addresses, then a match will occur if the request matches any of the IPs in the list.
When there are two or more categories, the default operation is still OR (in which case, a match will occur if the request matches any of the criteria in the list). However, when there are multiple categories, you can select AND instead. In this case, OR will still apply within each category, while AND will apply between the categories. Thus, for one list a, b, c and a second list i, j, k , the evaluation will be ((x==a) OR (x==b) OR (x==c)) AND ((x==i) OR (x==j) OR (x==k)). For example, if the match criteria are IP addresses and headers, then a match will occur if the request matches any of the IPs and any of the headers, but it will not occur if it matches an IP but none of the headers.
The applied to this path. Its name will be displayed in green if it is active; if displayed in red, it is currently disabled.
The applied to this path. Its name will be displayed in green if it is active; if displayed in red, it is currently disabled.
The number of assigned to this resource.
Every URL that Curiefense protects has a WAF Profile assigned to it in . (If none is assigned explicitly, the default is used.) A request sent to a URL might, or might not, be filtered according to the assigned Profile.
The request was blocked before WAF filtering would have occurred. (Before the WAF is used, .)
The applicable resulted in an Action of Bypass, which exempts the request from WAF filtering.
The WAF is not in for this URL.
By default, an incoming request will be compared to all the . If any parameter (any header, cookie, or argument within it) fails this evaluation, the request will be blocked.
The top part of the interface shows the ACL Profile that is currently being edited. Each Profile consists of one or more . Each tag is listed under the action which will be triggered when a request matches that particular tag.
The most common use is the "Deny Bot" column, as noted above. This means that all requestors (except for those previously Allowed or Bypassed) will be and verified to be humans.
Early in the (immediately after the pre-processing Cloud Functions are performed), Curiefense assigns Tags to an incoming request. Subsequently, the Tags can be used to make decisions about how the request is processed. After processing, a request's Tags remain associated with it, and they are available for display in the .
This page allows you to administer Profiling Lists, which are combinations of and the criteria for assigning them to requests.
The actions that can be performed as a result of the Tag assignments are administered separately in .
For a Profile based on an online source, simply enter its URL into the Source field. For example, to create a list based on the Spamhaus ASN DROP list, you would enter . Curiefense will then populate the list automatically.
Constraint | Meaning |
Max Length | The maximum allowable length of a header, cookie, or argument. |
Max Count | The maximum number of headers, cookies, or arguments allowed. |
Ignore Alphanumeric input | When this is selected, the WAF will not inspect requests that only contain alphanumeric characters. This reduces computational overhead by not evaluating benign requests. (Hostile requests such as SQLi, XSS, etc., will contain some non-alphanumeric characters.) |
Field | Value |
Parameter | The parameter whose value will be compared to the Matching Value. This can be provided as a specific Name (e.g., sessionid), or as a Regex to match multiple parameters (e.g., user_.+). Note that a Name will be marked with a capital "A", while a Regex will be marked with "</>". |
Matching Value | A regex pattern. If a parameter's value matches it, the parameter will be exempted from WAF Signature filtering. If it does not match, the Restrict? option becomes relevant. |
Restrict? | If a parameter does not match the Matching Value, and Restrict? is selected, then the request is blocked. If Restrict? is not selected, then the Exclude Sig option becomes relevant. |
Exclude Sig | A parameter which fails its Matching Value comparison, and for which Restrict? is not selected, will be filtered by all WAF Signatures except the ones whose IDs are listed here. (For example, some WAF Signatures filter out special characters; if a certain parameter can legitimately contain these characters, it would make sense to exempt that parameter from those specific filters .) It is also possible to always exempt a parameter from specific WAF Signatures:
|
Field | Meaning |
Name | A description that will be displayed within the Curiefense interface. |
Tags | One or more Tags (separated by spaces) that will be assigned to requests if the match conditions are fulfilled. Example: |
Source | For an Internet-sourced list, the URL of the source. Otherwise, this is |
Entries Relation | See "Specifying Match Conditions", below. |
Active | By default, the Profiling List will be applied to incoming requests. To prevent this, unselect the checkbox. |
Notes | An optional field for including additional information. |
Action | Comment |
Enforce Deny | Blocks the request. This is a useful way to quickly filter out, with minimal computing overhead, large numbers of requests that are obviously hostile or otherwise unwanted. |
Bypass | Allows the request, and bypasses/exempts it from subsequent evaluation by the WAF. |
Allow Bot |
Deny Bot | If the requestor has not previously been verified to be human, a bot challenge will be issued. If the challenge is passed, the request will continue in the evaluation process. Otherwise the request is blocked. A common use of this column is to add the "all" tag, which means to identify and block all bots that weren't previously Allowed or Bypassed. |
Allow | Passes the request to the WAF for further inspection. |
Deny | Blocks the request. |
Passes the request to the WAF for further inspection without triggering a bot (the process by which non-human traffic is identified and blocked). An example usage is to allow search engine spiders.
