This page describes the tasks necessary to deploy Curiefense using Docker Compose. It assumes that the instructions described in First Tasks have been completed successfully.

This process consists of the following tasks, described sequentially below:

• TLS Setup

• Set Database Credentials

• Set Deployment Variables

• Deploy Curiefense

• Test the Deployment

• Clean Up

If during this process you need to rebuild an image, see the instructions here: Building/Rebuilding an Image.

TLS Setup

If you want Curiefense to use TLS, then you should have already generated the certificates and keys.

To enable TLS for the protected site/application, go to curiefense/deploy/compose/curiesecrets/curieproxy_ssl/ and do the following:

• Edit site.crt and add the certificate.

• Edit site.key and add the key.

To enable TLS for the nginx server that is used by uiserver, go to curiefense/deploy/compose/curiesecrets/uiserver_ssl/and do the following:

• Edit ui.crt and add the certificate.

• Edit ui.key and add the key.

Set Database Credentials

The logdb database has two accounts:

• Thepostgres account has write access, and is used by curielogger.

• Thelogserver_ro account has read-only access, and is used by curielogserver.

If you wish to change the default passwords for these accounts, you must edit the files in which they are defined:

• The password for postgres is defined in curiesecrets/logdb/postgres_password.txt.

• The password for logserver_rois defined in curiesecrets/logdb/ro_password.txt.

Set Deployment Variables

Docker Compose deployments can be configured in two ways:

• By setting values for variables in deploy/compose/.env

• Or by setting OS environment variables (which will override any variables set in.env)

These variables are described below.

CURIE_BUCKET_LINK

Curiefense uses the storage defined here for synchronizing configuration changes between confserver and the Curiefense sidecars.

By default, this points to the local_bucket Docker volume:

$ grep CURIE_BUCKET_LINK .env
CURIE_BUCKET_LINK=file:///bucket/prod/manifest.json

For multi-node deployments, or to use S3 for a single node, replace this value with the URL of an S3 bucket:

CURIE_BUCKET_LINK=s3:///BUCKETNAME/prod/manifest.json

In that case, you will need to supply AWS credentials in deploy/compose/curiesecrets/s3cfg, following this template:

[default]
access_key = AAAAAAAAAAAAAAAAAAAA
secret_key = AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA

TARGET_ADDRESS and TARGET_PORT

The address of the destination service for which Curiefense acts as a reverse proxy. By default, this points to the echo container, which simply echoes the HTTP requests it receives.

DOCKER_TAG

Defaults to latest (the latest stable image). To run a version that matches the contents of your working directory, use the following command:

DOCKER_TAG="$(git describe --tag --long --dirty)-$(git rev-parse --short=12 HEAD:curiefense)"

Deploy Curiefense

Once the tasks above are completed, run these commands:

cd curiefense/deploy/compose/
docker-compose up

Test the Deployment

After deployment, the Echo service should be running and protected behind Curiefense. You can test the success of the deployment by querying it:

$ curl http://localhost:30081/
Echo

Also verify the following:

• The UIServer is now available at http://localhost:30080 (access logs are at http://localhost:30080/accesslog)

• Grafana is now available at http://localhost:30300

• The confserver is now available at http://localhost:30000/api/v1/

Clean Up

To stop all containers and remove any persistent data stored in volumes, run the following commands:

docker-compose rm -f && docker volume prune -f

The full deployment process for Curiefense:

• First, complete the general tasks described in First Tasks

• Then complete the specific tasks for your environment:

  • Docker Compose

  • Istio via Helm

Prerequisite: Ubuntu 20.04 LTS

In this guide, we will deploy Curiefense using Docker Compose, then test and configure.

If you want to use Helm instead of Docker Compose, or if you want more control over the deployment options, then do this:

1. Go to the Deployment in Depth: First Tasks page and follow its instructions.

2. Go to the appropriate Deployment in Depth page (Docker Compose or Helm) and follow its instructions.

3. Return here to the Verify the Deployment section below, and continue.

Let's begin

To ensure we have all tools in place, we will run:

sudo apt update
sudo apt install -y git docker.io docker-compose libpq-dev python3-dev gcc python3-psycopg2 vim
sudo usermod -aG docker $(whoami) && sudo -i -u $(whoami)

Grab the latest code from GitHub to get started:

git clone https://github.com/curiefense/curiefense.git
cd curiefense/deploy/compose/
docker-compose up

ProTip: Try Curiefense out on Katacoda, no install necessary.

Verify the Deployment

At this point you should have the following http interfaces:

Make sure everything is working by testing the echo server:

curl curie.demo:30081

Request served by 1cebc4ed2938

HTTP/1.1 GET /

Host: curie.demo:30081
X-Envoy-Expected-Rq-Timeout-Ms: 15000
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Accept-Encoding: gzip, deflate
X-Request-Id: 50d6bdea-58ed-4c9d-9d26-c8eeff432817
Cookie: csrftoken=u97m660E9Pji8PvSpAGWU3K4pVsl2N4mTscK00bvO8Jkb7h3nTTQQR8FuArzPgIk; CSRF-Token-X3TDN=bEYiHVfq9RvWY4UbyeLN4CWrfqyLevuj; redirect_to=%2F
Upgrade-Insecure-Requests: 1
X-Forwarded-For: 172.18.0.1
X-Forwarded-Proto: http
X-Envoy-Internal: true
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:81.0) Gecko/20100101 Firefox/81.0
Accept-Language: fr,fr-FR;q=0.8,en-US;q=0.5,en;q=0.3
Content-Length: 0

Deployed Containers Overview

This section describes the newly-described components of Curiefense. Feel free to skip it if desired, and go straight to the instructions for policy and rules configuration.

What Have We Just Deployed?

This diagram will help us understand the containers we just deployed, their connections, and the data flow:

Terminology and Concepts

Before diving in and making changes, let's discuss a few concepts of Curiefense's configuration.

• Git is the storage management used to keep track of changes. This means:

  1. Data can be stored anywhere a git repository can (local, remote, hosted, etc).

  2. Every change you made can be reverted.

  3. You can automate deployments based on tagging.

  4. Using a single configuration server, you can maintain configurations of multiple deployments (e.g. production, devops, qa, rc, etc.) by keeping each in a separate branch, and you can merge them the git way at any point in time, via the API and/or UI.

• Configurations are organized in Documents and Entries. More on this here.

Policy and Rules Configuration

Curiefense runs every incoming request (and in some cases, responses as well) through a series of mechanisms. We will now walk through some of them to understand how traffic is handled and processed.

During the procedures described below, you will set up some simple rules and then run some requests through Curiefense. By the end of this process, you will understand how to create security rules and policies, you will observe them being applied, and you will see how Curiefense reports on incoming requests and its reactions to them.

Open the UI

Open the UI management console by going to http://curie.demo:30080/. In the left sidebar, select Document Editor if it is not already selected.

At the top left of the page, in the second pulldown control, select Profiling Lists.

Create a Session Profiling list

Session profiling attaches tags to requests and sessions based on various criteria, from matching headers, cookies, arguments or URLs to traffic sources such as geolocations, IP addresses, CIDRs, and AS numbers. Subsequently, the tags are then used to make decisions about how the requests are handled.

Start by creating a new Profiling List:

Next, in the Tags text box, enter the value trusted .

Then, at the top of the (currently empty) list to the right, add a new entry by selecting the + button. Enter foo for its name, bar for value, and trust me for annotation. Now click on the "add" link.

Your screen should look similar to this:

We have created a simple profiling rule. Every request that contains a header named foo which matches the regex (PCRE) bar will receive a tag of trusted.

(The Annotation of trust me is a label for internal admin use, and does not affect traffic processing.)

Now save the new configuration:

And then publish it:

Now it is time to test our configuration. Let's run the following curl commands:

curl http://curie.demo:30081/no/header
curl http://curie.demo
:30081/with/header -H "foo: bar"

Navigating to Access Log in the left sidebar should show a screen similar to this:

Click on the /with/header entry, and notice the trusted tag on the right column:

Notice also that along the trusted tag you defined, Curiefense attached a number of tags that were generated automatically. More information about this.

Profiling Lists are a powerful feature of Curiefense (and are explained in depth here). We just demonstrated the ability to create a single-entry self-managed list that characterizes incoming requests based on a header. Curiefense allows you to attach tags based on complex combinations of headers, arguments, cookies, geolocation, methods, paths, and more. External data sources are also supported; sessions can be profiled based on data and rules defined by a third party, such as blocklists and whitelists.

Now that we know how to attach tags to incoming requests, let's tell Curiefense how to react to them.

Create an ACL (Access Control List)

We're going to block all requests with the trusted tag.

In the left menu, navigate to Document Editor. By default, ACL Profiles should already be selected.

Enter trusted into the DENY column, then save your changes.

Your screen should look similar to this:

Publish the new configuration again (and wait 15 seconds for the changes to propagate).

Test the ACL

Run this command:

curl http://curie.demo:30081/with/header -H "foo: bar"

In the Access Log, you should see the request listed. Expanding it will show this:

Note that the request was identified as a risk, because it contains the specified tag and therefore matches the DENY ACL.

However, Curiefense passed the request through the system, as seen in the green HTTP status code (200: successful). If it had been blocked, the code would instead be red with an error code.

This behavior is expected. By default, Curiefense's security profiles are in report/monitor mode; requests will be flagged but not blocked. This mode allows for testing and fine-tuning of new configurations without affecting traffic.

Let's assume that we've tested our new policies and want to make the ACL active.

Activate the ACL

Navigate to Document Editor and then choose to edit URL Maps in the upper dropdown list.

URL Maps assign security policies to paths within the protected application. You can assign policies at any scale, from globally down to individual URLs. (They are explained in depth here.)

We're going to edit Curiefense's default security profile: the one that applies to every URL which does not otherwise have any policies assigned to it.

Expand the default profile (the one assigned to path /) by selecting it. Then activate the profile by checking the Active Mode checkbox.

Save your changes and publish the configuration again.

After the change propagates, Curiefense should block the request:

curl http://curie.demo:30081/with/header -H "foo: bar"
curiefense - request denied

Add Rate Limiting

You have seen how to create security policies that filter hostile requests, and how to assign these policies to paths within your application.

Now we'll see how to filter requests that are not obviously hostile at first, but which display hostile intent with increased volume. For example, a user who fails a login attempt might have mistyped their password—but a user who fails ten login attempts in a short time is probably trying to guess the password for an account they do not own.

Add a Rate Limit

Return to URL Maps and select the default profile again. At the bottom of its map, there is the Rate Limit Rules section. Currently, it is empty.

Attach an existing rule by select the first "here" link. A pulldown list of available Rate Limit Rules will be displayed.

Open the list, and select its only entry (the one that comes by default with the system). This rule limits requests from a given IP address to a maximum of 5 requests per 60 seconds.

Select the "add" link to add this rule to the default URL Map.

Save your changes, and then publish.

After propagation, we can test it:

while true; 
do 
    curl -s -o /dev/null -w "%{http_code}" http://curie.demo:30081/with/header -H "foo: bar";
    printf ", "; 
    sleep 1s; 
done

Wait briefly, and you should see how the response changes from 403 (ACL) to 503 (Rate limit) on the 6th request.

403, 403, 403, 403, 403, 503, 503, 503, 503, 503, 503, 503, 503...

Add Multiple Rate Limits

Now that we're somewhat familiar with the system, let's set up multi-layered rate limiting to protect against a variety of attacks.

Previously, we used the default Rate Limit that comes with Curiefense, and applied it to the entire domain. Now we will create three specific Rate Limits for the login process of an API, and attach them to the relevant endpoints.

Creating a URL Map for the API

Create a new URL Map by duplicating the default one:

Set its name (the unlabeled field at the top) to API. Set Matching Names to api.curie.demo.

Adding Profiles

Note that the new URL Map contains a default profile. As you might expect, this will apply to every path within the scope set by the Matching Names entry. (In this tutorial, the Matching Names is a specific subdomain. In production use, this might be a regex describing a range of domains, subdomains, or URLs.)

To add a profile to a URL Map, open an existing profile (in this case, the default) and select the Fork profile button.

Edit the Name and Match condition of the profile and then repeat this process twice more, so that there are four profiles in total. The Names and Match Conditions are shown below:

At this point, we have four profiles: the default / which we left in place, /api/v1/ and /api/v2 to match API calls of different versions, and an entry for /login that catches both versions.

Setting Rate Limits

Note that the default has one Rate Limit attached to it (shown as a "1" in the RL column), because it already had this when it was duplicated. The other three profiles do not yet have any limits.

For /api/v1/ and /api/v2/ , set the standard rate limit rule as you did before:

For /api/v[1-2]/login, we will create three new rate limit rules.

Create New Rate Limits

Open the Rate Limits section of the UI by selecting Rate Limits at the top of the window (in the pulldown list that currently says URL Maps).

The default Rate Limit should be displayed. Add a new Rate Limit by duplicating it, using the Duplicate Document button in the top right part of the interface.

Rule #1: Limiting IPs

Edit the new Rate Limit so that it limits login attempts from the same IP address, allowing a maximum of three within 10 minutes:

Rule #2: Limiting locations

Create a second Rate Limit by duplicating an existing one. Edit it so that it limits a given User-Id header to a single country within a four hour timespan:

Rule #3: Limit number of User IDs

Create a third Rate Limit by duplicating an existing one. Edit it so that it limits each IP to submitting a maximum of eight unique User IDs:

Using the new Rate Limit rules

Return to the URL Maps section of the interface. Let's attach the new rules to the /api/v[1-2]/login matching path.

Expand the appropriate profile, select the link at the bottom ("To attach an existing rule..."), and select the first of the three new rules you created. Repeat for the second and third rules. When you're finished, you should see something similar to this:

Testing the new Rate Limits

Next, let's try to brute-force the login endpoint, using the same user id and IP, but with a new password each time:

while true; 
do 
  curl -s -o /dev/null -w "%{http_code}" \
     http://api.curie.demo:30081/api/v1/login \
     --data "userid=me@gmail.com&password=$RANDOM" ; 
  printf ", "; 
  sleep 1s; 
done
200, 200, 200, ... 503, 503, 503, 503, ...

To test the other rule, we will have curl send a unique User ID each time:

while true; 
do 
  curl -s -o /dev/null -w "%{http_code}" \
     http://api.curie.demo:30081/api/v1/login \
     --data "userid=$RANDOM@gmail.com&password=$RANDOM" ; 
  printf ", "; 
  sleep 1s; 
done

As traffic gets blocked (returning a 503 error by default) we can look at the Access Log. When expanding a 503 line, you can see the Risk Details, which include the rate limit rule that was violated (in this case, Dictionary Harvesting).

You can also see the system has generated a tag named after the rule name. Lastly, notice that every request is logged with all details.

Advanced Rate Limiting

The demonstration above is only the beginning of what can be done with Rate Limiting. You can configure Curiefense to limit complex combinations of conditions and events, customize its reactions (including blocking, redirecting, monitoring, responding with custom codes, adding headers to requests for backend evaluation), and more.

Here's a common example. Rate Limits like the ones demonstrated above will block a requestor who exceeds a defined limit within the given timespan. However, once the limit resets, the requestor would be able to try again, and could repeat this cycle as often as desired. To prevent this, you can configure Curiefense to ban a requestor (and block all of their requests) when multiple rate limits are violated.

A full explanation of Curiefense's Rate Limits and their capabilities is available here.

Grafana Dashboards

The docker-compose deployment process will create Grafana visualizations for Curiefense's traffic data.

Curiefense comes with two dashboards out of the box: Traffic Overview and Top Activities. They are available at http://curie.demo:30300/.

Traffic Overview Dashboard

Top Activities

Onwards

You now have a working Curiefense installation to experiment with. Its capabilities go far beyond those demonstrated in this tutorial; browsing through the rest of this Manual (especially the Document Editor section) will give you some ideas to test, so you can see what the platform can do.

Note that this tutorial was a Quick Start guide, and therefore, it used many default options for the deployment. You might want to change some of them; for example, you might want Curiefense to use TLS for its UI server and for communicating with the backend.

Modifying the deployment is straightforward. For Docker Compose, just go through the procedures described here and then re-run docker-compose up. For Helm, go through the procedures here.

Curiefense maintains its security parameters in Documents, which contain Entries. (Read more about .)

The Document Editor is where Documents and their Entries are created, defined, and administered.

Documents and traffic flow

Curiefense processes incoming requests according to this traffic flow:

• Each incoming request is inspected, and tags are assigned to it. For example, if the request's IP was found on the Spamhaus DROP list, it could be assigned a "spamhaus" tag. Some tags are generated automatically, while others are user-defined. (.)

• Next, Curiefense determines the security ruleset(s) that have been assigned to the request's target URL, and which match the tags. For example, there might be a ruleset defined for the "spamhaus" tag, or for the "devops" tag.

• Curiefense then enforces the ruleset(s), and takes the defined Action(s). For example, "block requests from Spamhaus-listed IPs", or "bypass devops requests from further filtering."

This process is based on the Documents as follows:

• is the Document which defines tags for external lists and custom lists.

• , , and define security rulesets, i.e. the actions to take when specific tags and/or other criteria are observed.

• assign the security rulesets to internal URLs.

Document Editor interface

This page is divided into three vertical sections. From top to bottom, they are:

• Administration

• Entry editing

• Versioning

Each is discussed below.

Entry Administration

The top section contains a toolbar with input controls.

• Configuration pulldown: Selects the branch/configuration for editing.

• Document pulldown: Selects the Document to display for editing.

• Entry pulldown: Selects the Entry (the ruleset) that is being displayed for editing.

• Duplicate button: Makes a copy of the currently displayed Entry.

• Download button: Downloads the currently displayed parameters.

• Add button: Creates a new Entry of the currently selected type.

• Save button: Saves all changes that were made since the last Save action.

• Delete button. Deletes the currently selected Entry.

Entry Editing

The UI in this section will vary, depending on the type of Document being edited. Each is discussed in more depth here:

At the bottom of this section, the URL of the current entry is shown in gray. It reflects the structure of the data.

For example, /conf/api/v1/configs/devops/d/urlmaps/e/__default__ shows that:

• The current Configuration (or branch) is devops.

• The current Document is urlmaps.

• The current Entry is __default__.

Document Versioning

Rate Limits are rules which define the number of requests with certain characteristics that are allowed within defined time frames. When a request is received that exceeds a Rate Limit, a specified action is performed.

The input controls at the top of this page are described here: . Specific editing of a Rate Limit is described below.

Rate Limit concepts

A Rate Limit defines the number of times that requests can match certain conditions within a certain time frame. Once that limit has been reached, subsequent requests matching those conditions within the same time frame will trigger an action.

By default, a Rate Limit will be enforced for all requests for the URL(s) to which it is assigned.

Meta Parameters

Rate Limit

Matching Conditions (the "Count by" input controls)

A condition consists of a field and a value. Within a Rate Limit, they play a role like this:

"More than <THRESHOLD> requests with the same <CONDITION-VALUE> <CONDITION-FIELD> sent to <ASSIGNED-LOCATION> within <TTL-PERIOD> will cause <ACTION>."

Condition Fields

A condition can be built upon any one of these four fields:

Multiple Conditions

Multiple conditions can be defined within the same Rate Limit. To create a new condition and open it for editing, select "New entry" below the list of conditions.

If multiple conditions are defined, they are evaluated by combining them together with a logical AND. In other words, the cumulative count toward the Threshold will be incremented whenever a request is seen that matches all of the conditions simultaneously. Different combinations of conditions will have separate Threshold counts maintained for them.

The "Event" option: Changing the meaning of the Rate Limit

Below the list of condition(s), there is another condition named "Event."

By default, this is set to "HTTP request,", which simply means to increment a counter each time a request is received that matches the conditions.

However, if the Event condition is changed to a different value, then the following applies.

Adding an Event Condition changes the evaluation process. An Event Condition is not logically combined with the preceding Count Condition; it is always evaluated separately.

More importantly, adding an Event Condition changes the meaning of the Rate Limit.

• If an Event Condition is not defined—in other words, if "HTTP request" is selected—then as discussed above, an internal counter is maintained for each Count Condition value, and incremented each time that value is encountered in a request.

• If an Event Condition is defined—in other words, if something other than "HTTP request" is selected—an internal counter is maintained for each Count Condition Value, and incremented each time a new, previously unobserved Event Condition value is encountered in a request.

Therefore, if an Event Condition is defined, the Rate Limit constrains the number of allowable Event Condition values for any given Count Condition value.

So, the evaluation becomes something like this:

"More than <THRESHOLD> <EVENT-CONDITION-VALUE> <EVENT-CONDITION-FIELD>s per any one <COUNT-CONDITION-VALUE><COUNT-CONDITION-FIELD> sent to <ASSIGNED-LOCATION> within <TTL-PERIOD> will cause <ACTION>."

Note that the number of Count Condition values is not being limited here. The limit is on the number of Event Condition values that each Count Condition value is allowed.

Action

When an incoming request exceeds the Threshold, the Action specified here will occur.

The Ban action

Most of the Actions listed above will not fully exclude an attacker that continues pressing the attack.

The Ban action can be used to block (or take some other Action in response to) a Rate Limit violator for an extended period of time.

Note that when setting up a Ban, the most common choices for its Action are to deny the violator's requests (via Default 503, Response, or Redirect). However, you can also choose Monitor (to observe the violator's actions during the ban period), Challenge (to verify that the violating activity is not being done by bots), or Header (to mark the requests for further scrutiny by the backend).

Limiting the scope of a Rate Limit

By default, an active Rate Limit rule will be enforced upon all incoming requests targeting URLs to which this rule has been assigned.

At the bottom of this page, you can add Include and/or Exclude parameters. These define the portion of the incoming traffic stream that will be evaluated for possible violation of the Rate Limit. In other words, they limit the scope of the Rate Limit's enforcement:

• The Include filter will limit enforcement to requests matching its parameters. All other requests in the traffic stream will not have this Rate Limit enforced upon them.

• The Exclude filter will exempt requests from enforcement that otherwise would have been subject to it.

(Internally, Curiefense evaluates Exclude parameters first, and then Include parameters.)

To add one or more filters, select "New entry", define the parameters, then select the "+" button. If more than one Include filter is specified, they are combined with a logical AND.

The instructions below show how to install Curiefense on a Kubernetes cluster, embedded in an Istio service mesh. It assumes that the instructions described in have been completed successfully.

The following tasks, each described below in sequence, should be performed:

At the bottom of this page is a describing the charts and configuration variables.

Setup Synchronization

An AWS S3 bucket must be available to synchronize configurations between the confserver and the Curiefense Istio sidecars. The following Curiefense variables must be set:

• In deploy/istio-helm/chart/values.yaml:

  • Setcurieconf_manifest_url to the bucket URL.

• In deploy/curiefense-helm/curiefense/values.yaml:

  • Set curieconf_manifest_url to the bucket URL.

  • Set curiefense_db_hostname to the hostname (or FQDN) of the database server if you choose to supply your own database. If you choose to use the instance from our curiefense chart, use the default logdb value.

Create a Kubernetes Cluster Running Helm

Access to a Kubernetes cluster running Helm v2 is required. Dynamic provisioning of persistent volumes must be supported. To set a StorageClass other than the default, change or override variable storage_class_name in deploy/curiefense-helm/curiefense/values.yaml.

Below are instructions for several ways to achieve this:

• Using minikube, Kubernetes 1.14.9 and Helm v2.13.1 (dynamic provisioning is enabled by default)

• Using Google GKE, Kubernetes 1.16.13 (with RBAC) and Helm v2.16.7 (dynamic provisioning is enabled by default)

• Using Amazon EKS, Kubernetes 1.18 (with RBAC) and Helm v2.16.7 (dynamic provisioning is enabled by default)

Option 1: Using minikube

This section describes the install for a single-node test setup (which is generally not useful for production).

Install minikube

Starting from a fresh ubuntu 20.04 VM:

• Allow your user to interact with docker: sudo usermod -aG docker $USER && newgrp docker

Start a screen or tmux, and keep the following command running:

Install Helm v2.13.1

Run the following commands:

Now install Helm to the Kubernetes cluster:

Option 2: Using Google GKE

This option uses a more recent Kubernetes, with RBAC enabled.

Install kubectl

Create a cluster

Install Helm v2.16.7

Now we must define RBAC authorizations. Helm needs to be able to deploy applications to both the curiefense and istio-system namespaces.

To do that, we provide an example configuration, which installs Tiller in the kube-system namespaces, and grants it cluster-admin permissions.

Finally, install Helm to the Kubernetes cluster:

Option 3: Using Amazon EKS

This option uses a more recent Kubernetes, with RBAC enabled.

Install kubectl

Create a cluster

Install Helm v2.16.7

Follow all the "Install Helm v2.16.7" instructions shown above in the Google GKE section.

Reset State

If you have a clean machine where Curiefense has never been installed, skip this step and go to the next.

Otherwise, run these commands:

Ensure that helm ls -a outputs nothing.

Create Namespaces

Run the following commands:

Setup Secrets

AWS credentials

Encode the AWS S3 credentials that have r/w access to the S3 bucket. This yields a base64 string:

Create a local file called s3cfg.yaml, with the contents below, replacing both occurrences of BASE64_S3CFG with the previously obtained base64 string:

Deploy this secret to the cluster:

Database configuration

Curiefense requires two database accounts. They will be automatically provisioned by the logdb container:

• One with read/write authorization (described below as BASE64_READWRITE_USERNAME andBASE64_READWRITE_PASSWORD). If the logdb container is used (which is the default), BASE64_READWRITE_USERNAME must be set to postgres.

• One with read-only permissions (described below as

  BASE64_READONLY_USERNAME and BASE64_READONLY_PASSWORD). If the logdb container is used (which is the default), BASE64_READONLY_USERNAME must be set to logserver_ro.

Create a local file called dbsecret.yaml, with the contents below, containing database credentials. Replace BASE64_READWRITE_USERNAME, BASE64_READONLY_USERNAME, BASE64_READWRITE_PASSWORD and BASE64_READONLY_PASSWORD with the correct base64-encoded values.

Deploy this secret to the cluster:

An example file with weak default credentials is provided at deploy/curiefense-helm/example-dbsecret.yaml.

Setup TLS

Using TLS is optional.

The UIServer can be made to be reachable over HTTPS. To do that, two secrets have to be created to hold the TLS certificate and TLS key.

Create a local file called uiserver-tls.yaml, replacing TLS_CERT_BASE64 with the base64-encoded PEM X509 TLS certificate, and TLS_KEY_BASE64 with the base64-encoded TLS key.

Deploy this secret to the cluster:

An example file with self-signed certificates is provided at deploy/curiefense-helm/example-uiserver-tls.yaml.

Deploy Istio and Curiefense Images

Deploy the Istio service mesh:

And then the Curiefense components:

Deploy the (Sample) App

The application to be protected by Curiefense should now be deployed. These instructions are for the sample application bookinfo.

Create the namespace

Create the Kubernetes namespace, and add the istio-injection=enabled label that will make Istio automatically inject necessary sidecars to applications that are deployed in this namespace.

Install the application

Test bookinfo

Check that bookinfo Pods are running (wait a bit if they are not):

Sample output example:

Check that the application is working by querying its API directly without going through the Istio service mesh:

Expected output:

Test access to bookinfo through Istio

(Replace "ip" with "hostname" if running in an environment where the LoadBalancer yields a FQDN, as is the case with Amazon's ELB.)

Expected output:

If this error occurs: Could not resolve host: a6fdxxxxxxxxxxxxxxxxxxxxxxxxxxxx-xxxxxxxx.us-west-2.elb.amazonaws.com ...the ELB is not ready yet. Wait and retry until it becomes available (typically a few minutes).

Check that logs reach the accesslog UI

Run this query (add random characters at the end)

(Replace "ip" with "hostname" if running in an environment where the LoadBalancer yields a FQDN, as is the case with Amazon's ELB.)

Run this to ensure that the logs have been recorded and are reachable from the UI server:

Check that TEST_STRING appears in the output.

Expose Curiefense Services using NodePorts

Run the following commands to expose Curiefense services through NodePorts. Warning: if the machine has a public IP, the services will be exposed on the Internet.

Start with this command:

The following command can be used to determine the IP address of your cluster nodes on which services will be exposed:

For minikube only:

If you are using minikube, also run the following commands on the host in order to expose services on the Internet:

For Amazon EKS only:

If you are using Amazon EKS, you will also need to allow inbound connections for port range 30000-30500 from your IP. Go to the EC2 page in the AWS console, select the EC2 instance for the cluster (named curiefense-eks-...-Node), select the "Security" pane, select the security group (named eks-cluster-sg-curiefense-eks-[0-9]+), then add the incoming rule.

Access Curiefense Services

The UIServer is now available on port 30080 over HTTP, and on port 30443 over HTTPS.

Grafana is now available on port 30300 over HTTP.

For the bookinfo sample app, the Book Review product page is now available on port 30081 over HTTP, and on port 30444 over HTTPS.

The confserver is now available on port 30000 over HTTP.

Reference: Description of Helm Charts

Curiefense charts

Helm charts are divided as follows:

• curiefense-admin - confserver, curielogserver and UIServer.

• curiefense-dashboards - Grafana and Prometheus.

• curiefense-log - logdb, namely: PostgreSQL.

• curiefense-proxy - curielogger, curiesync and redis.

Chart configuration variables

Configuration variables in deploy/curiefense-helm/curiefense/values.yaml can be modified or overridden to fit your deployment needs:

• Variables in the images section define the Docker image names for each component. Override this if you want to host images on your own private registry.

• storage_class_name is the StorageClass that is used for dynamic provisioning of Persistent Volumes. It defaults to null (default storage class, which works by default on EKS, GKE and minikube).

• ..._storage_size variables define the size of persistent volumes. The defaults are fine for a test or small-scale deployment.

• curieconf_manifest_url is the URL of the AWS S3 bucket that is used to synchronize configurations between the confserver and the Curiefense Istio sidecars.

• curiefense_db_hostname defines the hostname of the postgres server that will be used to store logs. Defaults to the provided logdb StatefulSet. Override this to replace the postgres instance with one you supply, or an AWS Aurora instance.

• docker_tag defines the image tag versions that should be used. deploy.sh will override this to deploy a version that matches the current working directory, unless the DOCKER_TAG environment variable is set.

Istio chart

Components added or modified by Curiefense are defined in deploy/istio-helm/chart/charts/gateways. Compared to the upstream Istio Kubernetes distribution, we add or change the following Pods:

• An initContainer called curiesync-initialpull has been added. It synchronizes configuration before running Envoy.

• A container called curiesync has been added. It periodically fetches the configuration that should be applied from an S3 bucket (configurable with the curieconf_manifest_url variable), and makes it available to Envoy. This configuration is used by the LUA code that inspects traffic.

• The container called istio-proxy now uses our custom Docker image, embedding our HTTP Filter, written in Lua.

• An EnvoyFilter has been added. It forwards access logs to curielogger (see curiefense_access_logs_filter.yaml).

• An EnvoyFilter has been added. It runs Curiefense's Lua code to inspect incoming traffic on the Ingress Gateways (see curiefense_lua_filter.yaml).

Chart configuration variables

Configuration variables in deploy/istio-helm/chart/values.yaml can be modified or overridden to fit your deployment needs:

• gw_image defines the name of the image that contains our filtering code and modified Envoy binary.

• curiesync_image defines the name of the image that contains scripts that synchronize local Envoy configuration with the AWS S3 bucket defined in curieconf_manifest_url.

• curieconf_manifest_url is the URL of the AWS S3 bucket that is used to synchronize configurations between the confserver and the Curiefense Istio sidecars.

• curiefense_namespace should contain the name of the namespace where Curiefense components defined in deploy/curiefense-helm/ are running.

• redis_host defines the hostname of the redis server that will be used by curieproxy. Defaults to the provided redis StatefulSet. Override this to replace the redis instance with one you supply.

• initial_curieconf_pull defines whether a configuration should be pulled from the AWS S3 bucket before running Envoy (true), or if traffic should be allowed to flow with a default configuration until the next synchronization (typically every 10s).

This page describes the initial tasks that must be performed when deploying Curiefense:

• .

• .

After completing the tasks below, proceed to the tasks for the specific method being used:

During this process, you might find it helpful to read the descriptions (which include the purpose, secrets, and network/port details) of the services and their containers:

Clone the Repository

Clone the repository, if you have not already done so:

This documentation assumes it has been cloned to ~/curiefense.

Select and Prepare for TLS

Curiefense can use TLS, but this is optional. (If you do not choose to set it up, HTTPS will be disabled.)

At this point in the setup process, you should decide whether or not to do so:

• An Istio Helm deployment can use TLS for communication with Curiefense's UI server.

• A Docker Compose deployment can use TLS for communication with Curiefense's UI server and also for the protected service.

If you do not want Curiefense to use TLS, then skip this step and proceed to the next section.

If you want Curiefense to use TLS, generate the certificate(s) and key(s) now. You will add them to Curiefense later.

Method-Specific Tasks

After performing the previous tasks, perform the tasks for the specific method being used:

Curiefense is an API-first, DevOps oriented web-defense HTTP-Filter adapter for . It provides multiple security technologies (WAF, application-layer DDoS protection, bot management, and more) along with real-time traffic monitoring and transparency.

Curiefense is . All configuration data (security rulesets, policies, etc.) can be maintained singularly, or as different branches for different environments, as you choose. All changes are versioned, and reverts can be done at any time.

Curiefense also has a UI console, discussed in the section.

Version

This documentation is for version 1.0.0

(To view docs for a different version, choose it at the top of the left sidebar.)

Architecture and Components

Curiefense provides traffic filtering that can be configured differently for multiple environments (e.g. dev/qa/prod), all of which can be administered from one central cluster if desired. Here is an overview of its components.

In the diagram above, the Server represents a resource protected by Curiefense (a site, app, service, or API). The User is a traffic source attempting to access that resource.

Incoming traffic passes through Envoy, which is using Curiefense as an HTTP filter. Hostile requests are blocked.

The other components in the diagram represent the Curiefense platform, as follows:

• Curiefense proxy (represented by the Curiefense logo): Plugs into Envoy and performs traffic filtering.

• Logs DB. Curiefense stores traffic data (headers, payloads, etc.) from all requests here.

• Metrics. A Prometheus store of traffic metrics.

• Dashboard. Grafana dashboard(s) with visual displays of traffic metrics.

• Web UI. Curiefense's web console for configuring the platform.

• Config Server: A service which:

  • Receives configuration edits from the Web UI

  • Receives configuration edits from API calls (not shown in the diagram)

  • Creates a new configuration version in response to edits

  • Stores the new version in one or more Cloud Storage buckets

• Cloud Storage: Stores versioned configurations. Each Curiefense proxy periodically checks Cloud Storage: when a new version is found there, the proxy downloads it and updates its security posture.

Deployment Options

Curiefense can run in variety of environments, depending on your specific needs. It can be adapted to many different use cases.

If you create an installation workflow for a situation that is not currently described in this manual, please feel free to submit it for inclusion.

Three Primary Roles

Conceptually, there are three primary roles performed by Curiefense:

• Configuration (allowing admins to define security policies, assign them to URLs, etc.)

• Filtering (applying the defined Configurations to incoming traffic and blocking hostile requests)

• Monitoring (displaying traffic data in real-time and in historical logs).

Configuration

Data Structures

Curiefense maintains its security parameters as Entries, which are contained in Documents, which are contained in Configurations.

A Configuration is a complete definition of Curiefense's behavior for a specific environment. An organization can maintain multiple Configurations (e.g., development, staging, and production).

A Configuration also includes data blobs, which currently are used to store the Maxmind geolocation database. This is where Curiefense obtains its geolocation data and ASN for each request it processes.

Configuration Administration

A Configuration is the atomic unit for all of Curiefense's parameters. Any edits to a Configuration result in a new Configuration being committed. Configurations are versioned, and can be reverted at any time.

When a Configuration is created or modified (whether by the UI console or an API call), the admin pushes it to a Cloud Storage bucket. An important feature of Curiefense is simultaneous publishing to multiple environments.

Filtering

Traffic filtering is performed by the Curiefense proxy, as shown in the first diagram above. In other words, this is where the security policies defined in the Configurations are enforced.

Some activities (such as rate limiting) require local data storage. Internally, Curiefense uses Redis for this. Other storage methods can be used instead if desired.

Monitoring

Each time a request goes through Curiefense, a detailed log message is pushed to the Logs DB. (Out of the box, Curiefense is setup to work with PostgreSQL, but any RDBMS can be used.)

Traffic data is available in several ways:

An ACL (Access Control List) Profile is a set of sequential criteria by which a request is evaluated. The input controls at the top of this page are described here: . Specific editing of an ACL Profile is described below.

In the interface, the tags are listed below their associated actions. Evaluation is performed from left to right:

• The incoming request is evaluated to see if it matches any of the tags in the far left column.

• If a match is found, that column's action is performed, and no further evaluation is done.

• If a match is not found, then the next column to the right is evaluated. A match will trigger that column's action; otherwise the next column is evaluated.

• This process continues until an action is taken or all columns have been evaluated.

• If a request does not match any tags in this Profile, it is passed to the WAF for further inspection.

Action columns

Detecting Bots, Verifying Humans

Much of the Curiefense interface focuses on configuration for:

• The detection and mitigation of hostile requests.

• The detection and mitigation of hostile behavior of requestors.

The challenge process is different. It is built into Curiefense, and provides a third approach to security. It mitigates threats based on the requestor's identity, environment, and behavior.

When Curiefense receives the first request from a previously unknown traffic source (below described as the "user") for a browser-based web application, this is the typical process that is followed.

1. Curiefense challenges the user's browsing environment. Curiefense uses a variety of proprietary, multi-faceted techniques to verify that this requestor is a human using a browser, instead of a bot using a headless browser or emulator.

   • The user’s browser is subjected to several dozen tests, verifying the features known to be supported by that browser. This includes hidden canvases, video and audio in various formats, WebRTC and other advanced networking protocols, screen resolution, and more.

   • The browser is subjected to an invisible “attack”: subtle errors are injected into the environment, and the browser engine’s reactions are captured and analyzed. Curiefense verifies that the exceptions and error messages are those which should be generated, if the browser is what it claims to be. (It is very difficult for threat actors to spoof this behavior using headless browsers and emulators, since there is an infinite number of possible errors to which any browser can be subjected, and threat actors need to replicate the actual reactions for each possible input.)

   • The above process only takes milliseconds, and it is completely transparent to the end user.

   • Once a browser has passed these tests, Curiefense signs it cryptographically. This signature accompanies all subsequent activity.

2. If the challenge is not passed, the request is suspected to be a bot, and another challenge is issued. This process continues until a challenge is passed, or a threshold is reached (e.g., via a Rate Limit) to ban the requestor.

3. If the challenge is passed, the browser's session is authenticated, and the browser receives cookies from Curiefense.

4. The browser then automatically resubmits the original request, but this time, the cookies are included. The user is granted access to the requested URL, resources, etc.

5. Subsequent requests will also include the cookies, and thus, they are not challenged.

This process happens quickly (in a few milliseconds), and is invisible to the user.

Active Challenges versus Passive Challenges

The process described above is the active challenge process. Out of the box, this is the challenge process that Curiefense uses.

Passive challenges add three additional benefits:

• They enable biometric behavioral profiling: a much more powerful means of identifying automated traffic. More on this below.

• In some situations, active challenges can interfere with certain metrics such as those provided by Google Analytics. (The initial referrer information is lost.) If this is a problem, active challenges can be disabled. In this situation, passive challenges can provide effective bot protection instead.

• When caching is being done by a CDN, active challenges will not occur for pages being served from the cache. Passive challenges are necessary for Curiefense to perform bot detection in this situation.

Biometric Behavioral Profiling

With Biometric Bot Detection, Curiefense continually gathers and analyzes stats such as client-side I/O events, triggered by the user’s keyboard, mouse, scroll, touch, zoom, device orientation, movements, and more. Based on these metrics, the platform uses Machine Learning to construct and maintain behavioral profiles of legitimate human visitors. Curiefense learns and understands how actual humans interact with the applications and services it is protecting. Continuous multivariate analysis verifies that each user is indeed conforming to expected behavioral patterns, and is thus a human user with legitimate intentions.

Using this approach, Curiefense bot detection accuracy is not only high, it is also robust and resistant to reverse-engineering by threat actors. Behavioral profiles are constructed based on private analytics data, and threat actors have no realistic way of obtaining this information.

Implementation

Implementing Passive Challenges is simple. Place this Javascript code within the pages of your web applications:

If desired, the script code can include async and/or defer attributes:

These usually are not necessary, and their effect will depend on the placement of the script within the page. Their use is left to your discretion.

Testing

To test the implementation, use a browser to visit a page containing the Javascript snippet. Once it runs, the browser should have a cookie named rbzid.

Disabling Active Challenges (only if necessary)

There are two primary situations where users sometimes want to disable Active Challenges:

• When a user needs site analytics to correctly reflect all referrers. (Active Challenges can interfere with this.)

• For API endpoints. Active Challenges are designed to verify the client's browser environment; for most API calls, there is no browser environment to verify.

Other than those situations, Active Challenges can be very beneficial.

This page displays the current System Settings in a JSON editor. These settings are used to populate the interface for Publish Configuration.

The settings are:

• buckets: the complete list of buckets to which a Configuration can be pushed.

• branch_buckets: the default list of buckets for each branch. When a Configuration is published, the default buckets for the branch will be initially selected in the Publish Configuration interface (but they can be changed before publishing).

The input controls at the top of this page are described here: Document Editor interface. Specific editing of a Profiling List is described below.

Overview

Early in the traffic evaluation process (immediately after the pre-processing Cloud Functions are performed), Curiefense assigns Tags to an incoming request. Subsequently, the Tags can be used to make decisions about how the request is processed. After processing, a request's Tags remain associated with it, and they are available for display in the Access Log.

This page allows you to administer Profiling Lists, which are combinations of user-defined Tags and the criteria for assigning them to requests.

Each Profiling List consists of:

• Match conditions: A list of possible characteristics that a request can match (e.g., a list of IP addresses that it might originate from), plus the logical operator to use when evaluating the match.

• One or more Tags to assign when a match occurs.

For each incoming request, Curiefense will evaluate all active Profiling Lists. A single request will receive Tags from all Lists which match it.

Two Types of Profiling Lists

Profiling Lists can be either Internet-sourced or self-managed.

• Internet-sourced Lists are based upon online sources (e.g., Spamhaus DROP lists). The entries for these lists are not editable within the interface, because they are obtained automatically; Curiefense updates them every 24 hours.

• Self-managed Lists are created manually. These are fully editable within the interface.

Profiling List Metadata

Specifying Match Conditions

Match conditions consist of two parts:

• A list of one or more criteria to match. The list will be entered differently for the two types of Profiling Lists.

• The logical condition to apply (either OR or AND), specified in the Entries Relation pulldown

Match Criteria for an Internet-sourced Profile

For a Profile based on an online source, simply enter its URL into the Source field. For example, to create a list based on the Spamhaus ASN DROP list, you would enter https://www.spamhaus.org/drop/asndrop.txt. Curiefense will then populate the list automatically.

If the list contains more than category--which is unlikely for an Internet-sourced Profile, but not impossible--also choose the appropriate value for Entries Relation, as discussed below.

At this point, the new Profiling List should be complete. Before exiting the page, be sure to save your work.

Match Criteria for a Self-Managed Profile

To add a match criterion, select the "+" button at the top of the criteria list. The following dialog will appear.

For most of the categories, the dialog will appear as it is above. Multiple entries can be made at once, with each entry on a separate line. Each line contains the value, plus a pound sign (#) followed by an annotation (a label for display within the Curiefense interface). Example:

For some categories, one entry can be made at a time, with each entry requiring multiple lines. Annotations are not preceded by a pound sign.

Here are some sample entries for the various categories. Notice that the logical operators are available.

Once created, these entries cannot be edited. If one needs to be modified, remove it and re-create it.

The Entries Relation Field

This specifies the logical operation used when evaluating a request against the match criteria.

When all the match criteria are in the same category, the operation is OR. For a request x and list of criteria a, b, c , then the evaluation will be (x==a) OR (x==b) OR (x==c) For example, if the match criteria are all IP addresses, then a match will occur if the request matches any of the IPs in the list.

When there are two or more categories, the default operation is still OR (in which case, a match will occur if the request matches any of the criteria in the list). However, when there are multiple categories, you can select AND instead. In this case, OR will still apply within each category, while AND will apply between the categories. Thus, for one list a, b, c and a second list i, j, k , the evaluation will be ((x==a) OR (x==b) OR (x==c)) AND ((x==i) OR (x==j) OR (x==k)). For example, if the match criteria are IP addresses and headers, then a match will occur if the request matches any of the IPs and any of the headers, but it will not occur if it matches an IP but none of the headers.

This section displays all the WAF Signatures defined within Curiefense. This has various uses; for example, looking up the underlying signature for a tag that appears in the Access Log, or getting the ID of a signature to exclude in a WAF Profile.

Curiefense comes with many WAF Signatures. You can view them by choosing different IDs in the pulldown on the upper right.

Note that this list is for reference only. The entries cannot be edited.

The input controls at the top of this page are described here: Document Editor interface. Specific editing of a URL Map is described below.

Overview

This page specifies a list of URLs and the security policies assigned to them.

Every incoming HTTP/S request targets a specific URL. Curiefense finds the best match for that URL in the URL Maps, and applies the security policies defined for it.

Components of a URL Map

A URL Map consists of:

• Host definition: The (sub)domain(s) within which the Path Maps will be found.

• Path Maps: one or more paths, and the security policies which will be applied to them.

Creating a URL Map

To add a new URL Map, use the buttons at the top of the window to duplicate an existing one or create a new one. Then fill in these fields.

Editing its Path Maps

To add a new Path Map, select an existing one in the middle of the window, expand it, and select "Fork Profile" at the bottom. The existing one will be cloned, and the new one will be displayed for editing.

To edit an existing Path Map definition, click on the "expand" button at the end of its listing. The edit window shown below will appear.

In addition to editing the fields discussed above, this window also provides the ability to:

• Deactivate the WAF Profile (by unselecting its Active Mode checkbox).

• Deactivate the ACL Profile (by unselecting its Active Mode checkbox).

• Assign (or remove) Rate Limit rules to this Path Map, via the + button (or remove button). The + button will only be shown if there are unassigned Rate Limit rules available.

• Create a copy of this Path Map, and open it for editing, via the Fork Profile button.

As the name suggests, this page allows you to publish a new or revised Configuration to the cloud, or revert to a previous version.

To do so, follow these steps:

1. In the pulldown on the top left, select the Configuration.

2. Below it is the Version History: a list of Configurations, with the most recently edited version at the top. If you wish to publish your most recent edits, ensure that the top entry is selected. If instead you wish to revert to a previous entry, find it in the list (using the View More option if necessary) and select it.

3. On the right is a list of Target Buckets. The default buckets for this branch (which are defined in System Settings) should be selected already. If different buckets are needed, select them instead.

4. Select the Publish Configuration button on the top right.

The input controls at the top of this page are described here: Document Editor interface. Specific editing of a WAF Profile is described below.

Overview

A WAF Profile is a set of security policies that are used by the Curiefense WAF (Web Application Firewall). Every deployment includes a default WAF Profile, and additional Profiles can be created.

Every URL that Curiefense protects has a WAF Profile assigned to it in URL Maps. (If none is assigned explicitly, the default is used.) A request sent to a URL might, or might not, be filtered according to the assigned Profile.

Reasons why WAF filtering might not occur:

• The request was blocked before WAF filtering would have occurred. (Before the WAF is used, several other stages of filtering occur first.)

• The applicable ACL Profile resulted in an Action of Bypass, which exempts the request from WAF filtering.

• The WAF is not in Active Mode for this URL.

Input Characteristics

At the top of the page, the following values are defined for incoming requests.

Content Filtering and Whitelisting

By default, an incoming request will be compared to all the WAF Signatures. If any parameter (any header, cookie, or argument within it) fails this evaluation, the request will be blocked.

However, parameters can be whitelisted and exempted from this filtering. For each parameter, this can be done in two ways:

• Full exemption is available by specifying a regex pattern which, if it matches the parameter's value, will exempt that parameter from WAF Signature evaluation.

• Partial exemption is available by specifying a list of WAF Signatures which will not be evaluated, even if the regex pattern is not matched.

Along with this content whitelisting, a "positive security" form of content filtering is also available. Curiefense can be configured to require certain content in a specified parameter, and reject requests that do not contain it.

Parameter Content Constraints

The bottom part of the UI defines Curiefense's behavior for both whitelisting and content filtering for each parameter.

In the following discussion, a constraint refers to the values in the UI input controls (Parameter, Matching Value, Restrict?, and Exclude Sig) that have been specified for one parameter.

Each incoming request is processed like this:

This behavior is defined in the following fields in the UI.

Constraints are defined in the same way for Headers, Cookies, and Arguments within their respective tabs.

As discussed in , Curiefense's data is stored within:

• Configurations

• Documents

• Entries

• Blobs

A Configuration is a set of Blobs and Documents. A Document is a set of Entries.

All of these data structures can be edited via API:

• A Document is a file treated as a JSON list of entries.

• An entry is a JSON dictionary with an id field. The id field value must be unique inside the document, and must be a valid part of an URL.

• A Blob is a file treated as binary data.

Versioning

Each time a Configuration is modified and , a new version is created. A Configuration can be .

Operations

The following types of operations are accessed as follows.

API endpoint namespaces ()

• configs (for manipulating Configurations)

• tools (for publishing, etc.)

• db (for accessing persistent data storage)

• blob (for manipulating blobs)

• conf (for manipulating Configurations)

• db (for manipulating persistent memory database)

• doc (for manipulating Documents)

• entry (for manipulating Entries)

• key (for manipulating keys inside the database)

• sync (to maintain synchronization, e.g. the configuration API server and cloud storage)

The Access Log displays requests that were received in the specified time period.

Filtering the Display

By default, this page shows all requests in the time period. To filter the display, type a search string into the filter box. For example, to see all the 503 errors, use "503" as the filter string.

Multiple filters can be entered, delimited by commas. For example, "waf, 503" will display all the requests that were blocked by the WAF with a 503 error.

It's helpful to spend a few minutes experimenting with the filter box. You can quickly drill down through large swaths of traffic, discovering events and patterns that can reveal many insights about your traffic. This is helpful when constructing and fine-tuning security policies, especially during attacks.

Viewing Request Details

The primary display shows a summary of each request. To view more information about a request, click on its listing, or on "expand" at the end of its listing. The display will expand to show its full details.

If a request was filtered, the "Risk Details" box highlights the reason(s) why it was viewed as a risk.

Out of the box, Curiefense stores metrics using , and provides dashboards and alerts via .

In a standard installation, the Grafana dashboards are accessible at .

Curiefense includes an API with the following namespaces:

• configs (for manipulating Configurations)

• tools (for publishing, etc.)

• db (for accessing persistent key value storage)

These can be explored and seen visually by using via the Curiefense SwaggerHub:

Each namespace contains various endpoints. For example, tools contains five:

Swagger is a useful way to experiment and interact with the API. For example, clicking on "/configs", selecting the "Try it out" button, and then selecting the "Execute" button will result in this:

Among other things, the cURL command line will be displayed, which can be copied and pasted for use elsewhere. It is also helpful sometimes to see the structure of the data.

For some endpoints, arguments need to be supplied. Swagger will provide input controls for them, as in this example:

This page displays previous versions of configurations, and allows you to revert/restore the system to any saved configuration.

To do this, hover the mouse over the desired configuration. The "Restore Version" button will appear, as shown above.

Select this button. Once the process is complete, . Curiefense will push the selected configuration out to all deployments.

Curiefense includes a command line tool: curieconf_cli.

In the discussion below, some examples will be given using the command line via this tool. Others will be shown for both curieconf_cli and curl.

Operations on Configurations

List all existing configurations:

Retrieve the full configuration named master:

Retrieve all the different versions of the configuration named master:

Create a new configuration from a dump file:

Revert configuration master to version 3a370d4:

Update a configuration from a batch update:

Get the detailed list of existing configurations:

Retrieve a complete configuration:

Create a new configuration, name is in the posted data:

Create a new configuration, name is provided and overrides posted data:

Update an existing configuration:

Delete a configuration:

Clone a configuration, new name is in POST data:

Clone a configuration, new name is in the URL:

Get all versions of a given configuration:

Retrieve a specific version of a configuration:

Create a new version for a configuration from an old version:

Format for full config dumps and batch updates:

Operations on Documents

Retrieve the list of existing documents in this configuration:

Get a complete document:

Get a given version of a document:

Retrieve the existing versions of a given document:

Create a new complete document:

Update an existing document:

Delete/empty a document:

Create a new version for a document from an old version:

Operations on Entries

Retrieve the list of entries in a document:

Retrieve an entry from a document:

Get the list of existing versions of a given entry in a document:

Get a given version of a document entry:

Create an entry in a document:

Update an entry in a document:

Delete an entry from a document:

Operations on Blobs

Retrieve the list of available blobs:

Retrieve a blob:

Create a new blob:

Replace a blob with new data:

Delete a blob:

Retrieve the list of versions of a given blob:

Retrieve the given version of a blob:

Create a new version for a blob from an old version:

Curiefense evaluates incoming traffic in a multi-stage filtering process. An HTTP/S request which passes all security tests will be forwarded to the backend.

When constructing its security posture, it can be helpful to understand the filtering stages. They are:

• Profiling: Curiefense assigns and to the requests.

• enforcement.

• enforcement.

• and enforcement.

• Transmission of request to the backend.

When an incoming request is received, Curiefense generates internal tags and assigns them to it.

Some tags are assigned early, and are used to make decisions about how the request is handled. For example, if a request's IP is found on the Spamhaus DROP list, it might be assigned a tag of "spamhaus". Then an might block the request because it contains that tag.

Some tags can be generated during processing; they reflect the decisions that were made. For example, a request that was blocked because it violated a will be assigned a tag containing that Rate Limit's name. The tag will be shown in the .

Some tags are defined by the user, while others are generated automatically by Curiefense.

User-Defined Tags

are user-defined lists of criteria with one or more associated tags. Requests which match the criteria will be assigned those tags.

A Profiling List can be based on an external list (e.g., the Spamhaus DROP list), or a user-defined custom list (e.g., a list of IP addresses used by the internal QA team).

Automatic Tags

Many tags are generated automatically by Curiefense. Examples:

• Every request receives a tag of "all".

• Every request receives several tags according to its source (the IP address, geolocation, etc.)

• Every request receives two tags for the that it matched (both at the domain level and at the Path Map level).

• Requests which violate a security policy or have other problems, receive tags with descriptive names (e.g., the name of the policy that was violated).

When tag names are generated from underlying values (IP addresses, security rule names, etc.), hyphens will replace spaces and special characters.

Examples of automatic tags

Sometimes a request will get two separate tags that seem to be redundant. For example:

• urlmap-default-entry

• urlmap-entry-default

When a URL Map is matched with a request, a tag is generated for the URL Map itself, and for the Path Map that was used. If the names are similar (which is true for default values, as in the example), then the tags can appear to be redundant.

Containers are built from recipes contained incuriefense/images/. Descriptions of the various images are below.

Image Tagging

Several tag formats are used.

Images can also have two-part tags to identify what is in the image. The parts are:

• the output of git describe --tag --long --dirty which contains the latest git tag, number of commits since last tag, and abbreviated current commit hash

• the shortened hash of the git tree HEAD:curiefense/ where Docker images are stored. It helps to see quickly whether image sources from two different commits are identical.

Common Images

confserver

• A REST API server to read, write, and maintain versioning of Curiefense's configuration.

• Flask is the web interface. Git is the storage engine for historical and versioned configuration. Nginx serves as the frontend for the Flask web application.

• Secrets:

  • If an S3 bucket is used as a synchronization mechanism between confserver and curieproxy instances (this is the default with Helm deployments), then this container requires credentials to access it.

  • In Docker Compose environments, S3 credentials are defined in deploy/compose/curiesecrets/s3cfg before deployment, then mounted in /var/run/secrets/s3cfg.

  • In Helm environments, S3 credentials are . They are expected to be in a Secret object called s3cfg, in the same namespace as confserver, which is mounted to make credentials available in /var/run/secrets/s3cfg/s3cfg.

• Network details:

  • Port 80 is the configuration API. A Swagger interface is available at endpoint /api/v1/ (reachable at in the sample Docker Compose and Helm deployments).

curielogger

• Receives access logs from Envoy (curieproxyimages) over gRPC, and does the following:

  • Pushes logs into the PostgreSQL server (logdb)

  • Aggregates metrics

  • Serves metrics over HTTP port 2112 for the Prometheus scraper.

• Secrets:

  • This container uses a postgres account (postgres) with read-write permissions. Its password is passed either in the CURIELOGGER_DBPASSWORD environment variable, or in a file whose path is contained in the CURIELOGGER_DBPASSWORD_FILE environment variable.

• Network details:

  • Port 9001 receives logs from Envoy over GRPC

  • Port 2112 exposes Prometheus metrics over HTTP

curielogserver

• A REST API server used by uiserver to read log records from logdb.

• Flask is the web interface for the REST API. Nginx serves as the frontend for the Flask web application.

• Secrets:

  • This container uses a postgres account (logserver_ro) with read-only permissions. Its password is passed in a file whose path is contained in the CURIELOGSERVER_DBPASSWORD_FILE environment variable

• Network details:

  • Port 80 exposes an API to retrieve logs

curiesync

• Periodically (every 10 seconds) polls the bucket located at CURIE_BUCKET_LINK, and extracts the active configuration to /config, which is shared with curiefense.

• Secrets: the same as described in confserver, above.

• Network details:

  • No exposed network service

grafana

• Grafana provides visualization for the metrics stored in Prometheus, and sends alerts based on anomaly thresholds.

• Its datasource is prometheus. Its basic dashboards are in the dashboards directory.

• Secrets

  • Default user: admin

  • Default password: admin

  • These can be changed upon the first connection.

• Network details:

logdb

• PostgreSQL server that stores access logs.

• Two accounts are used:

  • postgres has write access, is used by curielogger.

  • logserver_ro has read-only access, is used by curielogserver.

• Secrets:

  • The password for the logserver_ro account is passed in a file whose path is contained in the POSTGRES_READONLY_PASSWORD_FILE environment variable.

  • The password for the postgres account is passed in a file whose path is contained in the POSTGRES_PASSWORD_FILE environment variable.

• Network details:

  • Port 5432 for PostgreSQL access

prometheus

• Prometheus scrapes metrics from Envoy, curielogger, and Prometheus itself. These are defined in curiefense/images/prometheus/prometheus.yml.

• curielogger metrics are exposed as follows:

  • curiemetric_http_request_total: total number of requests.

  • curiemetric_request_bytes: total inbound (request) bytes.

  • curiemetric_response_bytes: total outbound (response) bytes.

  • curiemetric_session_details_total: Static labels are a fixed set of labels created for each request, such as "method", "path", "geo", etc.

  • curiemetric_session_tags_total: this metric stores the "dynamic" labels: tags and labels created dynamically on a request basis, and per context. For example, while the "blocked" label is set to 0 or 1 for each request, an actual blocked request may carry additional tags such as the block reasons and origin.

• Network details:

  • Port 9090 allows access to the Prometheus user interface over HTTP

redis

• Redis is accessed by curieproxy, and is used to synchronize Curiefense's advanced rate limiting and session control mechanisms.

• Network details:

  • Port 6379 receives Redis client queries

UIServer

• Serves the user interface. A Vue js app developed as single page app with NodeJS and serves the management console UI.

• The UI displays access logs to the user, and displays Curiefense's configuration for editing. API calls for configuration and access logs are routed to confserver and curielogserver by the Nginx inside the container. Nginx also used to serve the static parts of the UI such as HTML, CSS and JS.

• Secrets: This image will enable TLS on the nginx server if a TLS certificate and key are provided:

  • For Kubernetes (e.g. Helm) deployments, the certificate is expected at /run/secrets/uisslcrt/uisslcrt and the key at /run/secrets/uisslkey/uisslkey

  • For Docker Compose deployments, the certificate is expected at /run/secrets/uisslcrt and the key at /run/secrets/uisslkey

• Network details:

Additional Images for Docker-Compose Deployments

curieproxy-envoy

• Acts as a reverse proxy to TARGET_ADDRESS:TARGET_PORT.

• Filters traffic according to the active configuration.

• Sends access logs over GRPC tocurielogserver.

• Uses a custom-built Envoy binary, compiled with symbols needed by Lua. The custom Envoy compilation is described in curiefense/curieproxy/README.md.

• Network details:

  • Port 8001 is the Envoy administration interface

echo

• Simple http server that displays Echo.

• Network details:

  • Listens on port 5678

Additional Images for Istio-Helm Deployments

curieproxy-istio

• Acts as a reverse proxy to TARGET_ADDRESS:TARGET_PORT.

• Filters traffic according to the active configuration.

• Sends access logs over GRPC tocurielogserver.

• Uses a custom-built Envoy binary, compiled with symbols needed by Lua. The custom Envoy compilation is described in curiefense/curieproxy/README.md.

• In Helm deployments, two EnvoyFilters are defined in curiefense/deploy/istio-helm/chart/charts/gateways/templates/:

  • curiefense_lua_filter.yaml orders Envoy to apply the Lua HTTP filter to incoming requests.

  • curiefense_access_logs_filter.yaml orders Envoy to send access logs to curielogger.

• Network details:

  • Port 8001 is the Envoy administration interface

Building/Rebuilding Images

If for some reason you need to rebuild the images, run the following command: