The Challenge Process
Detecting Bots, Verifying Humans
Much of the Curiefense interface focuses on configuration for:
The detection and mitigation of hostile requests.
The detection and mitigation of hostile behavior of requestors.
The challenge process is different. It is built into Curiefense, and provides a third approach to security. It mitigates threats based on the requestor's identity, environment, and behavior.
Bot challenges are included in the Premium version of Curiefense.
When Curiefense receives the first request from a previously unknown traffic source (below described as the "user") for a browser-based web application, this is the typical process that is followed.
Curiefense challenges the user's browsing environment. Curiefense uses a variety of proprietary, multi-faceted techniques to verify that this requestor is a human using a browser, instead of a bot using a headless browser or emulator.
The user’s browser is subjected to several dozen tests, verifying the features known to be supported by that browser. This includes hidden canvases, video and audio in various formats, WebRTC and other advanced networking protocols, screen resolution, and more.
The browser is subjected to an invisible “attack”: subtle errors are injected into the environment, and the browser engine’s reactions are captured and analyzed. Curiefense verifies that the exceptions and error messages are those which should be generated, if the browser is what it claims to be. (It is very difficult for threat actors to spoof this behavior using headless browsers and emulators, since there is an infinite number of possible errors to which any browser can be subjected, and threat actors need to replicate the actual reactions for each possible input.)
The above process only takes milliseconds, and it is completely transparent to the end user.
Once a browser has passed these tests, Curiefense signs it cryptographically. This signature accompanies all subsequent activity.
If the challenge is not passed, the request is suspected to be a bot, and another challenge is issued. This process continues until a challenge is passed, or a threshold is reached (e.g., via a Rate Limit) to ban the requestor.
If the challenge is passed, the browser's session is authenticated, and the browser receives cookies from Curiefense.
The browser then automatically resubmits the original request, but this time, the cookies are included. The user is granted access to the requested URL, resources, etc.
Subsequent requests will also include the cookies, and thus, they are not challenged.
This process happens quickly (in a few milliseconds), and is invisible to the user.
Active Challenges versus Passive Challenges
The process described above is the active challenge process. Out of the box, this is the challenge process that Curiefense uses.
We recommend that whenever possible, users also enable passive challenges.
Passive challenges add three additional benefits:
They enable biometric behavioral profiling: a much more powerful means of identifying automated traffic. More on this below.
In some situations, active challenges can interfere with certain metrics such as those provided by Google Analytics. (The initial referrer information is lost.) If this is a problem, active challenges can be disabled. In this situation, passive challenges can provide effective bot protection instead.
When caching is being done by a CDN, active challenges will not occur for pages being served from the cache. Passive challenges are necessary for Curiefense to perform bot detection in this situation.
If possible, we recommend that users use both active and passive challenges.
Biometric Behavioral Profiling
With Biometric Bot Detection, Curiefense continually gathers and analyzes stats such as client-side I/O events, triggered by the user’s keyboard, mouse, scroll, touch, zoom, device orientation, movements, and more. Based on these metrics, the platform uses Machine Learning to construct and maintain behavioral profiles of legitimate human visitors. Curiefense learns and understands how actual humans interact with the applications and services it is protecting. Continuous multivariate analysis verifies that each user is indeed conforming to expected behavioral patterns, and is thus a human user with legitimate intentions.
Using this approach, Curiefense bot detection accuracy is not only high, it is also robust and resistant to reverse-engineering by threat actors. Behavioral profiles are constructed based on private analytics data, and threat actors have no realistic way of obtaining this information.
This is an important reason why we recommend that all users enable Passive Challenges if it is possible to do so. Biometric Behavioral Profiling provides much more robust detection of automated traffic than is possible without it.
Implementation
Implementing Passive Challenges is simple. Place this Javascript code within the pages of your web applications:
The code snippet can go into the header or at the end of the page. The best practice is to place it within the header. This ensures that subsequent calls contain authenticating cookies.
(This matters for the first page served to a visitor. Subsequent pages will already have the authenticating cookies to include with their requests.)
Most users set up the code snippet as a tag within their tag manager. This makes it simple to install it instantly across their entire site/application.
If desired, the script code can include async and/or defer attributes:
These usually are not necessary, and their effect will depend on the placement of the script within the page. Their use is left to your discretion.
Testing
To test the implementation, use a browser to visit a page containing the Javascript snippet. Once it runs, the browser should have a cookie named rbzid.
Disabling Active Challenges (only if necessary)
There are two primary situations where users sometimes want to disable Active Challenges:
When a user needs site analytics to correctly reflect all referrers. (Active Challenges can interfere with this.)
For API endpoints. Active Challenges are designed to verify the client's browser environment; for most API calls, there is no browser environment to verify.
Other than those situations, Active Challenges can be very beneficial.
We recommend that you keep Active Challenges enabled if possible. They automatically eliminate almost all DDoS traffic, scanning tools, and other hostile bot traffic.
If you wish to turn off Active Challenges, remove all tags from the "Deny Bot" column within ACL Profiles.
If you have not enabled Passive Challenges (and successfully tested them), disabling Active Challenges is not recommended.
Last updated