Planning your Scuba deployment

This document offers guidelines for planning a Scuba deployment that meets your company's current data analytics needs, as well as scaled growth for the future. 

Note that for most deployments, you will work with a Scuba representative to determine sizing, set up cluster(s), and deploy to production. If this is not the case, please reach out to help@scuba.io to ensure you have access to all necessary Scuba Admin documentation.

Scuba—behavioral analytics for your whole team

Scuba is full stack behavioral analytics software with a web-based visual interface and scalable distributed back-end datastore to process queries on event data.

Scuba's intuitive graphical user interface (GUI) provides interactive data exploration for a wide range of users across the spectrum of digital businesses. The Visual Explorer encourages rapid iteration with point-and-click query building and interactive visualizations. All without having to deal with complicated query syntax. You can go from any board to explore the underlying data, change parameters, and drill down to the granular details of the summary.

To plan a Scuba deployment that is optimized for your company, review the following topics:

• High-level overview—how Scuba works

• The basics—system requirements

• Production workflow—test, review, revise, and go

• Cluster configurations—the big picture

• Sizing guidelines—for today and tomorrow

• Data types and formats—consider your data source

• Platform-specific planning—how much is enough?

High-level overview—how Scuba works

Scuba is full stack behavioral analytics software that allows users to explore the activity of digital services. Scuba includes both its own web-based visual interface and a highly scalable distributed back end database to store the data and process queries. Scuba supports Ubuntu 14.04.x in cloud environments, as well as virtual machines or bare-metal systems. Scuba enables you to ingest data in a variety of ways, including live data streams.

The following image shows an example of the flow of data into the Scuba cluster from imported files (single and batch) to live data streams from HTTP and Kafka sources. Ingested data is transformed and stored in the appropriate node, data or string, then processed in queries, with the results delivered to the requesting user.

The basics—system requirements

It is important to understand the hardware, software, and networking infrastructure requirements to properly plan your Scuba deployment. Before making a resource investment, read through this entire document. The following sections explain sizing guidelines, data formats, and platform specific information needed to accurately estimate a Scuba production system for your company.

A Scuba cluster consists of the following nodes that can be installed on a single server (single-node cluster), or across multiple servers (multi-node cluster).

• config node — Node from which you administer the cluster. MySQL database (DB) is only installed on this node for storage of Scuba metadata. Configure this node first.

• api node — Serves the Scuba application, merges query results from data and string nodes, and then presents those results to the user. Nginx is only installed on the API node. 

• import (ingest) node — Connects to data repositories (cloud, live streaming, remote or local file system), streams live data, downloads new files, processes the data and then sends to data and string tiers, as appropriate.

• data node — Data storage. Must have enough space to accommodate all events and stream simultaneous query results.

• string node — String storage for the active strings in the dataset, stored in compressed format. Requires sufficient memory to hold the working set of strings accessed during queries.

• listener node — Streams live data from the web or cloud, also known as streaming ingest. Only applicable when ingesting from HTTP.

The following table outlines basic software, hardware, and network requirements for a Scuba deployment.

Guidelines for calculating the number of node type for your deployment are covered in the following sections.

Production workflow—test, review, revise, and go

It is strongly recommended that you first set up a sandbox test cluster (or work with your Scuba rep to have one set up for you) with a sample of your data, so any necessary adjustments can be made before deploying a production environment. This will allow you to assess the quality of your data and better determine the appropriate size for a production cluster. 

For self-service Scuba deployments, follow these steps:

1. Review the rest of this document to gain a preliminary assessment of the needs for your production cluster.

2. Install a single node cluster in a sandbox test environment.

3. Load a week's worth of data.

4. Modify your data formats until you get the desired results.

5. Review the usage for the string and data nodes.

6. From the usage for one week's worth of data, you can estimate usage for one month, and then one year.

7. Factor in the estimated growth percentage for your company to create a multi-year usage projection.

8. Set up your Scuba production environment and ingest your data.

Cluster configurations—the big picture

This section provides an overview of a several typical cluster configurations:

• Stacked vs. single nodes

• Single-node cluster

• Two node cluster

• Five node cluster

• Ten node cluster

Stacked vs. single nodes

The configuration of your cluster depends on the amount of data you have, its source, and projected growth. To validate the quality of your data and the analytics you wish to perform, we require a sample of your data prior to going into production. 

Stacked nodes are not recommended for large event volumes or workloads, or if you are planning to scale your cluster at a later date.

For smaller deployments, it is possible to stack nodes—install multiple services on one system—in the following ways:

• API, config, ingest—admin 

• string, data—worker

Review the following configuration examples and sizing guidelines, followed by data format and platform-specific issues.

Single-node cluster

The stacked single-node cluster is only recommended for sandbox test environments that are used to validate data and assess the requirements for a production cluster.

Two node cluster

The stacked two-node cluster configuration is for deployments with very low data volumes. In this configuration, the API and config nodes are stacked on the same node. A Listener node for streaming ingest from a live data source is not included, as this tier is optional.

Five node cluster

The following is an example of a basic five node cluster configuration, in which each functional component is installed on a separate system. A Listener node for streaming ingest from a live data source is not included, as this tier is optional.

Ten node cluster

The ten node cluster configuration illustrates how to scale a cluster for increased data capacity with added data and string nodes. To increase disk capacity, it is recommended that you scale the cluster horizontally to maintain your original performance.

Sizing guidelines—for today and tomorrow

Determining the appropriate Scuba cluster size for your company is largely based on the quantity of daily events that are generated. First, determine the current number of generated events, then estimate your company's projected data growth over a year, then multiply by the number of years in your expected growth cycle.

Consider the following when planning the size and capacity of your production Scuba cluster:

• Node configurations

• Capacity planning

• Planning the data tier

• Planning the ingest tier

• Planning the string tier

• Cluster configuration guidelines

Node configurations

The following table provides guidelines for Scuba cluster node configurations.

Capacity planning

Use the following guidelines to estimate the capacity for a Scuba production cluster.

Planning the data tier

The main constraint for a data tier is disk space. However, beware of under-provisioning CPU and memory resources as it can result in reduced performance. Consider the following guidelines when planning a production data tier:

For older generation CPUs, 8 MB of L3 cache is the required minimum. Pre-2010 CPUs are not supported.

Planning the import tier

Consider the following guidelines when planning a production import (ingest) tier:

Beware of over-provisioning. If the data or string tiers are over capacity, the ingest nodes can be affected causing a drop in performance. 

Planning the string tier

Consider the following guidelines when planning a production string tier:

Cluster configuration guidelines

It is recommended that you follow the production workflow, and first deploy a sandbox test cluster with sample data to properly assess your data and accurately estimate a production cluster. 

Use the following guidelines in planning your cluster:

• For a sandbox test cluster, start with one main dataset on the cluster, with 1 shard key. A separate table copy is required for each shard key. This means that if you want N shard keys for your dataset, you need N times the storage on your data tier. Defining the appropriate number and type of shard keys may be an iterative process for your production cluster. It is recommended that you start small and add additional shard keys as necessary, since each shard key requires a full copy of the dataset.

• Log data usually ranges from 50 - 150 bytes / event (storage required for 1 compressed event on the data tier). 80 bytes / event is a reasonable density estimate. High cardinality integer data, such as timestamps and unique identifiers and set columns, take up the most space in a dataset. If your data contains a lot of this type of data, estimate a few more data nodes for your production deployment.

• String data should be stored efficiently. The following estimates account for data with several high cardinality string columns: 1 or 2 parsed URL columns, 1 user agent column, and 1 IP address column. Hex identifier columns should be stored as integers, through the use of a hex transform (applied by default in Scuba). Other "string" identifiers (like base64 encoded session ids) are hashed into integers and stored on the data tier.

• Consider your expected data retention and adjust the configuration accordingly. You will need more data nodes for a 90-day retention than for 30 days.  

• Data input volume is another factor to consider when planning a cluster. For instance, a single-node cluster which has a total capacity of 24B events is adequate for 267M (24B / 90D) events per day. To import a higher volume of events per day, larger import and data tiers are required to keep up with the ingest rate and service queries in a timely manner.

Data types and formats—consider your source

It's important to consider the source of your data, as well the data type and how it's structured. Streaming ingest requires a special cluster configuration and installation. Be aware that some data types may require transformation for optimum analytics. 

Data types

Scuba accepts the following data types:

• JSON—Scuba's preferred data format. The JSON format is a flat set of name-value pairs (without any nesting), which is easy for Scuba to parse and interpret. If you use a different format, your data must be transformed into JSON format before it can be imported into Scuba. 

• Apache log format—Log files generated using mod_log_config. For details, see mod_log_config - Apache HTTP Server Version 2.4. It's helpful to provide the mod_log_config format string used to generate the logs, as Scuba can use that same format string to ingest the logs.

• CSV—Scuba accepts CSV format with some exceptions. First, ensure sure you have a complete header row. Next, ensure you are using a supported separator character (tab, comma, semicolon, or ascii character 001). Finally, ensure your separator is clean and well-escaped—for example, if you use comma as your separator, make sure to quote or escape any commas within your actual data set.

For more information, see the Data types reference. For best practices in logging your data, see Best practices for formatting your data.

Data sources

Scuba can ingest event data a variety of ways:

• Amazon Simple Storage Service (S3)—Cloud storage (Amazon Web Services, AWS)

• Microsoft Azure—Blob storage

• Google Cloud Platform—Unified object storage

• Live data on an HTTP port—Streaming ingest

• Local file systems—See the Ingest file types and formats reference

Logging and adding data

How your data is structured is important for optimum analytics results and performance. Review the following topics:

• What to think about before you add data

• Best practices for formatting your data

• How Scuba handles data at ingest

Platform-specific planning—how much is enough?

The following table compares system recommendations for cluster configurations on the following platforms:

• Amazon Web Services, AWS

• Microsoft Azure Blob storage

On-premise installations should only use local SSDs, no Storage Area Networks (SAN).

The Azure recommendations in the following table allow you to add premium storage.