Data Types Reference [v5]

This reference guide enumerates each of the data types supported by Scuba, and explains how Scuba determines which data type to assign at ingest time.

Data types

These are the core data types you will see in the Scuba query UI.

Data Type	Aggregations	Grouping (Split By)	Filter Operators	Value Typeahead
Identifier	Count Unique, First, Last	Yes	matches/does not match is empty/is not empty	No
Integer* / Decimal	Count Unique, First, Last, Min, Max, Sum, Average, Median, Percentile	Defaults to No, can be configured to Yes by admin	is one of/is not one of is less than (<) is greater than (>) is less than or equal to (<=) is greater than or equal to (>=)	No
Integer Set / Decimal Set		Defaults to No, can be configured to Yes by admin	matches/does not match [column/value] is empty/is not empty	No
String	Count Unique, First, Last	Yes	matches/does not match text contains/does not contain starts with/does not start with ends with/does not end with is empty/is not empty text matches regular expression	Yes
String Set	Count Unique	Yes	string value matches/does not match set contains/does not contain text starts with/does not start with ends with/does not end with is empty/is not empty text matches regular expression	Yes
Time	Count Unique, First, Last, Min, Max, Sum, Average, Median, Percentile	No	is one of, is not one of, is less than (<), is greater than (>), is less than or equal to (<=), is greater than or equal to (>=)	No

*Integers in Scuba are stored as 64-bit signed ints and so can only be a value from -(2^63) to 2^63 - 1.

Expansion types

You won't see these as being first-class data types in the Scuba query UI, but at ingest time we apply expansion rules based on these data types.

Expansion Type	Data Type of Main Column	Generated Subcolumns
IP Address	String	city, region, country, continent
URL	N/A (not loaded into Scuba by default)	scheme, hostname, path, filename, query, params, fragment
User Agent	N/A (not loaded into Scuba)	browser, device, platform, browser_majorver, browser_minorver, browser_patchver

Ingest time auto-transformations

At ingest time, we automatically perform the following transformations on the raw JSON data before applying any data type recognition rules or expansions.

Transformation Rule	Original JSON	Resulting JSON
Flatten Nested JSON Objects	{"column": {"a": 1, "b": "xxx"}}	{"column.a": 1, "column.b": "xxx"}
Shred Arrays of JSON Objects	{"column": [{"a": 1,"b": "zzz"}, {"a": 2,"b": "yyy"}]}	{"column.a": [1,2], "column.b": ["zzz","yyy"]}

String column transformations

If you have a regular string column that includes only the letters a through h, Scuba will auto-detect it as an integer and convert it to base 10. For example, "funding_series" which has values like (a, b, c, d, e) might fall into this scenario.

Original data	Transformed into...
"1234"	Integer values
"abc123"	Integer (converted to base 10), for values up to 16 hex digits
"abcdef-124"	Identifier, for values of 17 digits or more. The value can include hyphens.

Ingest Time Data Type Recognition Rules For JSON Number Columns

At ingest time, when we see a new column for the first time (and it is a JSON Number) we detect the data type of the new column using the following matching rules, in the precedence order listed below:

Parsing Rule	Raw Data	Data Type	Rule Details
Detect Time	{"abc" : 1448933490}	Time	Scuba will attempt to interpret JSON ints as epoch timestamps in one of (microseconds, milliseconds, seconds).
Detect Integer	{"abc" : 12345}	Integer	Simple JSON ints are interpreted as Integers by Scuba.
Detect Decimal	{"abc" : 12345.98}	Decimal
Detect Integer Set	{"abc" : [12345, 245, 99834]}	Integer Set
Detect Decimal Set	{"abc" : [12345.98, 245.2, 99834]}	Decimal Set

Based on the order of precedence, if there is ambiguity about whether a column value can be interpreted as an epoch timestamp or an int, Scuba will interpret it as an epoch time value.

Ingest time data type recognition rules for JSON string columns

At ingest time, when we see a new column for the first time (and it is a JSON String) we detect the data type of the new column using the following matching rules, in the precedence order listed below:

Parsing Rule	Raw Data	Data Type	Rule Details
Detect Time From JSON String	{"abc" : "2015-11-30 08:09:12"}	Time	Scuba will attempt to interpret JSON strings as timestamps using approximately 40 different format strings (including ISO-8601).
Detect Identifier from JSON String	{"abc" : "e41249ed-2398-4c29-a6fa-ee81116dd302"}	Identifier	Scuba will attempt to interpret JSON strings containing 16 or more characters as hexadecimal identifiers, including some common uuid formats. Note that non-hex characters (like hyphens or dots) are stripped out of the resulting data.
Detect Integer From JSON String	{"abc" : "12345"}	Integer
Detect Decimal From JSON String	{"abc" : "12345.98"}	Decimal
Detect Decimal From JSON String With Dollar Sign	{"abc" : "$12,345.98"}	Decimal	Note that dollar signs and commas are stripped out of the resulting data.
Detect String Set	{"abc" : ["hello", "goodbye", "nice", "to", "see", "you"]}	String Set
Detect URL	{"abc" : "http://www.site.com/landing/"}	URL
Detect IP Address	{"abc" : "127.0.0.1"}	IP Address
Detect User Agent	{"abc" : "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_0)"}	User Agent	Scuba will attempt to interpret JSON strings as User Agents using a regex matching scheme.

If you have any questions, please reach out to help@scuba.io or your technical customer success manager (TCSM).