API Reference

IP

clx.ip.hostmask(ips, prefixlen=16)

Compute a column of hostmasks for a column of IP addresses. Addresses must be IPv4. IPv6 not yet supported.

Parameters

ipsIPv4 address: IP addresses to be checked
prefixlen: integer: Length of the network prefix, in bits, for IPv4 addresses

Returns

rtypecudf.Series, hostmask: Hostmask ouput from set of IP address

Examples

>>> import clx.ip
>>> import cudf
>>> clx.ip.hostmask(cudf.Series(["192.168.0.1","10.0.0.1"], prefixlen=16)
0    0.0.255.255
1    0.0.255.255
Name: hostmask, dtype: object

clx.ip.int_to_ip(values)

Convert integer column to IP addresses. Addresses must be IPv4. IPv6 not yet supported.

Parameters

valuescudf.Series, integer: Integer representations of IP addresses

Returns

rtypecudf.Series, IPv4 address: IP addresses to be converted

Examples

>>> import clx.ip
>>> import cudf
>>> clx.ip.int_to_ip(cudf.Series([3232235521, 167772161]))
0     5.79.97.178
1    94.130.74.45
dtype: object

clx.ip.ip_to_int(values)

Convert string column of IP addresses to integer values. Addresses must be IPv4. IPv6 not yet supported.

Parameters

valuescudf.Series, IPv4 address: IP addresses to be converted

Returns

rtypecudf.Series, integer: Integer representations of IP addresses

Examples

>>> import clx.ip
>>> import cudf
>>> clx.ip.ip_to_int(cudf.Series(["192.168.0.1","10.0.0.1"]))
0      89088434
1    1585596973
dtype: int64

clx.ip.is_global(ips)

Indicates whether each address is global. Addresses must be IPv4. IPv6 not yet supported.

Parameters

ipsIPv4 address: IP addresses to be checked

Returns

rtypecudf.Series, booleans: Boolean values true or false

Examples

>>> import clx.ip
>>> import cudf
>>> clx.ip.is_global(cudf.Series(["127.0.0.1","207.46.13.151"]))
0    False
1    True
dtype: bool

clx.ip.is_ip(ips)

Indicates whether each address is an ip string. Addresses must be IPv4. IPv6 not yet supported.

Parameters

ipsIPv4 address: IP addresses to be checked

Returns

rtypecudf.Series, booleans: Boolean values true or false

Examples

>>> import clx.ip
>>> import cudf
>>> clx.ip.is_ip(cudf.Series(["192.168.0.1","10.123.0"]))
0     True
1    False
dtype: bool

clx.ip.is_link_local(ips)

Indicates whether each address is link local. Addresses must be IPv4. IPv6 not yet supported.

Parameters

ipsIPv4 address: IP addresses to be checked

Returns

rtypecudf.Series, booleans: Boolean values true or false

Examples

>>> import clx.ip
>>> import cudf
>>> clx.ip.is_link_local(cudf.Series(["127.0.0.1","169.254.123.123"]))
0    False
1    True
dtype: bool

clx.ip.is_loopback(ips)

Indicates whether each address is loopback. Addresses must be IPv4. IPv6 not yet supported.

Parameters

ipsIPv4 address: IP addresses to be checked

Returns

rtypecudf.Series, booleans: Boolean values true or false

Examples

>>> import clx.ip
>>> import cudf
>>> clx.ip.is_loopback(cudf.Series(["127.0.0.1","10.0.0.1"]))
0     True
1    False
dtype: bool

clx.ip.is_multicast(ips)

Indicates whether each address is multicast. Addresses must be IPv4. IPv6 not yet supported.

Parameters

ipsIPv4 address: IP addresses to be checked

Returns

rtypecudf.Series, booleans: Boolean values true or false

Examples

>>> import clx.ip
>>> import cudf
>>> clx.ip.is_multicast(cudf.Series(["127.0.0.1","224.0.0.0"]))
0    False
1    True
dtype: bool

clx.ip.is_private(ips)

Indicates whether each address is private. Addresses must be IPv4. IPv6 not yet supported.

Parameters

ipsIPv4 address: IP addresses to be checked

Returns

rtypecudf.Series, booleans: Boolean values true or false

Examples

>>> import clx.ip
>>> import cudf
>>> clx.ip.is_private(cudf.Series(["127.0.0.1","207.46.13.151"]))
0    True
1    False
dtype: bool

clx.ip.is_reserved(ips)

Indicates whether each address is reserved. Addresses must be IPv4. IPv6 not yet supported.

Parameters

ipsIPv4 address: IP addresses to be checked

Returns

rtypecudf.Series, booleans: Boolean values true or false

Examples

>>> import clx.ip
>>> import cudf
>>> clx.ip.is_reserved(cudf.Series(["127.0.0.1","10.0.0.1"]))
0    False
1    False
dtype: bool

clx.ip.is_unspecified(ips)

Indicates whether each address is unspecified. Addresses must be IPv4. IPv6 not yet supported.

Parameters

ipsIPv4 address: IP addresses to be checked

Returns

rtypecudf.Series, booleans: Boolean values true or false

Examples

>>> import clx.ip
>>> import cudf
>>> clx.ip.is_unspecified(cudf.Series(["127.0.0.1","10.0.0.1"]))
0    False
1    False
dtype: bool

clx.ip.mask(ips, masks)

Apply a mask to a column of IP addresses. Addresses must be IPv4. IPv6 not yet supported.

Parameters

ipsIPv4 address: IP addresses to be checked
masks: Subnet mask value: The host or subnet masks to be applied

Returns

rtypecudf.Series, masked IPv4 address: Masked IP address from list of IPs
Examples

>>> import clx.ip
    ..

>>> import cudf
    ..

>>> input_ips = cudf.Series(["192.168.0.1","10.0.0.1"])
    ..

>>> input_masks = cudf.Series(["255.255.0.0", "255.255.0.0"])
    ..

>>> clx.ip.mask(input_ips, input_masks)
    ..

0 192.168.0.0
1 10.0.0.0
Name: mask, dtype: object

clx.ip.netmask(ips, prefixlen=16)

Compute a column of netmasks for a column of IP addresses. Addresses must be IPv4. IPv6 not yet supported.

Parameters

ipsIPv4 address: IP addresses to be checked
prefixlen: integer: Length of the network prefix, in bits, for IPv4 addresses

Returns

rtypecudf.Series, netmask: Netmask ouput from set of IP address

Examples

>>> import clx.ip
>>> import cudf
>>> clx.ip.netmask(cudf.Series(["192.168.0.1","10.0.0.1"]), prefixlen=16)
0    255.255.0.0
1    255.255.0.0
Name: net_mask, dtype: object

Features

clx.features.binary(dataframe, entity_id, feature_id)

Create binary feature dataframe using provided dataset, entity, and feature.

Parameters

dataframe (cudf.DataFrame) – Input dataframe to create binary features
entity_id (str) – Entity ID. Must be a column within dataframe
feature_id (str) – Feature ID. Must be a column within dataframe

Returns

dataframe

Return type

cudf.DataFrame

Examples

>>> import cudf
>>> import clx.features
>>> df = cudf.DataFrame(
        {
            "time": [1, 2, 3],
            "user": ["u1", "u2", "u1",],
            "computer": ["c1", "c1", "c3"],
        }
    )
>>> output = clx.features.binary(df, "user", "computer")
>>> output
        c1  c3
    user
    u1      1.0     1.0
    u2      1.0     0.0

clx.features.frequency(dataframe, entity_id, feature_id)

Create frequency feature dataframe using provided dataset, entity, and feature.

Parameters

dataframe (cudf.DataFrame) – Input dataframe to create binary features
entity_id (str) – Entity ID. Must be a column within dataframe
feature_id (str) – Feature ID. Must be a column within dataframe

Returns

dataframe

Return type

cudf.DataFrame

Examples

>>> import cudf
>>> import clx.features
>>> df = cudf.DataFrame(
        {
            "time": [1, 2, 3],
            "user": ["u1", "u2", "u1",],
            "computer": ["c1", "c1", "c3"],
        }
    )
>>> output = clx.features.binary(df, "user", "computer")
>>> output
        c1  c3
    user
    u1      0.5     0.5
    u2      1.0     0.0

Analytics

class clx.analytics.asset_classification.AssetClassification(layers=[200, 100], drops=[0.001, 0.01], emb_drop=0.04, is_reg=False, is_multi=True, use_bn=True)

Supervised asset classification on tabular data containing categorical and/or continuous features.

Parameters

layers – linear layer follow the input layer
drops – drop out percentage
emb_drop – drop out percentage at embedding layers
is_reg – is regression
is_multi – is classification
use_bn – use batch normalization

Methods

`load_model`(fname)	Load a saved model.
`predict`(gdf, cat_cols, cont_cols)	Predict the class with the trained model
`save_model`(fname)	Save trained model
`train_model`(train_gdf, cat_cols, cont_cols, ...)	This function is used for training fastai tabular model with a given training dataset.

load_model(fname)

Load a saved model.

Parameters: fname (str) – directory path to model

Examples

>>> from clx.analytics.asset_classification import AssetClassification
>>> ac = AssetClassification()
>>> ac.load_model("ac.mdl")

predict(gdf, cat_cols, cont_cols)

Predict the class with the trained model

Parameters

gdf (cudf.DataFrame) – prediction input dataset with categorized int16 feature columns
cat_cols – array of categorical column names in gdf
cont_col – array of continuous column names in gdf

Examples

>>> cat_cols = ["1", "2", "3", "4", "5", "6", "7", "8", "9"]
>>> cont_cols = ["10"]
>>> ac.predict(X_test, cat_cols, cont_cols).values_host
0       0
1       0
2       0
3       0
4       2
    ..
8204    0
8205    4
8206    0
8207    3
8208    0
Length: 8209, dtype: int64

save_model(fname)

Save trained model

Parameters: save_to_path (str) – directory path to save model

Examples

>>> from clx.analytics.asset_classification import AssetClassification
>>> ac = AssetClassification()
>>> cat_cols = ["1", "2", "3", "4", "5", "6", "7", "8", "9"]
>>> cont_cols = ["10"]
>>> ac.train_model(X_train, cat_cols, cont_cols, "label", batch_size, epochs, lr=0.01, wd=0.0)
>>> ac.save_model("ac.mdl")

train_model(train_gdf, cat_cols, cont_cols, label_col, batch_size, epochs, lr=0.01, wd=0.0)

This function is used for training fastai tabular model with a given training dataset.

Parameters

train_gdf (cudf.DataFrame) – training dataset with categorized and/or continuous feature columns
cat_cols – array of categorical column names in train_gdf
cont_col – array of continuous column names in train_gdf
label_col (str) – column name of label column in train_gdf
batch_size (int) – train_gdf will be partitioned into multiple dataframes of this size
epochs (int) – number of epochs to be adjusted depending on convergence for a specific dataset
lr (float) – learning rate
wd (float) – wd

Examples

>>> from clx.analytics.asset_classification import AssetClassification
>>> ac = AssetClassification()
>>> cat_cols = ["1", "2", "3", "4", "5", "6", "7", "8", "9"]
>>> cont_cols = ["10"]
>>> ac.train_model(X_train, cat_cols, cont_cols, "label", batch_size, epochs, lr=0.01, wd=0.0)

class clx.analytics.detector.Detector(lr=0.001)

Attributes

criterion
model
optimizer

Methods

`leverage_model`(model)	This function leverages model by setting parallelism parameters.
`load_model`(file_path)	This function load already saved model and sets cuda parameters.
`save_model`(file_path)	This function saves model to a given location.

init_model
predict
train_model

leverage_model(model)

This function leverages model by setting parallelism parameters.

Parameters: model (RNNClassifier) – Model instance.

load_model(file_path)

This function load already saved model and sets cuda parameters.

Parameters: file_path (string) – File path of a model to be loaded.

save_model(file_path)

This function saves model to a given location.

Parameters: file_path (string) – File path of a model to be saved.

class clx.analytics.dga_dataset.DGADataset(df, truncate)

Constructor to create DGADataset instance.

Parameters

df (cudf.DataFrame) – Input dataframe.
truncate (int) – Truncate string to n number of characters.

class clx.analytics.dga_detector.DGADetector(lr=0.001)

This class provides multiple functionalities such as build, train and evaluate the RNNClassifier model to distinguish legitimate and DGA domain names.

Methods

`evaluate_model`(dataloader)	This function evaluates the trained model to verify it's accuracy.
`init_model`([char_vocab, hidden_size, ...])	This function instantiates RNNClassifier model to train.
`load_checkpoint`(file_path)	This function load already saved model checkpoint and sets cuda parameters.
`predict`(domains[, probability, truncate])	This function accepts cudf series of domains as an argument to classify domain names as benign/malicious and returns the learned label for each object in the form of cudf series.
`save_checkpoint`(file_path)	This function saves model checkpoint to given location.
`train_model`(train_data, labels[, ...])	This function is used for training RNNClassifier model with a given training dataset.

evaluate_model(dataloader)

This function evaluates the trained model to verify it’s accuracy.

Parameters: dataloader (DataLoader) – Instance holds preprocessed data.
Returns: Model accuracy
Return type: decimal

Examples

>>> dd = DGADetector()
>>> dd.init_model()
>>> dd.evaluate_model(dataloader)
Evaluating trained model ...
Test set accuracy: 3/4 (0.75)

init_model(char_vocab=128, hidden_size=100, n_domain_type=2, n_layers=3)

This function instantiates RNNClassifier model to train. And also optimizes to scale it and keep running on parallelism.

Parameters

char_vocab (int) – Vocabulary size is set to 128 ASCII characters.
hidden_size (int) – Hidden size of the network.
n_domain_type (int) – Number of domain types.
n_layers (int) – Number of network layers.

load_checkpoint(file_path)

This function load already saved model checkpoint and sets cuda parameters.

Parameters: file_path (string) – File path of a model checkpoint to be loaded.

predict(domains, probability=False, truncate=100)

This function accepts cudf series of domains as an argument to classify domain names as benign/malicious and returns the learned label for each object in the form of cudf series.

Parameters

domains (cudf.Series) – List of domains.
truncate (int) – Truncate string to n number of characters.

Returns

Predicted results with respect to given domains.

Return type

cudf.Series

Examples

>>> dd.predict(['nvidia.com', 'dgadomain'])
0    0.010
1    0.924
Name: dga_probability, dtype: decimal

save_checkpoint(file_path)

This function saves model checkpoint to given location.

Parameters: file_path (string) – File path to save model checkpoint.

train_model(train_data, labels, batch_size=1000, epochs=5, train_size=0.7, truncate=100)

This function is used for training RNNClassifier model with a given training dataset. It returns total loss to determine model prediction accuracy.

Parameters

train_data (cudf.Series) – Training data
labels (cudf.Series) – labels data
batch_size (int) – batch size
epochs (int) – Number of epochs for training
train_size (int) – Training size for splitting training and test data
truncate (int) – Truncate string to n number of characters.

Examples

>>> from clx.analytics.dga_detector import DGADetector
>>> dd = DGADetector()
>>> dd.init_model()
>>> dd.train_model(train_data, labels)
1.5728906989097595

class clx.analytics.loda.Loda(n_bins=None, n_random_cuts=100)

Anomaly detection using Lightweight Online Detector of Anomalies (LODA). LODA detects anomalies in a dataset by computing the likelihood of data points using an ensemble of one-dimensional histograms.

Parameters

n_bins (int) – Number of bins for each histogram. If None a heuristic is used to compute the number of bins.
n_random_cuts (int) – Number of projection to use.

Methods

`explain`(anomaly[, scaled])	Explain anomaly based on contributions (t-scores) of each feature across histograms.
`fit`(train_data)	Fit training data and construct histograms.
`load_model`(file_path)	This function load already saved model and sets cuda parameters.
`save_model`(file_path)	This function save model to given location.
`score`(input_data)	Calculate anomaly scores using negative likelihood across n_random_cuts histograms.

explain(anomaly, scaled=True)

Explain anomaly based on contributions (t-scores) of each feature across histograms.

Parameters

anomaly (cupy.ndarray) – selected anomaly from input dataset
scaled (boolean) – set to scale output feature importance scores

Examples

>>> loda_ad.explain(x[5]) # x[5] is found anomaly
array([[1.        ],
[0.        ],
[0.69850349],
[0.91081035],
[0.78774349]])

fit(train_data)

Fit training data and construct histograms.

Parameters: train_data (cupy.ndarray) – NxD training sample

Examples

>>> from clx.analytics.loda import Loda
>>> import cupy as cp
>>> x = cp.random.randn(100,5) # 5-D multivariate synthetic dataset
>>> loda_ad = Loda(n_bins=None, n_random_cuts=100)
>>> loda_ad.fit(x)

classmethod load_model(file_path): This function load already saved model and sets cuda parameters. :param file_path: File path of a model to load. :type filel_path: string

save_model(file_path)

This function save model to given location.

Parameters: file_path (string) – File path to save model.

score(input_data)

Calculate anomaly scores using negative likelihood across n_random_cuts histograms.

Parameters: input_data (cupy.ndarray) – NxD training sample

Examples

>>> from clx.analytics.loda import Loda
>>> import cupy as cp
>>> x = cp.random.randn(100,5) # 5-D multivariate synthetic dataset
>>> loda_ad = Loda(n_bins=None, n_random_cuts=100)
>>> loda_ad.fit(x)
>>> loda_ad.score(x)
array([0.04295848, 0.02853553, 0.04587308, 0.03750692, 0.05050418,
0.02671958, 0.03538646, 0.05606504, 0.03418612, 0.04040502,
0.03542846, 0.02801463, 0.04884918, 0.02943411, 0.02741364,
0.02702433, 0.03064191, 0.02575712, 0.03957355, 0.02729784,
...
0.03943715, 0.02701243, 0.02880341, 0.04086408, 0.04365477])

class clx.analytics.model.rnn_classifier.RNNClassifier(input_size, hidden_size, output_size, n_layers, bidirectional=True)

Methods

forward(input, seq_lengths)

Defines the computation performed at every call.

forward(input, seq_lengths)

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class clx.analytics.model.tabular_model.TabularModel(emb_szs, n_cont, out_sz, layers, drops, emb_drop, use_bn, is_reg, is_multi)

Basic model for tabular data

Methods

forward(x_cat, x_cont)

Defines the computation performed at every call.

forward(x_cat, x_cont)

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

clx.analytics.anomaly_detection.dbscan(feature_dataframe, min_samples=3, eps=0.3)

Pass a feature dataframe to this function to detect anomalies in your feature dataframe. This function uses cuML DBSCAN to detect anomalies and outputs associated labels 0,1,-1.

Parameters

:param feature_dataframe: Feature dataframe to be used for clustering
:type feature_dataframe: cudf.DataFrame
:param min_samples: Minimum samples to use for dbscan
:type min_samples: int
:param eps: Max distance to use for dbscan
:type eps: float

Examples

>>> import cudf
>>> import clx.features
>>> import clx.analytics.anomaly_detection
>>> df = cudf.DataFrame(
>>>         {
>>>             "time": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14],
>>>             "user": ["u1","u1","u1","u1","u1","u1","u1","u1","u1","u1","u5","u4","u2","u3"],
>>>             "computer": ["c1","c2","c3","c1","c2","c3","c1","c1","c2","c3","c1","c1","c5","c6"],
>>>         }
>>>     )
>>> feature_df = clx.features.frequency(df, entity_id="user", feature_id="computer")
>>> labels = clx.analytics.anomaly_detection.dbscan(feature_df, min_samples=2, eps=0.5)
>>> labels
    0   -1
    1   -1
    2   -1
    dtype: int32

clx.analytics.periodicity_detection.filter_periodogram(prdg, p_value)

Select important frequencies by filtering periodogram by p-value. Filtered out frequencies are set to zero.

Parameters

prdg – periodogram to be filtered
p_value – p-value to filter by

Returns

CuPy array representing periodogram

Return type

cupy.ndarray

clx.analytics.periodicity_detection.to_periodogram(signal)

Returns periodogram of signal for finding frequencies that have high energy.

Parameters: signal (cudf.Series) – signal (time domain)
Returns: CuPy array representing periodogram
Return type: cupy.ndarray

clx.analytics.periodicity_detection.to_time_domain(prdg)

Convert the signal back to time domain.

Parameters: prdg (cupy.ndarray) – periodogram (frequency domain)
Returns: CuPy array representing reconstructed signal
Return type: cupy.ndarray

clx.analytics.stats.rzscore(series, window)

Calculates rolling z-score

Parameters

seriescudf.Series: Series for which to calculate rolling z-score
windowint: Window size

Returns

cudf.Series: Series with rolling z-score values

Examples

>>> import clx.analytics.stats
>>> import cudf
>>> sequence = [3,4,5,6,1,10,34,2,1,11,45,34,2,9,19,43,24,13,23,10,98,84,10]
>>> series = cudf.Series(sequence)
>>> zscores_df = cudf.DataFrame()
>>> zscores_df['zscore'] = clx.analytics.stats.rzscore(series, 7)
>>> zscores_df
            zscore
         null
         null
         null
         null
         null
         null
  2.374423424
 -0.645941275
 -0.683973734
  0.158832461
 1.847751909
 0.880026019
-0.950835449
-0.360593742
 0.111407599
 1.228914145
-0.074966331
-0.570321249
 0.327849973
-0.934372308
 2.296828498
 1.282966989
-0.795223674

DNS Extractor

clx.dns.dns_extractor.extract_hostnames(url_series)

This function extracts hostnames from the given urls.

Parameters: url_series (cudf.Series) – Urls that are to be handled.
Returns: Hostnames extracted from the urls.
Return type: cudf.Series

Examples

>>> from cudf import DataFrame
>>> from clx.dns import dns_extractor as dns
>>> input_df = DataFrame(
...     {
...         "url": [
...             "http://www.google.com",
...             "gmail.com",
...             "github.com",
...             "https://pandas.pydata.org",
...         ]
...     }
... )
>>> dns.extract_hostnames(input_df["url"])
0       www.google.com
1            gmail.com
2           github.com
3    pandas.pydata.org
Name: 0, dtype: object

clx.dns.dns_extractor.generate_tld_cols(hostname_split_df, hostnames, col_len)

This function generates tld columns.

Parameters

hostname_split_df (cudf.DataFrame) – Hostname splits.
hostnames (cudf.DataFrame) – Hostnames.
col_len – Hostname splits dataframe columns length.

Returns

Tld columns with all combination.

Return type

cudf.DataFrame

Examples

>>> import cudf
>>> from clx.dns import dns_extractor as dns
>>> hostnames = cudf.Series(["www.google.com", "pandas.pydata.org"])
>>> hostname_splits = dns.get_hostname_split_df(hostnames)
>>> print(hostname_splits)
     2       1       0
0  com  google     www
1  org  pydata  pandas
>>> col_len = len(hostname_split_df.columns) - 1
>>> col_len = len(hostname_splits.columns) - 1
>>> dns.generate_tld_cols(hostname_splits, hostnames, col_len)
     2       1       0 tld2        tld1               tld0
0  com  google     www  com  google.com     www.google.com
1  org  pydata  pandas  org  pydata.org  pandas.pydata.org

clx.dns.dns_extractor.parse_url(url_series, req_cols=None)

This function extracts subdomain, domain and suffix for a given url.

Parameters

url_df_col (cudf.Series) – Urls that are to be handled.
req_cols (set(strings)) – Columns requested to extract such as (domain, subdomain, suffix and hostname).

Returns

Extracted information of requested columns.

Return type

cudf.DataFrame

Examples

>>> from cudf import DataFrame
>>> from clx.dns import dns_extractor as dns
>>>
>>> input_df = DataFrame(
...     {
...         "url": [
...             "http://www.google.com",
...             "gmail.com",
...             "github.com",
...             "https://pandas.pydata.org",
...         ]
...     }
... )
>>> dns.parse_url(input_df["url"])
            hostname  domain suffix subdomain
0     www.google.com  google    com       www
1          gmail.com   gmail    com
2         github.com  github    com
3  pandas.pydata.org  pydata    org    pandas
>>> dns.parse_url(input_df["url"], req_cols={'domain', 'suffix'})
   domain suffix
0  google    com
1   gmail    com
2  github    com
3  pydata    org

Exploratory Data Analysis

class clx.eda.EDA(dataframe)

An EDA (Exploratory Data Analysis) Object. EDA is used to explore different features of a given dataframe.

Parameters: dataframe (cudf.DataFrame) – Dataframe to be used for analysis

Examples

>>> from clx.eda import EDA
>>> import cudf
>>> import pandas as pd
>>> df = cudf.DataFrame()
>>> df['a'] = [1,2,3,4]
>>> df['b'] = ['a','b','c','c']
>>> df['c'] = [True, False, True, True]
>>> df['d'] = cudf.Series(pd.date_range("2000-01-01", periods=3,freq="m"))
>>> eda = EDA(df)
>>> eda
    {
        "SummaryStatistics": {
            "a": {
                "dtype": "int64",
                "summary": {
                    "unique": "4",
                    "total": "4"
                }
            },
            "b": {
                "dtype": "object",
                "summary": {
                    "unique": "3",
                    "total": "4"
                }
            },
            "c": {
                "dtype": "bool",
                "summary": {
                    "true_percent": "0.75"
                }
            },
            "d": {
                "dtype": "datetime64[ns]",
                "summary": {
                    "timespan": "60 days, 2880 hours, 0 minutes, 0 seconds"
                }
            }
        }
    }

Attributes

analysis: Analysis results as a dict
dataframe: Dataframe used for analysis

Methods

`cuxfilter_dashboard`()	Create cuxfilter dashboard for Exploratory Data Analysis.
`save_analysis`(dirpath)	Save analysis output to directory path.

property analysis: Analysis results as a dict

cuxfilter_dashboard()

Create cuxfilter dashboard for Exploratory Data Analysis.

Returns: cuxfilter dashboard with populated with data and charts.
Return type: cuxfilter.DashBoard

property dataframe: Dataframe used for analysis

save_analysis(dirpath)

Save analysis output to directory path.

Parameters: dirpath (str) – Directory path to save analysis output.

Heuristics

clx.heuristics.ports.major_ports(addr_col, port_col, min_conns=1, eph_min=10000)

Find major ports for each address. This is done by computing the mean number of connections across all ports for each address and then filters out all ports that don’t cross this threshold. Also adds column for IANA service name correspondingto each port.

Parameters

addr_col (cudf.Series) – Column of addresses as strings
port_col (cudf.Series) – Column of corresponding port numbers as ints
min_conns (int) – Filter out ip:port rows that don’t have at least this number of connections (default: 1)
eph_min (int) – Ports greater than or equal to this will be labeled as an ephemeral service (default: 10000)

Returns

DataFrame with columns for address, port, IANA service corresponding to port, and number of connections

Return type

cudf.DataFrame

Examples

>>> import clx.heuristics.ports as ports
>>> import cudf
>>> input_addr_col = cudf.Series(["10.0.75.1","10.0.75.1","10.0.75.1","10.0.75.255","10.110.104.107", "10.110.104.107"])
>>> input_port_col = cudf.Series([137,137,7680,137,7680, 7680])
>>> ports.major_ports(input_addr_col, input_port_col, min_conns=2, eph_min=7000)
            addr  port     service  conns
0      10.0.75.1   137  netbios-ns      2
1 10.110.104.107  7680   ephemeral      2

OSI (Open Source Integration)

class clx.osi.farsight.FarsightLookupClient(server, apikey, limit=None, http_proxy=None, https_proxy=None)

Wrapper class to query DNSDB record in various ways Example: by IP, DomainName

Parameters

server – Farsight server
apikey – API key
limit – limit
http_proxy – HTTP proxy
https_proxy – HTTPS proxy

Methods

`query_rdata_ip`(rdata_ip[, before, after])	Query to find DNSDB records matching a specific IP address with given time range.
`query_rdata_name`(rdata_name[, rrtype, ...])	Query matches only a single DNSDB record of given owner name and time ranges.
`query_rrset`(oname[, rrtype, bailiwick, ...])	Batch version of querying DNSDB by given domain name and time ranges.

query_rdata_ip(rdata_ip, before=None, after=None)

Query to find DNSDB records matching a specific IP address with given time range.

Parameters

rdata_ip (str) – The VALUE is one of an IPv4 or IPv6 single address, with a prefix length, or with an address range. If a prefix is provided, the delimiter between the network address and prefix length is a single comma (“,”) character rather than the usual slash (“/”) character to avoid clashing with the HTTP URI path name separator..
before (UNIX timestamp) – Output results seen before this time.
after (UNIX timestamp) – Output results seen after this time.

Returns

Response

Return type

dict

Examples

>>> from clx.osi.farsight import FarsightLookupClient
>>> client = FarsightLookupClient("https://localhost", "your-api-key", limit=1)
>> client.query_rdata_ip("100.0.0.1")
{"status_code": 200,...}
>>> client.query_rdata_ip("100.0.0.1", before=1428433465, after=1538014110)
{"status_code": 200,...}

query_rdata_name(rdata_name, rrtype=None, before=None, after=None)

Query matches only a single DNSDB record of given owner name and time ranges.

Parameters

rdata_name (str) – DNS domain name.
rrtype (str) – The resource record type of the resource record, either using the standard DNS type mnemonic, or an RFC 3597 generic type, i.e. the string TYPE immediately followed by the decimal RRtype number.
before (UNIX timestamp) – Output results seen before this time.
after (UNIX timestamp) – Output results seen after this time.

Returns

Response

Return type

dict

Examples

>>> from clx.osi.farsight import FarsightLookupClient
>>> client = FarsightLookupClient("https://localhost", "your-api-key", limit=1)
>>> client.query_rdata_name("www.farsightsecurity.com")
{"status_code": 200,...}
>>> client.query_rdata_name("www.farsightsecurity.com", rrtype="PTR", before=1386638408, after=1561176503)
{"status_code": 200,...}

query_rrset(oname, rrtype=None, bailiwick=None, before=None, after=None)

Batch version of querying DNSDB by given domain name and time ranges.

Parameters

oname (str) – DNS domain name.
rrtype (str) – The resource record type of the resource record, either using the standard DNS type mnemonic, or an RFC 3597 generic type, i.e. the string TYPE immediately followed by the decimal RRtype number.
bailiwick (str) – The “bailiwick” of an RRset in DNSDB observed via passive DNS replication is the closest enclosing zone delegated to a nameserver which served the RRset.
before (UNIX timestamp) – Output results seen before this time.
after (UNIX timestamp) – Output results seen after this time.

Returns

Response

Return type

dict

Examples

>>> from clx.osi.farsight import FarsightLookupClient
>>> client = FarsightLookupClient("https://localhost", "your-api-key")
>>> client.query_rrset("www.dnsdb.info")
{"status_code": 200,...}
>>> client.query_rrset("www.dnsdb.info", rrtype="CNAME", bailiwick="dnsdb.info.", before=1374184718, after=1564909243,)
{"status_code": 200,...}

class clx.osi.virus_total.VirusTotalClient(api_key=None, proxies=None)

Wrapper class to query VirusTotal database.

Parameters

apikey – API key
proxies – proxies

Attributes

api_key
proxies
vt_endpoint_dict

Methods

`domain_report`(domain)	Retrieve report using domain.
`file_report`(*resource)	Retrieve file scan reports
`file_rescan`(*resource)	This function rescan given files.
`file_scan`(file)	This function allows you to send a file for scanning with VirusTotal.
`ipaddress_report`(ip)	Retrieve report using ip address.
`put_comment`(resource, comment)	Post comment for a file or URL
`scan_big_file`(files)	Scanning files larger than 32MB
`url_report`(*resource)	Retrieve URL scan reports
`url_scan`(*url)	Retrieve URL scan reports

domain_report(domain)

Retrieve report using domain.

Parameters: domain (str) – A domain name
Returns: Response
Return type: dict

Examples

>>> from clx.osi.virus_total import VirusTotalClient
>>> client = VirusTotalClient(api_key='your-api-key')
>>> client.domain_report("027.ru")
{'status_code': 200, 'json_resp': {'BitDefender category': 'parked', 'undetected_downloaded_samples'...}}

file_report(*resource)

Retrieve file scan reports

Parameters

*resource –

The resource argument can be the MD5, SHA-1 or SHA-256 of a file for which you want to retrieve

the most recent antivirus report. You may also specify a scan_id returned by the /file/scan endpoint. :type *resource: str :return: Response :rtype: dict

Examples

>>> from clx.osi.virus_total import VirusTotalClient
>>> client = VirusTotalClient(api_key='your-api-key')
>>> client.file_report(["99017f6eebbac24f351415dd410d522d"])
{'status_code': 200, 'json_resp': {'scans': {'Bkav': {'detected': True, 'version': '1.3.0.9899', 'result': 'W32.AIDetectVM.malware1'...}}

file_rescan(*resource)

This function rescan given files.

Parameters

*resource –

The resource argument can be the MD5, SHA-1 or SHA-256 of the file you want to re-scan.

Returns

Response

Return type

dict

Examples

>>> from clx.osi.virus_total import VirusTotalClient
>>> client = VirusTotalClient(api_key='your-api-key')
>>> client.file_rescan('70c0942965354dbb132c05458866b96709e37f44')
{'status_code': 200, 'json_resp': {'scan_id': ...}}

file_scan(file)

This function allows you to send a file for scanning with VirusTotal. Before performing submissions it would be nice to retrieve the latest report on the file. File size limit is 32MB, in order to submit files up to 200MB in size it is mandatory to use scan_big_file feature

Parameters: file (str) – File to be scanned
Returns: Response
Return type: dict

Examples

>>> from clx.osi.virus_total import VirusTotalClient
>>> client = VirusTotalClient(api_key='your-api-key')
>>> client.file_scan('test.sh')
{'status_code': 200, 'json_resp': {'scan_id': '0204e88255a0bd7807547e9186621f0478a6bb2c43e795fb5e6934e5cda0e1f6-1605914572', 'sha1': '70c0942965354dbb132c05458866b96709e37f44'...}

ipaddress_report(ip)

Retrieve report using ip address.

Parameters: ip (str) – An IP address
Returns: Response
Return type: dict

Examples

>>> from clx.osi.virus_total import VirusTotalClient
>>> client = VirusTotalClient(api_key='your-api-key')
>>> client.ipaddress_report("90.156.201.27")
{'status_code': 200, 'json_resp': {'asn': 25532, 'undetected_urls...}}

put_comment(resource, comment)

Post comment for a file or URL

Parameters: resource (str) – Either an md5/sha1/sha256 hash of the file you want to review or the URL itself that you want to comment on.
Returns: Response
Return type: dict

Examples

>>> from clx.osi.virus_total import VirusTotalClient
>>> client = VirusTotalClient(api_key='your-api-key')
>>> client.put_comment("75efd85cf6f8a962fe016787a7f57206ea9263086ee496fc62e3fc56734d4b53", "This is a test comment")
{'status_code': 200, 'json_resp': {'response_code': 0, 'verbose_msg': 'Duplicate comment'}}

scan_big_file(files)

Scanning files larger than 32MB

Parameters: file (str) – File to be scanned
Returns: Response
Return type: dict

Examples

>>> from clx.osi.virus_total import VirusTotalClient
>>> client = VirusTotalClient(api_key='your-api-key')
>>> client.scan_big_file('test.sh')
{'status_code': 200, 'json_resp': {'scan_id': '0204e88255a0bd7807547e9186621f0478a6bb2c43e795fb5e6934e5cda0e1f6-1605914572', 'sha1': '70c0942965354dbb132c05458866b96709e37f44'...}

url_report(*resource)

Retrieve URL scan reports

Parameters

*resource –

The resource argument must be the URL to retrieve the most recent report.

Returns

Response

Return type

dict

Examples

>>> from clx.osi.virus_total import VirusTotalClient
>>> client = VirusTotalClient(api_key='your-api-key')
>>> client.url_report(["virustotal.com"])
{'status_code': 200, 'json_resp': {'scan_id': 'a354494a73382ea0b4bc47f4c9e8d6c578027cd4598196dc88f05a22b5817293-1605914280'...}

url_scan(*url)

Retrieve URL scan reports

Parameters

*url –

A URL for which you want to retrieve the most recent report. You may also specify a scan_id (sha256-timestamp as returned by the URL submission API) to access a specific report.

Returns

Response

Return type

dict

Examples

>>> from clx.osi.virus_total import VirusTotalClient
>>> client = VirusTotalClient(api_key='your-api-key')
>>> client.url_scan(["virustotal.com"])
{'status_code': 200, 'json_resp': {'permalink': 'https://www.virustotal.com/gui/url/...}}

class clx.osi.whois.WhoIsLookupClient(sep=',', datetime_format='%m-%d-%Y %H:%M:%S')

Methods

whois(domains[, arr2str])

Function to access parsed WhoIs data for a given domain.

datetime_arr_keys = ['creation_date', 'updated_date', 'expiration_date']

Wrapper class to query WhoIs API.

Parameters

sep – Delimiter to concat nested list values from the Whois response.
datetime_format – Format to convert WhoIs response datetime object.

whois(domains, arr2str=True)

Function to access parsed WhoIs data for a given domain.

Parameters

domains (list) – Domains to perform whois lookup.
arr2str (boolean) – Convert WhoIs lookup response object to list of strings.

Returns

WhoIs information with respect to given domains.

Return type

list/obj

Examples

>>> from clx.osi.whois import WhoIsLookupClient
>>> domains = ["nvidia.com"]
>>> client = WhoIsLookupClient()
>>> client.whois(domains)
[{'domain_name': 'NVIDIA.COM', 'registrar': 'Safenames Ltd', 'whois_server': 'whois.safenames.net'...}]

class clx.osi.slashnext.SlashNextClient(api_key, snx_ir_workspace, base_url='https://oti.slashnext.cloud/api')

Attributes

conn

Methods

`api_quota`()	Find information about your API quota, like current usage, quota left etc.
`download_html`(scanid)	Downloads a web page HTML against a previous URL scan request.
`download_screenshot`(scanid[, resolution])	Downloads a screenshot of a web page against a previous URL scan request.
`download_text`(scanid)	Downloads the text of a web page against a previous URL scan request.
`host_report`(host)	Queries the SlashNext cloud database and retrieves a detailed report.
`host_reputation`(host)	Queries the SlashNext cloud database and retrieves the reputation of a host.
`host_urls`(host[, limit])	Queries the SlashNext cloud database and retrieves a list of all URLs.
`scan_report`(scanid[, extended_info])	Retrieve URL scan results against a previous scan request.
`url_scan`(url[, extended_info])	Perform a real-time URL reputation scan with SlashNext cloud-based SEER threat detection engine.
`url_scan_sync`(url[, extended_info, timeout])	Perform a real-time URL scan with SlashNext cloud-based SEER threat detection engine in a blocking mode.
`verify_connection`()	Verify SlashNext cloud database connection.

api_quota()

Find information about your API quota, like current usage, quota left etc.

:return Query response as list. :rtype: list

Examples

>>> from clx.osi.slashnext import SlashNextClient
>>> api_key = 'slashnext_cloud_apikey'
>>> snx_ir_workspace_dir = 'snx_ir_workspace'
>>> slashnext = SlashNextClient(api_key, snx_ir_workspace_dir)
>>> response_list = slashnext.api_quota()
>>> type(response_list[0])
<class 'dict'>

download_html(scanid)

Downloads a web page HTML against a previous URL scan request.

Parameters: scanid (str) – Scan ID of the scan for which to get the report. Can be retrieved from the “slashnext-url-scan” action or “slashnext-url-scan-sync” action.

:return Query response as list. :rtype: list

Examples

>>> from clx.osi.slashnext import SlashNextClient
>>> api_key = 'slashnext_cloud_apikey'
>>> snx_ir_workspace_dir = 'snx_ir_workspace'
>>> slashnext = SlashNextClient(api_key, snx_ir_workspace_dir)
>>> response_list = slashnext.download_html('2-ba57-755a7458c8a3')
>>> type(response_list[0])
<class 'dict'>

download_screenshot(scanid, resolution='high')

Downloads a screenshot of a web page against a previous URL scan request.

Parameters

scanid (str) – Scan ID of the scan for which to get the report. Can be retrieved from the “slashnext-url-scan” action or “slashnext-url-scan-sync” action.
resolution (str) – Resolution of the web page screenshot. Can be “high” or “medium”. Default is “high”.

:return Query response as list. :rtype: list

Examples

>>> from clx.osi.slashnext import SlashNextClient
>>> api_key = 'slashnext_cloud_apikey'
>>> snx_ir_workspace_dir = 'snx_ir_workspace'
>>> slashnext = SlashNextClient(api_key, snx_ir_workspace_dir)
>>> response_list = slashnext.download_screenshot('2-ba57-755a7458c8a3')
>>> type(response_list[0])
<class 'dict'>

download_text(scanid)

Downloads the text of a web page against a previous URL scan request.

Parameters: scanid (str) – Scan ID of the scan for which to get the report. Can be retrieved from the “slashnext-url-scan” action or “slashnext-url-scan-sync” action.

:return Query response as list. :rtype: list

Examples

>>> from clx.osi.slashnext import SlashNextClient
>>> api_key = 'slashnext_cloud_apikey'
>>> snx_ir_workspace_dir = 'snx_ir_workspace'
>>> slashnext = SlashNextClient(api_key, snx_ir_workspace_dir)
>>> response_list = slashnext.download_text('2-ba57-755a7458c8a3')
>>> type(response_list[0])
<class 'dict'>

host_report(host)

Queries the SlashNext cloud database and retrieves a detailed report.

Parameters: host (str) – The host to look up in the SlashNext Threat Intelligence database. Can be either a domain name or an IPv4 address.

:return Query response as list. :rtype: list

Examples

>>> from clx.osi.slashnext import SlashNextClient
>>> api_key = 'slashnext_cloud_apikey'
>>> snx_ir_workspace_dir = 'snx_ir_workspace'
>>> slashnext = SlashNextClient(api_key, snx_ir_workspace_dir)
>>> response_list = slashnext.host_report('google.com')
>>> type(response_list[0])
<class 'dict'>

host_reputation(host)

Queries the SlashNext cloud database and retrieves the reputation of a host.

Parameters: host (str) – The host to look up in the SlashNext Threat Intelligence database. Can be either a domain name or an IPv4 address.

:return Query response as list. :rtype: list

Examples

>>> from clx.osi.slashnext import SlashNextClient
>>> api_key = 'slashnext_cloud_apikey'
>>> snx_ir_workspace_dir = 'snx_ir_workspace'
>>> slashnext = SlashNextClient(api_key, snx_ir_workspace_dir)
>>> response_list = slashnext.host_reputation('google.com')
>>> type(response_list[0])
<class 'dict'>

host_urls(host, limit=10)

Queries the SlashNext cloud database and retrieves a list of all URLs.

Parameters

host (str) – The host to look up in the SlashNext Threat Intelligence database, for which to return a list of associated URLs. Can be either a domain name or an IPv4 address.
limit (int) – The maximum number of URL records to fetch. Default is “10”.

:return Query response as list. :rtype: list

Examples

>>> from clx.osi.slashnext import SlashNextClient
>>> api_key = 'slashnext_cloud_apikey'
>>> snx_ir_workspace_dir = 'snx_ir_workspace'
>>> slashnext = SlashNextClient(api_key, snx_ir_workspace_dir)
>>> response_list = slashnext.host_urls('google.com', limit=1)
>>> type(response_list[0])
<class 'dict'>

scan_report(scanid, extended_info=True)

Retrieve URL scan results against a previous scan request.

Parameters

scanid (str) – Scan ID of the scan for which to get the report. Can be retrieved from the “slashnext-url-scan” action or “slashnext-url-scan-sync” action.
extended_info (boolean) – Whether to download forensics data, such as screenshot, HTML, and rendered text.

:return Query response as list. :rtype: list

Examples

>>> from clx.osi.slashnext import SlashNextClient
>>> api_key = 'slashnext_cloud_apikey'
>>> snx_ir_workspace_dir = 'snx_ir_workspace'
>>> slashnext = SlashNextClient(api_key, snx_ir_workspace_dir)
>>> response_list = slashnext.scan_report('2-ba57-755a7458c8a3', extended_info=False)
>>> type(response_list[0])
<class 'dict'>

url_scan(url, extended_info=True)

Perform a real-time URL reputation scan with SlashNext cloud-based SEER threat detection engine.

Parameters

url (str) – The URL that needs to be scanned.
extended_info (boolean) – Whether to download forensics data, such as screenshot, HTML, and rendered text.

:return Query response as list. :rtype: list

Examples

>>> from clx.osi.slashnext import SlashNextClient
>>> api_key = 'slashnext_cloud_apikey'
>>> snx_ir_workspace_dir = 'snx_ir_workspace'
>>> slashnext = SlashNextClient(api_key, snx_ir_workspace_dir)
>>> response_list = slashnext.url_scan('http://ajeetenterprises.in/js/kbrad/drive/index.php', extended_info=False)
>>> type(response_list[0])
<class 'dict'>

url_scan_sync(url, extended_info=True, timeout=60)

Perform a real-time URL scan with SlashNext cloud-based SEER threat detection engine in a blocking mode.

Parameters

url (str) – The URL that needs to be scanned.
extended_info (boolean) – Whether to download forensics data, such as screenshot, HTML, and rendered text.
timeout (int) – A timeout value in seconds. If no timeout value is specified, a default timeout value is 60 seconds.

:return Query response as list. :rtype: list

Examples

>>> from clx.osi.slashnext import SlashNextClient
>>> api_key = 'slashnext_cloud_apikey'
>>> snx_ir_workspace_dir = 'snx_ir_workspace'
>>> slashnext = SlashNextClient(api_key, snx_ir_workspace_dir)
>>> response_list = slashnext.url_scan_sync('http://ajeetenterprises.in/js/kbrad/drive/index.php', extended_info=False, timeout=10)
>>> type(response_list[0])
<class 'dict'>

verify_connection()

Verify SlashNext cloud database connection.

Examples

>>> from clx.osi.slashnext import SlashNextClient
>>> api_key = 'slashnext_cloud_apikey'
>>> snx_ir_workspace_dir = 'snx_ir_workspace'
>>> slashnext = SlashNextClient(api_key, snx_ir_workspace_dir)
>>> slashnext.verify_connection()
Successfully connected to SlashNext cloud.
'success'

Parsers

class clx.parsers.event_parser.EventParser(columns, event_name)

This is an abstract class for all event log parsers.

Parameters

columns (set(string)) – Event column names.
event_name (string) – Event name

Attributes

columns: List of columns that are being processed.
event_name: Event name define type of logs that are being processed.

Methods

`filter_by_pattern`(df, column, pattern)	Retrieve events only which satisfies given regex pattern.
`parse`(dataframe, raw_column)	Abstract method 'parse' triggers the parsing functionality.
`parse_raw_event`(dataframe, raw_column, ...)	Processes parsing of a specific type of raw event records received as a dataframe.

property columns

List of columns that are being processed.

Returns: Event column names.
Return type: set(string)

property event_name

Event name define type of logs that are being processed.

Returns: Event name
Return type: string

filter_by_pattern(df, column, pattern)

Retrieve events only which satisfies given regex pattern.

Parameters

df (cudf.DataFrame) – Raw events to be filtered.
column (string) – Raw data contained column name.
pattern (string) – Regex pattern to retrieve events that are required.

Returns

filtered dataframe.

Return type

cudf.DataFrame

abstract parse(dataframe, raw_column): Abstract method ‘parse’ triggers the parsing functionality. Subclasses are required to implement and execute any parsing pre-processing steps.

parse_raw_event(dataframe, raw_column, event_regex)

Processes parsing of a specific type of raw event records received as a dataframe.

Parameters

dataframe (cudf.DataFrame) – Raw events to be parsed.
raw_column (string) – Raw data contained column name.
event_regex (dict) – Required regular expressions for a given event type.

Returns

parsed information.

Return type

cudf.DataFrame

class clx.parsers.splunk_notable_parser.SplunkNotableParser

This is class parses splunk notable logs.

Methods

parse(dataframe, raw_column)

Parses the Splunk notable raw events.

parse(dataframe, raw_column)

Parses the Splunk notable raw events.

Parameters

dataframe (cudf.DataFrame) – Raw events to be parsed.
raw_column (string) – Raw data contained column name.

Returns

parsed information.

Return type

cudf.DataFrame

class clx.parsers.windows_event_parser.WindowsEventParser(interested_eventcodes=None)

This is class parses windows event logs.

Parameters: interested_eventcodes (set(int)) – This parameter provides flexibility to parse only interested eventcodes.

Methods

`clean_raw_data`(dataframe, raw_column)	Lower casing and replacing escape characters.
`get_columns`()	Get columns of windows event codes.
`parse`(dataframe, raw_column)	Parses the Windows raw event.

clean_raw_data(dataframe, raw_column)

Lower casing and replacing escape characters.

Parameters

dataframe (cudf.DataFrame) – Raw events to be parsed.
raw_column (string) – Raw data contained column name.

Returns

Clean raw information.

Return type

cudf.DataFrame

get_columns()

Get columns of windows event codes.

Returns: Columns of all configured eventcodes, if no interested eventcodes specified.
Return type: set(string)

parse(dataframe, raw_column)

Parses the Windows raw event.

Parameters

dataframe (cudf.DataFrame) – Raw events to be parsed.
raw_column (string) – Raw data contained column name.

Returns

Parsed information.

Return type

cudf.DataFrame

clx.parsers.zeek.parse_log_file(filepath)

Parse Zeek log file and return cuDF dataframe. Uses header comments to get column names/types and configure parser.

Parameters: filepath (string) – filepath for Zeek log file
Returns: Zeek log dataframe
Return type: cudf.DataFrame

Utils

class clx.utils.data.dataloader.DataLoader(dataset, batchsize=1000)

Wrapper class is used to return dataframe partitions based on batchsize.

Attributes

dataset
dataset_len

Methods

get_chunks()

A generator function that yields each chunk of original input dataframe based on batchsize :return: Partitioned dataframe.

get_chunks(): A generator function that yields each chunk of original input dataframe based on batchsize :return: Partitioned dataframe. :rtype: cudf.DataFrame

class clx.utils.data.dataset.Dataset(df)

Attributes

data: Retruns dataframe
length: Returns dataframe length

property data: Retruns dataframe

property length: Returns dataframe length

clx.utils.data.utils: alias of <module ‘clx.utils.data.utils’ from ‘/opt/conda/envs/rapids/lib/python3.9/site-packages/clx-22.12.0a0+1.g10ffcc3-py3.9.egg/clx/utils/data/utils.py’>

Workflow

class clx.workflow.workflow.Workflow(name, source=None, destination=None)

Attributes

destination: Dictionary of configuration parameters for the data destination (writer)
name: Name of the workflow for logging purposes.
source: Dictionary of configuration parameters for the data source (reader)

Methods

`benchmark`()	Decorator used to capture a benchmark for a given function
`run_workflow`()	Run workflow.
`set_destination`(destination)	Set destination.
`set_source`(source)	Set source.
`stop_workflow`()	Close workflow.
`workflow`(dataframe)	The pipeline function performs the data enrichment on the data.

benchmark(): Decorator used to capture a benchmark for a given function

property destination: Dictionary of configuration parameters for the data destination (writer)

property name: Name of the workflow for logging purposes.

run_workflow(): Run workflow. Reader (source) fetches data. Workflow implementation is executed. Workflow output is written to destination.

set_destination(destination)

Set destination.

Parameters: destination – dict of configuration parameters for the destination (writer)

set_source(source)

Set source.

Parameters: source – dict of configuration parameters for data source (reader)

property source: Dictionary of configuration parameters for the data source (reader)

stop_workflow(): Close workflow. This includes calling close() method on reader (source) and writer (destination)

abstract workflow(dataframe): The pipeline function performs the data enrichment on the data. Subclasses must define this function. This function will return a gpu dataframe with enriched data.

class clx.workflow.splunk_alert_workflow.SplunkAlertWorkflow(name, source=None, destination=None, interval='day', threshold=2.5, window=7, raw_data_col_name='_raw')

Attributes

interval: Interval can be set to day or hour by which z score will be calculated
raw_data_col_name: Dataframe column name containing raw splunk alert data
threshold: Threshold by which to flag z score.
window: Window by which to calculate rolling z score

Methods

workflow(dataframe)

The pipeline function performs the data enrichment on the data.

property interval: Interval can be set to day or hour by which z score will be calculated

property raw_data_col_name: Dataframe column name containing raw splunk alert data

property threshold: Threshold by which to flag z score. Threshold will be flagged for scores >threshold or <-threshold

property window: Window by which to calculate rolling z score

workflow(dataframe): The pipeline function performs the data enrichment on the data. Subclasses must define this function. This function will return a gpu dataframe with enriched data.

I/O

class clx.io.reader.kafka_reader.KafkaReader(batch_size, consumer, time_window=30)

Reads from Kafka based on config object.

Parameters

batch_size – batch size
consumer – Kafka consumer
time_window – Max window of time that queued events will wait to be pushed to workflow

Attributes

consumer
has_data
time_window

Methods

`close`()	Close Kafka reader
`fetch_data`()	Fetch data from Kafka based on provided config object

close(): Close Kafka reader

fetch_data(): Fetch data from Kafka based on provided config object

class clx.io.reader.dask_fs_reader.DaskFileSystemReader(config)

Uses Dask to read from file system based on config object.

Parameters: config – dictionary object of config values for type, input_format, input_path, and dask reader optional keyword args

Methods

`close`()	Close dask reader
`fetch_data`()	Fetch data using dask based on provided config object

close(): Close dask reader

fetch_data(): Fetch data using dask based on provided config object

class clx.io.reader.fs_reader.FileSystemReader(config)

Uses cudf to read from file system based on config object.

Parameters: config – dictionary object of config values for type, input_format, input_path (or output_path), and cudf reader optional keyword args

Methods

`close`()	Close cudf reader
`fetch_data`()	Fetch data using cudf based on provided config object

close(): Close cudf reader

fetch_data(): Fetch data using cudf based on provided config object

class clx.io.writer.kafka_writer.KafkaWriter(kafka_topic, batch_size, delimiter, producer)

Publish to Kafka topic based on config object.

Parameters

kafka_topic – Kafka topic
batch_size – batch size
delimiter – delimiter
producer – producer

Attributes

delimiter
producer

Methods

`close`()	Close Kafka writer
`write_data`(df)	publish messages to kafka topic

close(): Close Kafka writer

write_data(df)

publish messages to kafka topic

Parameters: df – dataframe to publish

class clx.io.writer.fs_writer.FileSystemWriter(config)

Uses cudf to write to file system based on config object.

Parameters: config – dictionary object of config values for type, output_format, output_path (or output_path), and cudf writer optional keyword args

Methods

`close`()	Close cudf writer
`write_data`(df)	Write data to file system using cudf based on provided config object

close(): Close cudf writer

write_data(df): Write data to file system using cudf based on provided config object