Pickling cuML Models for Persistence¶
This notebook demonstrates simple pickling of both single-GPU and multi-GPU cuML models for persistence
[1]:
import warnings
warnings.filterwarnings("ignore", category=FutureWarning)
Single GPU Model Pickling¶
All single-GPU estimators are pickleable. The following example demonstrates the creation of a synthetic dataset, training, and pickling of the resulting model for storage. Trained single-GPU models can also be used to distribute the inference on a Dask cluster, which the Distributed Model Pickling section below demonstrates.
[2]:
from cuml.datasets import make_blobs
X, y = make_blobs(n_samples=50,
n_features=10,
centers=5,
cluster_std=0.4,
random_state=0)
[3]:
from cuml.cluster import KMeans
model = KMeans(n_clusters=5)
model.fit(X)
[3]:
KMeans()
[4]:
import pickle
pickle.dump(model, open("kmeans_model.pkl", "wb"))
[5]:
model = pickle.load(open("kmeans_model.pkl", "rb"))
[6]:
model.cluster_centers_
[6]:
array([[ 4.6749854 , 8.213466 , -9.075721 , 9.568374 , 8.454808 ,
-1.2327975 , 3.3903713 , -7.828242 , -0.8454461 , 0.6288572 ],
[-4.2439985 , 5.6107073 , -5.669777 , -1.7957245 , -9.255528 ,
0.71774375, 4.4435906 , -2.8747153 , -5.0900965 , 9.684121 ],
[ 5.2615476 , -4.0487256 , 4.464928 , -2.9367518 , 3.5061095 ,
-4.0168314 , -3.463885 , 6.0784483 , -6.953326 , -1.004144 ],
[-3.0082612 , 4.625961 , -4.483249 , 2.228457 , 1.643532 ,
-2.4505193 , -5.258201 , -1.6679404 , -7.985754 , 2.8311472 ],
[-5.6072407 , 2.2695985 , -3.7516537 , -1.8182003 , -5.143028 ,
7.599363 , 2.8252366 , 8.773042 , 1.6198314 , 1.1772048 ]],
dtype=float32)
Distributed Model Pickling¶
The distributed estimator wrappers inside of the cuml.dask are not intended to be pickled directly. The Dask cuML estimators provide a function get_combined_model(), which returns the trained single-GPU model for pickling. The combined model can be used for inference on a single-GPU, and the ParallelPostFit wrapper from the Dask-ML library can be used to perform distributed inference on a Dask cluster.
[7]:
from dask.distributed import Client
from dask_cuda import LocalCUDACluster
cluster = LocalCUDACluster()
client = Client(cluster)
client
2022-05-27 10:40:19,255 - distributed.preloading - INFO - Import preload module: dask_cuda.initialize
[7]:
Client
Client-67888780-dda9-11ec-8da3-2f31cbf4fb68
| Connection method: Cluster object | Cluster type: dask_cuda.LocalCUDACluster |
| Dashboard: http://127.0.0.1:8787/status |
Cluster Info
LocalCUDACluster
0e400a26
| Dashboard: http://127.0.0.1:8787/status | Workers: 1 |
| Total threads: 1 | Total memory: 251.79 GiB |
| Status: running | Using processes: True |
Scheduler Info
Scheduler
Scheduler-d964498f-0799-4d8c-832e-489fac0bd1f3
| Comm: tcp://127.0.0.1:37205 | Workers: 1 |
| Dashboard: http://127.0.0.1:8787/status | Total threads: 1 |
| Started: Just now | Total memory: 251.79 GiB |
Workers
Worker: 0
| Comm: tcp://127.0.0.1:40813 | Total threads: 1 |
| Dashboard: http://127.0.0.1:38813/status | Memory: 251.79 GiB |
| Nanny: tcp://127.0.0.1:37531 | |
| Local directory: /rapids/cuml/docs/source/dask-worker-space/worker-ismngvqv | |
| GPU: Tesla V100-PCIE-32GB | GPU memory: 31.75 GiB |
[8]:
from cuml.dask.datasets import make_blobs
n_workers = len(client.scheduler_info()["workers"].keys())
X, y = make_blobs(n_samples=5000,
n_features=30,
centers=5,
cluster_std=0.4,
random_state=0,
n_parts=n_workers*5)
X = X.persist()
y = y.persist()
[9]:
from cuml.dask.cluster import KMeans
dist_model = KMeans(n_clusters=5)
[10]:
dist_model.fit(X)
2022-05-27 10:40:25,611 - distributed.worker - WARNING - Compute Failed
Key: _get_model_attr-bf119129-b5dd-49b7-a90c-41b170696865
Function: _get_model_attr
args: (KMeansMG(), '_ipython_canary_method_should_not_exist_')
kwargs: {}
Exception: 'AttributeError("Attribute _ipython_canary_method_should_not_exist_ does not exist on model <class \'cuml.cluster.kmeans_mg.KMeansMG\'>")'
2022-05-27 10:40:25,621 - distributed.worker - WARNING - Compute Failed
Key: _get_model_attr-cb59f993-4b2b-4d10-92d2-f6e9e1195aaf
Function: _get_model_attr
args: (KMeansMG(), '_ipython_display_')
kwargs: {}
Exception: 'AttributeError("Attribute _ipython_display_ does not exist on model <class \'cuml.cluster.kmeans_mg.KMeansMG\'>")'
2022-05-27 10:40:25,633 - distributed.worker - WARNING - Compute Failed
Key: _get_model_attr-2e20bb9c-96b3-4b85-85c2-d6a04444e13a
Function: _get_model_attr
args: (KMeansMG(), '_ipython_canary_method_should_not_exist_')
kwargs: {}
Exception: 'AttributeError("Attribute _ipython_canary_method_should_not_exist_ does not exist on model <class \'cuml.cluster.kmeans_mg.KMeansMG\'>")'
2022-05-27 10:40:25,644 - distributed.worker - WARNING - Compute Failed
Key: _get_model_attr-530c412a-d729-46b9-b677-d21f1ac97d65
Function: _get_model_attr
args: (KMeansMG(), '_repr_mimebundle_')
kwargs: {}
Exception: 'AttributeError("Attribute _repr_mimebundle_ does not exist on model <class \'cuml.cluster.kmeans_mg.KMeansMG\'>")'
2022-05-27 10:40:25,658 - distributed.worker - WARNING - Compute Failed
Key: _get_model_attr-7af8e8e4-4df3-4ff7-8562-8c05556e16f5
Function: _get_model_attr
args: (KMeansMG(), '_ipython_canary_method_should_not_exist_')
kwargs: {}
Exception: 'AttributeError("Attribute _ipython_canary_method_should_not_exist_ does not exist on model <class \'cuml.cluster.kmeans_mg.KMeansMG\'>")'
2022-05-27 10:40:25,681 - distributed.worker - WARNING - Compute Failed
Key: _get_model_attr-adfb5f11-e8a7-44a7-b05d-2e6fdd59c453
Function: _get_model_attr
args: (KMeansMG(), '_repr_html_')
kwargs: {}
Exception: 'AttributeError("Attribute _repr_html_ does not exist on model <class \'cuml.cluster.kmeans_mg.KMeansMG\'>")'
2022-05-27 10:40:25,700 - distributed.worker - WARNING - Compute Failed
Key: _get_model_attr-d3fbbc3a-551d-4670-be77-aef3534190cc
Function: _get_model_attr
args: (KMeansMG(), '_ipython_canary_method_should_not_exist_')
kwargs: {}
Exception: 'AttributeError("Attribute _ipython_canary_method_should_not_exist_ does not exist on model <class \'cuml.cluster.kmeans_mg.KMeansMG\'>")'
2022-05-27 10:40:25,713 - distributed.worker - WARNING - Compute Failed
Key: _get_model_attr-e0cea3c3-55d7-416e-9221-3cf95f0da948
Function: _get_model_attr
args: (KMeansMG(), '_repr_markdown_')
kwargs: {}
Exception: 'AttributeError("Attribute _repr_markdown_ does not exist on model <class \'cuml.cluster.kmeans_mg.KMeansMG\'>")'
2022-05-27 10:40:25,725 - distributed.worker - WARNING - Compute Failed
Key: _get_model_attr-2a73a5a1-430b-44c1-b2fe-b6b0e9220458
Function: _get_model_attr
args: (KMeansMG(), '_ipython_canary_method_should_not_exist_')
kwargs: {}
Exception: 'AttributeError("Attribute _ipython_canary_method_should_not_exist_ does not exist on model <class \'cuml.cluster.kmeans_mg.KMeansMG\'>")'
2022-05-27 10:40:25,744 - distributed.worker - WARNING - Compute Failed
Key: _get_model_attr-b8f55d6e-7958-4502-9bdf-fad4fa5cc5b6
Function: _get_model_attr
args: (KMeansMG(), '_repr_svg_')
kwargs: {}
Exception: 'AttributeError("Attribute _repr_svg_ does not exist on model <class \'cuml.cluster.kmeans_mg.KMeansMG\'>")'
2022-05-27 10:40:25,764 - distributed.worker - WARNING - Compute Failed
Key: _get_model_attr-cd8d1ea0-cd9a-4af5-9dff-c5dae59ed993
Function: _get_model_attr
args: (KMeansMG(), '_ipython_canary_method_should_not_exist_')
kwargs: {}
Exception: 'AttributeError("Attribute _ipython_canary_method_should_not_exist_ does not exist on model <class \'cuml.cluster.kmeans_mg.KMeansMG\'>")'
2022-05-27 10:40:25,777 - distributed.worker - WARNING - Compute Failed
Key: _get_model_attr-9602b9d9-e2a5-4210-b533-7d202fae3b73
Function: _get_model_attr
args: (KMeansMG(), '_repr_png_')
kwargs: {}
Exception: 'AttributeError("Attribute _repr_png_ does not exist on model <class \'cuml.cluster.kmeans_mg.KMeansMG\'>")'
2022-05-27 10:40:25,791 - distributed.worker - WARNING - Compute Failed
Key: _get_model_attr-af04bd1f-74c0-4199-8ce6-d879b039429e
Function: _get_model_attr
args: (KMeansMG(), '_ipython_canary_method_should_not_exist_')
kwargs: {}
Exception: 'AttributeError("Attribute _ipython_canary_method_should_not_exist_ does not exist on model <class \'cuml.cluster.kmeans_mg.KMeansMG\'>")'
2022-05-27 10:40:25,804 - distributed.worker - WARNING - Compute Failed
Key: _get_model_attr-5f8f5955-d65b-4f5c-bb6c-f5568bef46bc
Function: _get_model_attr
args: (KMeansMG(), '_repr_pdf_')
kwargs: {}
Exception: 'AttributeError("Attribute _repr_pdf_ does not exist on model <class \'cuml.cluster.kmeans_mg.KMeansMG\'>")'
2022-05-27 10:40:25,817 - distributed.worker - WARNING - Compute Failed
Key: _get_model_attr-6603971f-995d-4561-9f93-6caa171f3670
Function: _get_model_attr
args: (KMeansMG(), '_ipython_canary_method_should_not_exist_')
kwargs: {}
Exception: 'AttributeError("Attribute _ipython_canary_method_should_not_exist_ does not exist on model <class \'cuml.cluster.kmeans_mg.KMeansMG\'>")'
2022-05-27 10:40:25,830 - distributed.worker - WARNING - Compute Failed
Key: _get_model_attr-fc55ee26-c3cd-4af8-b767-2a408c010ecb
Function: _get_model_attr
args: (KMeansMG(), '_repr_jpeg_')
kwargs: {}
Exception: 'AttributeError("Attribute _repr_jpeg_ does not exist on model <class \'cuml.cluster.kmeans_mg.KMeansMG\'>")'
2022-05-27 10:40:25,842 - distributed.worker - WARNING - Compute Failed
Key: _get_model_attr-f2d179df-8ccf-4582-9cfb-b27c60508ab2
Function: _get_model_attr
args: (KMeansMG(), '_ipython_canary_method_should_not_exist_')
kwargs: {}
Exception: 'AttributeError("Attribute _ipython_canary_method_should_not_exist_ does not exist on model <class \'cuml.cluster.kmeans_mg.KMeansMG\'>")'
2022-05-27 10:40:25,854 - distributed.worker - WARNING - Compute Failed
Key: _get_model_attr-b5cc8f5a-2ebb-4c63-b4d8-6dba23fc76cd
Function: _get_model_attr
args: (KMeansMG(), '_repr_latex_')
kwargs: {}
Exception: 'AttributeError("Attribute _repr_latex_ does not exist on model <class \'cuml.cluster.kmeans_mg.KMeansMG\'>")'
2022-05-27 10:40:25,868 - distributed.worker - WARNING - Compute Failed
Key: _get_model_attr-19bea344-61c2-480c-b5ae-60ca06bfb28b
Function: _get_model_attr
args: (KMeansMG(), '_ipython_canary_method_should_not_exist_')
kwargs: {}
Exception: 'AttributeError("Attribute _ipython_canary_method_should_not_exist_ does not exist on model <class \'cuml.cluster.kmeans_mg.KMeansMG\'>")'
2022-05-27 10:40:25,881 - distributed.worker - WARNING - Compute Failed
Key: _get_model_attr-751a7ee2-09df-4fc1-bcca-ac15ee0171c1
Function: _get_model_attr
args: (KMeansMG(), '_repr_json_')
kwargs: {}
Exception: 'AttributeError("Attribute _repr_json_ does not exist on model <class \'cuml.cluster.kmeans_mg.KMeansMG\'>")'
2022-05-27 10:40:25,894 - distributed.worker - WARNING - Compute Failed
Key: _get_model_attr-0091e7d5-d8df-470c-9e6f-0fb3e5f08a11
Function: _get_model_attr
args: (KMeansMG(), '_ipython_canary_method_should_not_exist_')
kwargs: {}
Exception: 'AttributeError("Attribute _ipython_canary_method_should_not_exist_ does not exist on model <class \'cuml.cluster.kmeans_mg.KMeansMG\'>")'
2022-05-27 10:40:25,906 - distributed.worker - WARNING - Compute Failed
Key: _get_model_attr-c63c1489-c6f9-423e-8c98-953dd7cd8e74
Function: _get_model_attr
args: (KMeansMG(), '_repr_javascript_')
kwargs: {}
Exception: 'AttributeError("Attribute _repr_javascript_ does not exist on model <class \'cuml.cluster.kmeans_mg.KMeansMG\'>")'
[10]:
<cuml.dask.cluster.kmeans.KMeans at 0x7fd2a430b3a0>
[11]:
import pickle
single_gpu_model = dist_model.get_combined_model()
pickle.dump(single_gpu_model, open("kmeans_model.pkl", "wb"))
[12]:
single_gpu_model = pickle.load(open("kmeans_model.pkl", "rb"))
[13]:
single_gpu_model.cluster_centers_
[13]:
array([[-4.6475406e+00, -9.5672169e+00, 6.6923518e+00, 4.4359236e+00,
2.1902733e+00, 2.5834751e+00, 5.9448928e-01, 6.2568665e+00,
-8.7821636e+00, -4.1232008e-01, 9.8151779e+00, 7.5641570e+00,
1.0003009e+01, -5.8680434e+00, -1.2743106e+00, -2.5393455e+00,
-1.0847499e+00, -5.2629199e+00, -9.3071022e+00, 4.6179357e+00,
-9.7068951e-02, -3.9351203e+00, 6.1767282e+00, -7.4346886e+00,
5.6496067e+00, -8.5544834e+00, -7.5265574e+00, -5.5195603e+00,
4.8197637e+00, 2.5235438e+00],
[ 6.2794304e+00, 9.2293949e+00, 8.3403702e+00, 9.0330496e+00,
7.6893487e+00, -9.9538219e-01, -6.2780766e+00, 1.3599335e+00,
-6.9744492e+00, -5.9463463e+00, 1.0695117e+00, -8.0422489e-03,
2.8183141e+00, 1.8317667e+00, -8.2557344e+00, 3.0514317e+00,
-8.4958019e+00, 9.7238483e+00, -7.7455091e+00, 3.4521689e+00,
-3.9248333e+00, -4.1106424e+00, 2.6693091e+00, 1.2985628e+00,
1.0421574e+00, 5.2490621e+00, -1.6496238e+00, 6.1451659e+00,
-6.9103327e+00, -9.6390314e+00],
[-2.8796387e+00, 4.4348760e+00, -4.4264803e+00, 2.3959141e+00,
1.7286435e+00, -2.4991984e+00, -5.1794524e+00, -1.6771443e+00,
-8.1329174e+00, 2.6659234e+00, -4.3131094e+00, 5.5827808e+00,
-5.7318316e+00, -1.7427169e+00, -9.3456116e+00, 7.1365565e-01,
4.4255495e+00, -2.9118381e+00, -4.9467440e+00, 9.6786423e+00,
8.4222746e+00, -6.2710247e+00, -6.3596506e+00, 1.9645509e+00,
4.1715999e+00, -9.1683636e+00, 4.6156683e+00, 8.7916498e+00,
6.8754420e+00, 2.2288749e+00],
[-6.9536943e+00, -9.7635880e+00, -6.5648251e+00, -4.3536150e-01,
6.0998111e+00, 3.7550368e+00, -3.9558539e+00, 6.1595516e+00,
-1.8599317e+00, 5.0400305e+00, -6.8397551e+00, 1.3435434e+00,
8.9749002e+00, -9.9621755e-01, 9.6651945e+00, 9.8009653e+00,
-8.6188345e+00, 5.9978366e+00, 2.2295930e+00, -3.6477709e+00,
7.0758505e+00, -7.3772254e+00, -5.3214226e+00, -6.9927959e+00,
-7.9296322e+00, 6.6705360e+00, -5.5850172e+00, 7.1526046e+00,
6.5703211e+00, -8.3389378e+00],
[ 4.8136687e+00, 8.3985281e+00, -9.2161245e+00, 9.4185524e+00,
8.5280886e+00, -1.0969981e+00, 3.3253176e+00, -7.8064375e+00,
-5.9660631e-01, 2.5423869e-01, 5.5004086e+00, -4.1162968e+00,
4.2832375e+00, -2.8173418e+00, 3.6207724e+00, -4.1576214e+00,
-3.6048706e+00, 6.2125406e+00, -6.9080997e+00, -1.0732135e+00,
-5.8362117e+00, 2.2357666e+00, -3.8588789e+00, -1.6835879e+00,
-5.3240361e+00, 7.5769200e+00, 2.9358530e+00, 8.5267372e+00,
1.5667248e+00, 1.0779675e+00]], dtype=float32)
Exporting cuML Random Forest models for inferencing on machines without GPUs¶
Starting with cuML version 21.06, you can export cuML Random Forest models and run predictions with them on machines without an NVIDIA GPUs. The Treelite package defines an efficient exchange format that lets you portably move the cuML Random Forest models to other machines. We will refer to the exchange format as “checkpoints.”
Here are the steps to export the model:
Call
to_treelite_checkpoint()to obtain the checkpoint file from the cuML Random Forest model.
[14]:
from cuml.ensemble import RandomForestClassifier as cumlRandomForestClassifier
from sklearn.datasets import load_iris
import numpy as np
X, y = load_iris(return_X_y=True)
X, y = X.astype(np.float32), y.astype(np.int32)
clf = cumlRandomForestClassifier(max_depth=3, random_state=0, n_estimators=10)
clf.fit(X, y)
checkpoint_path = './checkpoint.tl'
# Export cuML RF model as Treelite checkpoint
clf.convert_to_treelite_model().to_treelite_checkpoint(checkpoint_path)
/opt/conda/envs/rapids/lib/python3.9/site-packages/cuml/internals/api_decorators.py:794: UserWarning: For reproducible results in Random Forest Classifier or for almost reproducible results in Random Forest Regressor, n_streams=1 is recommended. If n_streams is > 1, results may vary due to stream/thread timing differences, even when random_state is set
return func(**kwargs)
Copy the generated checkpoint file
checkpoint.tlto another machine on which you’d like to run predictions.On the target machine, install Treelite by running
pip install treeliteorconda install -c conda-forge treelite. The machine does not need to have an NVIDIA GPUs and does not need to have cuML installed.You can now load the model from the checkpoint, by running the following on the target machine:
[15]:
import treelite
# The checkpoint file has been copied over
checkpoint_path = './checkpoint.tl'
tl_model = treelite.Model.deserialize(checkpoint_path)
out_prob = treelite.gtil.predict(tl_model, X, pred_margin=True)
print(out_prob)
[[1. 0. 0. ]
[1. 0. 0. ]
[1. 0. 0. ]
[1. 0. 0. ]
[1. 0. 0. ]
[1. 0. 0. ]
[1. 0. 0. ]
[1. 0. 0. ]
[1. 0. 0. ]
[1. 0. 0. ]
[1. 0. 0. ]
[1. 0. 0. ]
[1. 0. 0. ]
[1. 0. 0. ]
[1. 0. 0. ]
[1. 0. 0. ]
[1. 0. 0. ]
[1. 0. 0. ]
[0.9 0.1 0. ]
[1. 0. 0. ]
[1. 0. 0. ]
[1. 0. 0. ]
[1. 0. 0. ]
[1. 0. 0. ]
[1. 0. 0. ]
[1. 0. 0. ]
[1. 0. 0. ]
[1. 0. 0. ]
[1. 0. 0. ]
[1. 0. 0. ]
[1. 0. 0. ]
[1. 0. 0. ]
[1. 0. 0. ]
[1. 0. 0. ]
[1. 0. 0. ]
[1. 0. 0. ]
[1. 0. 0. ]
[1. 0. 0. ]
[1. 0. 0. ]
[1. 0. 0. ]
[1. 0. 0. ]
[1. 0. 0. ]
[1. 0. 0. ]
[1. 0. 0. ]
[1. 0. 0. ]
[1. 0. 0. ]
[1. 0. 0. ]
[1. 0. 0. ]
[1. 0. 0. ]
[1. 0. 0. ]
[0. 0.9981133 0.00188679]
[0. 0.9981133 0.00188679]
[0. 0.5274104 0.47258964]
[0. 0.9870021 0.0129979 ]
[0. 0.9981133 0.00188679]
[0. 0.9981133 0.00188679]
[0. 0.9981133 0.00188679]
[0. 0.7036687 0.29633126]
[0. 0.9981133 0.00188679]
[0. 0.9870021 0.0129979 ]
[0. 0.9536688 0.04633124]
[0. 0.9981133 0.00188679]
[0. 0.9981133 0.00188679]
[0. 0.9981133 0.00188679]
[0. 0.9981133 0.00188679]
[0. 0.9981133 0.00188679]
[0. 0.9981133 0.00188679]
[0. 0.9981133 0.00188679]
[0. 0.9981133 0.00188679]
[0. 0.9981133 0.00188679]
[0. 0.23634085 0.7636592 ]
[0. 0.9981133 0.00188679]
[0. 0.567213 0.432787 ]
[0. 0.9981133 0.00188679]
[0. 0.9981133 0.00188679]
[0. 0.9981133 0.00188679]
[0. 0.60401857 0.3959814 ]
[0. 0.41788664 0.5821134 ]
[0. 0.9981133 0.00188679]
[0. 0.9981133 0.00188679]
[0. 0.9870021 0.0129979 ]
[0. 0.9870021 0.0129979 ]
[0. 0.9981133 0.00188679]
[0. 0.30794033 0.6920597 ]
[0.1 0.89811325 0.00188679]
[0. 0.9981133 0.00188679]
[0. 0.9981133 0.00188679]
[0. 0.9981133 0.00188679]
[0. 0.9981133 0.00188679]
[0. 0.9870021 0.0129979 ]
[0. 0.9870021 0.0129979 ]
[0. 0.9981133 0.00188679]
[0. 0.9981133 0.00188679]
[0. 0.9536688 0.04633124]
[0. 0.9981133 0.00188679]
[0. 0.9981133 0.00188679]
[0. 0.9981133 0.00188679]
[0. 0.9981133 0.00188679]
[0. 0.9870021 0.0129979 ]
[0. 0.9981133 0.00188679]
[0. 0.01794118 0.9820588 ]
[0. 0.05149381 0.9485062 ]
[0. 0.01169118 0.9883088 ]
[0. 0.01169118 0.9883088 ]
[0. 0.01169118 0.9883088 ]
[0. 0.01169118 0.9883088 ]
[0. 0.5655735 0.4344265 ]
[0. 0.01169118 0.9883088 ]
[0. 0.05149381 0.9485062 ]
[0. 0.01794118 0.9820588 ]
[0. 0.01794118 0.9820588 ]
[0. 0.05149381 0.9485062 ]
[0. 0.01169118 0.9883088 ]
[0. 0.19290936 0.80709064]
[0. 0.03274381 0.9672562 ]
[0. 0.01794118 0.9820588 ]
[0. 0.01169118 0.9883088 ]
[0. 0.01794118 0.9820588 ]
[0. 0.05149381 0.9485062 ]
[0. 0.44935593 0.5506441 ]
[0. 0.01794118 0.9820588 ]
[0. 0.2170165 0.78298354]
[0. 0.03274381 0.9672562 ]
[0. 0.2357665 0.7642335 ]
[0. 0.01794118 0.9820588 ]
[0. 0.01794118 0.9820588 ]
[0. 0.2511435 0.74885654]
[0. 0.19596387 0.80403614]
[0. 0.03274381 0.9672562 ]
[0. 0.10335784 0.8966421 ]
[0. 0.03274381 0.9672562 ]
[0. 0.01794118 0.9820588 ]
[0. 0.03274381 0.9672562 ]
[0. 0.28919035 0.7108097 ]
[0. 0.20982714 0.79017293]
[0. 0.01169118 0.9883088 ]
[0. 0.01794118 0.9820588 ]
[0. 0.01169118 0.9883088 ]
[0. 0.23009086 0.7699092 ]
[0. 0.01169118 0.9883088 ]
[0. 0.01169118 0.9883088 ]
[0. 0.01169118 0.9883088 ]
[0. 0.05149381 0.9485062 ]
[0. 0.01794118 0.9820588 ]
[0. 0.01794118 0.9820588 ]
[0. 0.01169118 0.9883088 ]
[0. 0.19290936 0.80709064]
[0. 0.01169118 0.9883088 ]
[0. 0.01794118 0.9820588 ]
[0. 0.01169118 0.9883088 ]]