Pickling cuML Models for Persistence

This notebook demonstrates simple pickling of both single-GPU and multi-GPU cuML models for persistence

[1]:
import warnings
warnings.filterwarnings("ignore", category=FutureWarning)

Single GPU Model Pickling

All single-GPU estimators are pickleable. The following example demonstrates the creation of a synthetic dataset, training, and pickling of the resulting model for storage. Trained single-GPU models can also be used to distribute the inference on a Dask cluster, which the Distributed Model Pickling section below demonstrates.

[2]:
from cuml.datasets import make_blobs

X, y = make_blobs(n_samples=50,
                  n_features=10,
                  centers=5,
                  cluster_std=0.4,
                  random_state=0)
[3]:
from cuml.cluster import KMeans

model = KMeans(n_clusters=5)

model.fit(X)
[3]:
KMeans()
[4]:
import pickle

pickle.dump(model, open("kmeans_model.pkl", "wb"))
[5]:
model = pickle.load(open("kmeans_model.pkl", "rb"))
[6]:
model.cluster_centers_
[6]:
array([[ 4.6749854,  8.213465 , -9.075721 ,  9.568374 ,  8.454807 ,
        -1.2327974,  3.3903713, -7.828242 , -0.8454461,  0.6288572],
       [-4.2439985,  5.6107073, -5.669777 , -1.7957243, -9.255529 ,
         0.7177438,  4.4435906, -2.8747156, -5.0900965,  9.684121 ],
       [ 5.2615476, -4.0487256,  4.464928 , -2.9367518,  3.5061095,
        -4.0168314, -3.463885 ,  6.078449 , -6.953326 , -1.004144 ],
       [-3.008261 ,  4.6259604, -4.483249 ,  2.228457 ,  1.6435319,
        -2.4505196, -5.258201 , -1.6679403, -7.9857535,  2.8311467],
       [-5.6072407,  2.2695985, -3.7516537, -1.8182005, -5.1430273,
         7.599363 ,  2.8252368,  8.773042 ,  1.6198314,  1.1772048]],
      dtype=float32)

Distributed Model Pickling

The distributed estimator wrappers inside of the cuml.dask are not intended to be pickled directly. The Dask cuML estimators provide a function get_combined_model(), which returns the trained single-GPU model for pickling. The combined model can be used for inference on a single-GPU, and the ParallelPostFit wrapper from the Dask-ML library can be used to perform distributed inference on a Dask cluster.

[7]:
from dask.distributed import Client
from dask_cuda import LocalCUDACluster

cluster = LocalCUDACluster()
client = Client(cluster)
client
2022-08-18 11:14:50,730 - distributed.preloading - INFO - Creating preload: dask_cuda.initialize
2022-08-18 11:14:50,730 - distributed.preloading - INFO - Import preload module: dask_cuda.initialize
[7]:

Client

Client-fa70c34f-1ee6-11ed-9035-0242ac110007

Connection method: Cluster object Cluster type: dask_cuda.LocalCUDACluster
Dashboard: http://127.0.0.1:8787/status

Cluster Info

[8]:
from cuml.dask.datasets import make_blobs

n_workers = len(client.scheduler_info()["workers"].keys())

X, y = make_blobs(n_samples=5000,
                  n_features=30,
                  centers=5,
                  cluster_std=0.4,
                  random_state=0,
                  n_parts=n_workers*5)

X = X.persist()
y = y.persist()
[9]:
from cuml.dask.cluster import KMeans

dist_model = KMeans(n_clusters=5)
[10]:
dist_model.fit(X)
2022-08-18 11:14:56,943 - distributed.worker - WARNING - Compute Failed
Key:       _get_model_attr-db783c77-f73c-439a-b5aa-18eed94adc99
Function:  _get_model_attr
args:      (KMeansMG(), '_ipython_canary_method_should_not_exist_')
kwargs:    {}
Exception: 'AttributeError("Attribute _ipython_canary_method_should_not_exist_ does not exist on model <class \'cuml.cluster.kmeans_mg.KMeansMG\'>")'

2022-08-18 11:14:56,962 - distributed.worker - WARNING - Compute Failed
Key:       _get_model_attr-ac5c4d8c-b984-4a0d-901a-fc97b1e6a7cb
Function:  _get_model_attr
args:      (KMeansMG(), '_ipython_display_')
kwargs:    {}
Exception: 'AttributeError("Attribute _ipython_display_ does not exist on model <class \'cuml.cluster.kmeans_mg.KMeansMG\'>")'

2022-08-18 11:14:56,974 - distributed.worker - WARNING - Compute Failed
Key:       _get_model_attr-dbd745c0-2983-4680-af56-80ad5e7afc55
Function:  _get_model_attr
args:      (KMeansMG(), '_ipython_canary_method_should_not_exist_')
kwargs:    {}
Exception: 'AttributeError("Attribute _ipython_canary_method_should_not_exist_ does not exist on model <class \'cuml.cluster.kmeans_mg.KMeansMG\'>")'

2022-08-18 11:14:56,985 - distributed.worker - WARNING - Compute Failed
Key:       _get_model_attr-893b58fc-9f81-4f3f-9321-91f188aac609
Function:  _get_model_attr
args:      (KMeansMG(), '_repr_mimebundle_')
kwargs:    {}
Exception: 'AttributeError("Attribute _repr_mimebundle_ does not exist on model <class \'cuml.cluster.kmeans_mg.KMeansMG\'>")'

2022-08-18 11:14:57,007 - distributed.worker - WARNING - Compute Failed
Key:       _get_model_attr-5e03b254-43eb-4666-8fe2-28607dcf2120
Function:  _get_model_attr
args:      (KMeansMG(), '_ipython_canary_method_should_not_exist_')
kwargs:    {}
Exception: 'AttributeError("Attribute _ipython_canary_method_should_not_exist_ does not exist on model <class \'cuml.cluster.kmeans_mg.KMeansMG\'>")'

2022-08-18 11:14:57,019 - distributed.worker - WARNING - Compute Failed
Key:       _get_model_attr-9c9e099c-9e03-4a53-9ff3-cea5e771f5c8
Function:  _get_model_attr
args:      (KMeansMG(), '_repr_html_')
kwargs:    {}
Exception: 'AttributeError("Attribute _repr_html_ does not exist on model <class \'cuml.cluster.kmeans_mg.KMeansMG\'>")'

2022-08-18 11:14:57,030 - distributed.worker - WARNING - Compute Failed
Key:       _get_model_attr-117523f0-5719-45b5-9169-d5fd78a6217f
Function:  _get_model_attr
args:      (KMeansMG(), '_ipython_canary_method_should_not_exist_')
kwargs:    {}
Exception: 'AttributeError("Attribute _ipython_canary_method_should_not_exist_ does not exist on model <class \'cuml.cluster.kmeans_mg.KMeansMG\'>")'

2022-08-18 11:14:57,043 - distributed.worker - WARNING - Compute Failed
Key:       _get_model_attr-d6bec4b3-78be-4765-89cf-feb8d1b32a0f
Function:  _get_model_attr
args:      (KMeansMG(), '_repr_markdown_')
kwargs:    {}
Exception: 'AttributeError("Attribute _repr_markdown_ does not exist on model <class \'cuml.cluster.kmeans_mg.KMeansMG\'>")'

2022-08-18 11:14:57,054 - distributed.worker - WARNING - Compute Failed
Key:       _get_model_attr-b666e90e-7f08-4207-936a-76745327c13b
Function:  _get_model_attr
args:      (KMeansMG(), '_ipython_canary_method_should_not_exist_')
kwargs:    {}
Exception: 'AttributeError("Attribute _ipython_canary_method_should_not_exist_ does not exist on model <class \'cuml.cluster.kmeans_mg.KMeansMG\'>")'

2022-08-18 11:14:57,065 - distributed.worker - WARNING - Compute Failed
Key:       _get_model_attr-14873d39-1deb-424e-b2ee-936e1d9652a7
Function:  _get_model_attr
args:      (KMeansMG(), '_repr_svg_')
kwargs:    {}
Exception: 'AttributeError("Attribute _repr_svg_ does not exist on model <class \'cuml.cluster.kmeans_mg.KMeansMG\'>")'

2022-08-18 11:14:57,076 - distributed.worker - WARNING - Compute Failed
Key:       _get_model_attr-bdb2b29d-a2df-4e65-80ba-be669158aa26
Function:  _get_model_attr
args:      (KMeansMG(), '_ipython_canary_method_should_not_exist_')
kwargs:    {}
Exception: 'AttributeError("Attribute _ipython_canary_method_should_not_exist_ does not exist on model <class \'cuml.cluster.kmeans_mg.KMeansMG\'>")'

2022-08-18 11:14:57,087 - distributed.worker - WARNING - Compute Failed
Key:       _get_model_attr-5f065be5-252b-4abe-866f-3d35ab7b0de4
Function:  _get_model_attr
args:      (KMeansMG(), '_repr_png_')
kwargs:    {}
Exception: 'AttributeError("Attribute _repr_png_ does not exist on model <class \'cuml.cluster.kmeans_mg.KMeansMG\'>")'

2022-08-18 11:14:57,099 - distributed.worker - WARNING - Compute Failed
Key:       _get_model_attr-fac6631a-4e17-4f8f-85a6-17c865162ec4
Function:  _get_model_attr
args:      (KMeansMG(), '_ipython_canary_method_should_not_exist_')
kwargs:    {}
Exception: 'AttributeError("Attribute _ipython_canary_method_should_not_exist_ does not exist on model <class \'cuml.cluster.kmeans_mg.KMeansMG\'>")'

2022-08-18 11:14:57,112 - distributed.worker - WARNING - Compute Failed
Key:       _get_model_attr-604da8f6-5fa3-446f-a247-dd06854e34bf
Function:  _get_model_attr
args:      (KMeansMG(), '_repr_pdf_')
kwargs:    {}
Exception: 'AttributeError("Attribute _repr_pdf_ does not exist on model <class \'cuml.cluster.kmeans_mg.KMeansMG\'>")'

2022-08-18 11:14:57,127 - distributed.worker - WARNING - Compute Failed
Key:       _get_model_attr-7123826d-748f-4703-a019-1b0e3b421ca3
Function:  _get_model_attr
args:      (KMeansMG(), '_ipython_canary_method_should_not_exist_')
kwargs:    {}
Exception: 'AttributeError("Attribute _ipython_canary_method_should_not_exist_ does not exist on model <class \'cuml.cluster.kmeans_mg.KMeansMG\'>")'

2022-08-18 11:14:57,146 - distributed.worker - WARNING - Compute Failed
Key:       _get_model_attr-4f0b08ad-aa68-41f6-8d22-070bd393bfe4
Function:  _get_model_attr
args:      (KMeansMG(), '_repr_jpeg_')
kwargs:    {}
Exception: 'AttributeError("Attribute _repr_jpeg_ does not exist on model <class \'cuml.cluster.kmeans_mg.KMeansMG\'>")'

2022-08-18 11:14:57,158 - distributed.worker - WARNING - Compute Failed
Key:       _get_model_attr-989a48f2-74c3-43d5-99f2-458879db3ccd
Function:  _get_model_attr
args:      (KMeansMG(), '_ipython_canary_method_should_not_exist_')
kwargs:    {}
Exception: 'AttributeError("Attribute _ipython_canary_method_should_not_exist_ does not exist on model <class \'cuml.cluster.kmeans_mg.KMeansMG\'>")'

2022-08-18 11:14:57,168 - distributed.worker - WARNING - Compute Failed
Key:       _get_model_attr-4f0902e1-1698-4954-8ecd-c4a387b6adac
Function:  _get_model_attr
args:      (KMeansMG(), '_repr_latex_')
kwargs:    {}
Exception: 'AttributeError("Attribute _repr_latex_ does not exist on model <class \'cuml.cluster.kmeans_mg.KMeansMG\'>")'

2022-08-18 11:14:57,179 - distributed.worker - WARNING - Compute Failed
Key:       _get_model_attr-c4fc8041-6a33-4870-809b-ae66882832e1
Function:  _get_model_attr
args:      (KMeansMG(), '_ipython_canary_method_should_not_exist_')
kwargs:    {}
Exception: 'AttributeError("Attribute _ipython_canary_method_should_not_exist_ does not exist on model <class \'cuml.cluster.kmeans_mg.KMeansMG\'>")'

2022-08-18 11:14:57,200 - distributed.worker - WARNING - Compute Failed
Key:       _get_model_attr-bfc01adc-5052-4262-a191-348e2839170c
Function:  _get_model_attr
args:      (KMeansMG(), '_repr_json_')
kwargs:    {}
Exception: 'AttributeError("Attribute _repr_json_ does not exist on model <class \'cuml.cluster.kmeans_mg.KMeansMG\'>")'

2022-08-18 11:14:57,212 - distributed.worker - WARNING - Compute Failed
Key:       _get_model_attr-9b9629a8-e428-47ea-83c1-c7f8945cdc93
Function:  _get_model_attr
args:      (KMeansMG(), '_ipython_canary_method_should_not_exist_')
kwargs:    {}
Exception: 'AttributeError("Attribute _ipython_canary_method_should_not_exist_ does not exist on model <class \'cuml.cluster.kmeans_mg.KMeansMG\'>")'

2022-08-18 11:14:57,224 - distributed.worker - WARNING - Compute Failed
Key:       _get_model_attr-ff3849d1-d339-4eca-93d9-db4fc9699a58
Function:  _get_model_attr
args:      (KMeansMG(), '_repr_javascript_')
kwargs:    {}
Exception: 'AttributeError("Attribute _repr_javascript_ does not exist on model <class \'cuml.cluster.kmeans_mg.KMeansMG\'>")'

[10]:
<cuml.dask.cluster.kmeans.KMeans at 0x7ff7bbbfe670>
[11]:
import pickle

single_gpu_model = dist_model.get_combined_model()
pickle.dump(single_gpu_model, open("kmeans_model.pkl", "wb"))
[12]:
single_gpu_model = pickle.load(open("kmeans_model.pkl", "rb"))
[13]:
single_gpu_model.cluster_centers_
[13]:
array([[-4.6475396e+00, -9.5672169e+00,  6.6923518e+00,  4.4359241e+00,
         2.1902733e+00,  2.5834751e+00,  5.9448940e-01,  6.2568660e+00,
        -8.7821636e+00, -4.1232014e-01,  9.8151779e+00,  7.5641570e+00,
         1.0003010e+01, -5.8680434e+00, -1.2743108e+00, -2.5393455e+00,
        -1.0847499e+00, -5.2629204e+00, -9.3071022e+00,  4.6179361e+00,
        -9.7068958e-02, -3.9351208e+00,  6.1767278e+00, -7.4346881e+00,
         5.6496067e+00, -8.5544844e+00, -7.5265574e+00, -5.5195603e+00,
         4.8197632e+00,  2.5235438e+00],
       [ 6.2794294e+00,  9.2293949e+00,  8.3403702e+00,  9.0330496e+00,
         7.6893492e+00, -9.9538219e-01, -6.2780757e+00,  1.3599331e+00,
        -6.9744487e+00, -5.9463463e+00,  1.0695115e+00, -8.0422480e-03,
         2.8183143e+00,  1.8317667e+00, -8.2557344e+00,  3.0514317e+00,
        -8.4958019e+00,  9.7238483e+00, -7.7455091e+00,  3.4521694e+00,
        -3.9248335e+00, -4.1106420e+00,  2.6693091e+00,  1.2985628e+00,
         1.0421573e+00,  5.2490625e+00, -1.6496240e+00,  6.1451659e+00,
        -6.9103327e+00, -9.6390314e+00],
       [-2.8796387e+00,  4.4348760e+00, -4.4264812e+00,  2.3959141e+00,
         1.7286435e+00, -2.4991984e+00, -5.1794519e+00, -1.6771443e+00,
        -8.1329174e+00,  2.6659226e+00, -4.3131099e+00,  5.5827813e+00,
        -5.7318306e+00, -1.7427170e+00, -9.3456125e+00,  7.1365553e-01,
         4.4255495e+00, -2.9118381e+00, -4.9467444e+00,  9.6786413e+00,
         8.4222746e+00, -6.2710242e+00, -6.3596516e+00,  1.9645507e+00,
         4.1715999e+00, -9.1683636e+00,  4.6156688e+00,  8.7916498e+00,
         6.8754416e+00,  2.2288744e+00],
       [-6.9536948e+00, -9.7635870e+00, -6.5648260e+00, -4.3536150e-01,
         6.0998106e+00,  3.7550373e+00, -3.9558542e+00,  6.1595521e+00,
        -1.8599318e+00,  5.0400305e+00, -6.8397546e+00,  1.3435434e+00,
         8.9749022e+00, -9.9621767e-01,  9.6651945e+00,  9.8009644e+00,
        -8.6188335e+00,  5.9978361e+00,  2.2295928e+00, -3.6477709e+00,
         7.0758505e+00, -7.3772259e+00, -5.3214221e+00, -6.9927959e+00,
        -7.9296312e+00,  6.6705351e+00, -5.5850182e+00,  7.1526051e+00,
         6.5703216e+00, -8.3389387e+00],
       [ 4.8136683e+00,  8.3985281e+00, -9.2161236e+00,  9.4185524e+00,
         8.5280886e+00, -1.0969982e+00,  3.3253171e+00, -7.8064380e+00,
        -5.9660631e-01,  2.5423869e-01,  5.5004086e+00, -4.1162963e+00,
         4.2832375e+00, -2.8173413e+00,  3.6207721e+00, -4.1576214e+00,
        -3.6048703e+00,  6.2125401e+00, -6.9080997e+00, -1.0732133e+00,
        -5.8362122e+00,  2.2357669e+00, -3.8588791e+00, -1.6835878e+00,
        -5.3240371e+00,  7.5769196e+00,  2.9358530e+00,  8.5267372e+00,
         1.5667247e+00,  1.0779673e+00]], dtype=float32)

Exporting cuML Random Forest models for inferencing on machines without GPUs

Starting with cuML version 21.06, you can export cuML Random Forest models and run predictions with them on machines without an NVIDIA GPUs. The Treelite package defines an efficient exchange format that lets you portably move the cuML Random Forest models to other machines. We will refer to the exchange format as “checkpoints.”

Here are the steps to export the model:

  1. Call to_treelite_checkpoint() to obtain the checkpoint file from the cuML Random Forest model.

[14]:
from cuml.ensemble import RandomForestClassifier as cumlRandomForestClassifier
from sklearn.datasets import load_iris
import numpy as np

X, y = load_iris(return_X_y=True)
X, y = X.astype(np.float32), y.astype(np.int32)
clf = cumlRandomForestClassifier(max_depth=3, random_state=0, n_estimators=10)
clf.fit(X, y)

checkpoint_path = './checkpoint.tl'
# Export cuML RF model as Treelite checkpoint
clf.convert_to_treelite_model().to_treelite_checkpoint(checkpoint_path)
/opt/conda/envs/rapids/lib/python3.9/site-packages/cuml/internals/api_decorators.py:794: UserWarning: For reproducible results in Random Forest Classifier or for almost reproducible results in Random Forest Regressor, n_streams=1 is recommended. If n_streams is > 1, results may vary due to stream/thread timing differences, even when random_state is set
  return func(**kwargs)
  1. Copy the generated checkpoint file checkpoint.tl to another machine on which you’d like to run predictions.

  2. On the target machine, install Treelite by running pip install treelite or conda install -c conda-forge treelite. The machine does not need to have an NVIDIA GPUs and does not need to have cuML installed.

  3. You can now load the model from the checkpoint, by running the following on the target machine:

[15]:
import treelite

# The checkpoint file has been copied over
checkpoint_path = './checkpoint.tl'
tl_model = treelite.Model.deserialize(checkpoint_path)
out_prob = treelite.gtil.predict(tl_model, X, pred_margin=True)
print(out_prob)
[[1.         0.         0.        ]
 [1.         0.         0.        ]
 [1.         0.         0.        ]
 [1.         0.         0.        ]
 [1.         0.         0.        ]
 [1.         0.         0.        ]
 [1.         0.         0.        ]
 [1.         0.         0.        ]
 [1.         0.         0.        ]
 [1.         0.         0.        ]
 [1.         0.         0.        ]
 [1.         0.         0.        ]
 [1.         0.         0.        ]
 [1.         0.         0.        ]
 [0.90789473 0.09210526 0.        ]
 [0.90789473 0.09210526 0.        ]
 [1.         0.         0.        ]
 [1.         0.         0.        ]
 [0.90789473 0.09210526 0.        ]
 [1.         0.         0.        ]
 [1.         0.         0.        ]
 [1.         0.         0.        ]
 [1.         0.         0.        ]
 [1.         0.         0.        ]
 [1.         0.         0.        ]
 [1.         0.         0.        ]
 [1.         0.         0.        ]
 [1.         0.         0.        ]
 [1.         0.         0.        ]
 [1.         0.         0.        ]
 [1.         0.         0.        ]
 [1.         0.         0.        ]
 [1.         0.         0.        ]
 [1.         0.         0.        ]
 [1.         0.         0.        ]
 [1.         0.         0.        ]
 [1.         0.         0.        ]
 [1.         0.         0.        ]
 [1.         0.         0.        ]
 [1.         0.         0.        ]
 [1.         0.         0.        ]
 [1.         0.         0.        ]
 [1.         0.         0.        ]
 [1.         0.         0.        ]
 [1.         0.         0.        ]
 [1.         0.         0.        ]
 [1.         0.         0.        ]
 [1.         0.         0.        ]
 [1.         0.         0.        ]
 [1.         0.         0.        ]
 [0.00789474 0.96374357 0.02836171]
 [0.00789474 0.96374357 0.02836171]
 [0.00789474 0.7819892  0.2101161 ]
 [0.         0.99018145 0.00981859]
 [0.00789474 0.96374357 0.02836171]
 [0.00789474 0.975835   0.01627021]
 [0.00789474 0.96374357 0.02836171]
 [0.         0.8568481  0.14315192]
 [0.00789474 0.96374357 0.02836171]
 [0.         0.99018145 0.00981859]
 [0.         0.99018145 0.00981859]
 [0.00789474 0.975835   0.01627021]
 [0.00789474 0.975835   0.01627021]
 [0.00789474 0.975835   0.01627021]
 [0.10789473 0.88228667 0.00981859]
 [0.00789474 0.96374357 0.02836171]
 [0.00789474 0.98228663 0.00981859]
 [0.00789474 0.975835   0.01627021]
 [0.00789474 0.975835   0.01627021]
 [0.10789473 0.88228667 0.00981859]
 [0.         0.25154537 0.7484547 ]
 [0.00789474 0.975835   0.01627021]
 [0.00789474 0.7819892  0.2101161 ]
 [0.00789474 0.975835   0.01627021]
 [0.00789474 0.96374357 0.02836171]
 [0.00789474 0.96374357 0.02836171]
 [0.00789474 0.79310024 0.199005  ]
 [0.         0.451291   0.54870903]
 [0.00789474 0.975835   0.01627021]
 [0.10789473 0.87583506 0.01627021]
 [0.         0.99018145 0.00981859]
 [0.         0.99018145 0.00981859]
 [0.10789473 0.87583506 0.01627021]
 [0.         0.18708128 0.8129188 ]
 [0.         0.99018145 0.00981859]
 [0.00789474 0.975835   0.01627021]
 [0.00789474 0.96374357 0.02836171]
 [0.00789474 0.96374357 0.02836171]
 [0.00789474 0.98228663 0.00981859]
 [0.         0.99018145 0.00981859]
 [0.         0.99018145 0.00981859]
 [0.00789474 0.975835   0.01627021]
 [0.00789474 0.975835   0.01627021]
 [0.         0.99018145 0.00981859]
 [0.00789474 0.98228663 0.00981859]
 [0.00789474 0.975835   0.01627021]
 [0.00789474 0.975835   0.01627021]
 [0.00789474 0.975835   0.01627021]
 [0.         0.99018145 0.00981859]
 [0.00789474 0.975835   0.01627021]
 [0.         0.01258634 0.9874137 ]
 [0.         0.0481924  0.9518076 ]
 [0.         0.01258634 0.9874137 ]
 [0.         0.01258634 0.9874137 ]
 [0.         0.01258634 0.9874137 ]
 [0.         0.01258634 0.9874137 ]
 [0.         0.6901814  0.3098186 ]
 [0.         0.01258634 0.9874137 ]
 [0.         0.01258634 0.9874137 ]
 [0.         0.01258634 0.9874137 ]
 [0.         0.01258634 0.9874137 ]
 [0.         0.01258634 0.9874137 ]
 [0.         0.01258634 0.9874137 ]
 [0.         0.15504913 0.84495085]
 [0.         0.0481924  0.9518076 ]
 [0.         0.01258634 0.9874137 ]
 [0.         0.01258634 0.9874137 ]
 [0.         0.01258634 0.9874137 ]
 [0.         0.01258634 0.9874137 ]
 [0.         0.40701494 0.59298503]
 [0.         0.01258634 0.9874137 ]
 [0.         0.14688587 0.8531141 ]
 [0.         0.01258634 0.9874137 ]
 [0.         0.28718114 0.71281886]
 [0.         0.01258634 0.9874137 ]
 [0.         0.01258634 0.9874137 ]
 [0.         0.22427265 0.7757274 ]
 [0.         0.21316154 0.7868384 ]
 [0.         0.01258634 0.9874137 ]
 [0.         0.17874795 0.82125205]
 [0.         0.01258634 0.9874137 ]
 [0.         0.01258634 0.9874137 ]
 [0.         0.01258634 0.9874137 ]
 [0.         0.21682489 0.7831751 ]
 [0.         0.22515821 0.77484185]
 [0.         0.01258634 0.9874137 ]
 [0.         0.01258634 0.9874137 ]
 [0.         0.01258634 0.9874137 ]
 [0.         0.22427265 0.7757274 ]
 [0.         0.01258634 0.9874137 ]
 [0.         0.01258634 0.9874137 ]
 [0.         0.01258634 0.9874137 ]
 [0.         0.0481924  0.9518076 ]
 [0.         0.01258634 0.9874137 ]
 [0.         0.01258634 0.9874137 ]
 [0.         0.01258634 0.9874137 ]
 [0.         0.20179601 0.798204  ]
 [0.         0.01258634 0.9874137 ]
 [0.         0.02091968 0.9790803 ]
 [0.         0.0481924  0.9518076 ]]