Batch training & tuning on Ray Tune#

Batch training and tuning are common tasks in simple machine learning use-cases such as time series forecasting. They require fitting of simple models on multiple data batches corresponding to locations, products, etc.

‘Batch training’ is a workload that trains model(s) on subsets of a dataset. This notebook showcases how to conduct batch training using Ray Tune.

Batch training diagram

For the data, we will use the NYC Taxi dataset. This popular tabular dataset contains historical taxi pickups by timestamp and location in NYC.

In the notebook, we will split the data by dropoff location and train a separate regression model for each dropoff location predicting trip_duration. Specifically, we will use the dropoff_location_id column in the dataset to group the dataset into data batches. Then we will fit a separate model for each batch and evaluate it.

Contents#

In this this tutorial, you will learn about:

  1. Define how to load and prepare Parquet data

  2. Define your Ray Tune Search Space and Search Algorithm

  3. Define a Trainable (callable) function

  4. Run batch training on Ray Tune

  5. Load a model from checkpoint and perform inference

Walkthrough#

Tip

Prerequisite for this notebook: Read the Key Concepts page for Ray Tune.

Let us start by importing a few required libraries, including open-source Ray itself!

import os

print(f"Number of CPUs in this system: {os.cpu_count()}")
from typing import Tuple, List, Union, Optional, Callable
import time
import pandas as pd
import numpy as np
import pyarrow
import pyarrow.parquet as pq
import pyarrow.dataset as pds

print(f"pyarrow: {pyarrow.__version__}")

Number of CPUs in this system: 16
pyarrow: 10.0.0

import ray

if ray.is_initialized():
    ray.shutdown()
ray.init()

print(ray.cluster_resources())

{'CPU': 32.0, 'memory': 42071657678.0, 'node:172.31.92.227': 1.0, 'object_store_memory': 23720183398.0, 'node:172.31.119.49': 1.0}

# import standard sklearn libraries
import sklearn
from sklearn.base import BaseEstimator
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression
from sklearn.tree import DecisionTreeRegressor
from sklearn.metrics import mean_absolute_error

print(f"sklearn: {sklearn.__version__}")

# import ray libraries
from ray import air, tune
from ray.air import session
from ray.air.checkpoint import Checkpoint

# set global random seed for sklearn models
np.random.seed(415)

sklearn: 1.1.3

# For benchmarking purposes, we can print the times of various operations.
# In order to reduce clutter in the output, this is set to False by default.
PRINT_TIMES = False


def print_time(msg: str):
    if PRINT_TIMES:
        print(msg)


# To speed things up, we’ll only use a small subset of the full dataset consisting of two last months of 2019.
# You can choose to use the full dataset for 2018-2019 by setting the SMOKE_TEST variable to False.
SMOKE_TEST = True

Define how to load and prepare Parquet data #

First, we need to load some data. Since the NYC Taxi dataset is fairly large, we will filter files first into a PyArrow dataset. And then in the next cell after, we will filter the data on read into a PyArrow table and convert that to a pandas dataframe.

Tip

Use PyArrow dataset and table for reading or writing large parquet files, since its native multithreaded C++ adpater is faster than pandas read_parquet, even using engine=pyarrow.

# Define some global variables.
target = "trip_duration"
s3_partitions = pds.dataset(
    "s3://anonymous@air-example-data/ursa-labs-taxi-data/by_year/",
    partitioning=["year", "month"],
)
s3_files = [f"s3://anonymous@{file}" for file in s3_partitions.files]

# Obtain all location IDs
all_location_ids = (
    pq.read_table(s3_files[0], columns=["dropoff_location_id"])["dropoff_location_id"]
    .unique()
    .to_pylist()
)
# drop [264, 265]
all_location_ids.remove(264)
all_location_ids.remove(265)

# Use smoke testing or not.
starting_idx = -1 if SMOKE_TEST else 0
# note: use location 199 to test error-handling
sample_locations = [90, 145, 152, 204] if SMOKE_TEST else all_location_ids

# Display what data will be used.
s3_files = s3_files[starting_idx:]
print(f"NYC Taxi using {len(s3_files)} file(s)!")
print(f"s3_files: {s3_files}")
print(f"Locations: {sample_locations}")

# Display what data will be used.
s3_files = s3_files[starting_idx:]
print(f"NYC Taxi using {len(s3_files)} file(s)!")
print(f"s3_files: {s3_files}")
print(f"Locations: {sample_locations}")

NYC Taxi using 18 file(s)!
s3_files: ['s3://air-example-data/ursa-labs-taxi-data/by_year/2018/01/data.parquet/4d6bc4368704460d90c92c22e05a2220_000000.parquet', 's3://air-example-data/ursa-labs-taxi-data/by_year/2018/02/data.parquet/e817946252d1409b93964685130e76cb_000000.parquet', 's3://air-example-data/ursa-labs-taxi-data/by_year/2018/03/data.parquet/0b7e5121a4904c64be5e91ceec0eee2f_000000.parquet', 's3://air-example-data/ursa-labs-taxi-data/by_year/2018/04/data.parquet/f40c2c2806e548bfac8336de9c19a423_000000.parquet', 's3://air-example-data/ursa-labs-taxi-data/by_year/2018/05/data.parquet/a5de27164fda47988dec2576685656ae_000000.parquet', 's3://air-example-data/ursa-labs-taxi-data/by_year/2018/06/data.parquet/df104576ffed4e308b72941df90f7790_000000.parquet', 's3://air-example-data/ursa-labs-taxi-data/by_year/2018/07/data.parquet/ccdef45e50de4678b7e589155f372a3d_000000.parquet', 's3://air-example-data/ursa-labs-taxi-data/by_year/2018/08/data.parquet/9d605bf8abf84655997d491bc5a10a4c_000000.parquet', 's3://air-example-data/ursa-labs-taxi-data/by_year/2018/09/data.parquet/b200f3d9bf9f485ebd3b20c0c08711e1_000000.parquet', 's3://air-example-data/ursa-labs-taxi-data/by_year/2018/10/data.parquet/20624e28db574114b47de3e43065f014_000000.parquet', 's3://air-example-data/ursa-labs-taxi-data/by_year/2018/11/data.parquet/9c3fe546f3d746eeb3225b8150fb26e6_000000.parquet', 's3://air-example-data/ursa-labs-taxi-data/by_year/2018/12/data.parquet/d9829239c5d34340a7d9ba256917ed98_000000.parquet', 's3://air-example-data/ursa-labs-taxi-data/by_year/2019/01/data.parquet/ecce6478ad09480cbc8539e0b6197c2d_000000.parquet', 's3://air-example-data/ursa-labs-taxi-data/by_year/2019/02/data.parquet/5bc40cf9bc1145cbb0867d39064daa01_000000.parquet', 's3://air-example-data/ursa-labs-taxi-data/by_year/2019/03/data.parquet/8b894872a484458cbd5a6cd0425b77df_000000.parquet', 's3://air-example-data/ursa-labs-taxi-data/by_year/2019/04/data.parquet/7e490662e39c4bfe8c64c6a2c45c9e8b_000000.parquet', 's3://air-example-data/ursa-labs-taxi-data/by_year/2019/05/data.parquet/359c21b3e28f40328e68cf66f7ba40e2_000000.parquet', 's3://air-example-data/ursa-labs-taxi-data/by_year/2019/06/data.parquet/ab5b9d2b8cc94be19346e260b543ec35_000000.parquet']
Locations: [24, 140, 141, 257, 239, 143, 238, 170, 229, 113, 224, 79, 100, 189, 236, 162, 61, 75, 233, 125, 146, 234, 263, 237, 166, 249, 151, 231, 116, 145, 82, 164, 68, 161, 48, 90, 13, 148, 88, 87, 213, 74, 25, 158, 246, 12, 142, 66, 186, 232, 4, 262, 202, 107, 50, 42, 205, 137, 80, 114, 43, 181, 255, 7, 126, 89, 220, 133, 209, 152, 160, 41, 144, 211, 203, 65, 20, 167, 168, 129, 139, 223, 33, 227, 26, 36, 217, 226, 193, 3, 112, 163, 62, 117, 95, 21, 261, 17, 127, 174, 198, 243, 256, 241, 70, 244, 108, 173, 197, 260, 67, 259, 208, 72, 52, 40, 230, 73, 216, 37, 228, 10, 14, 225, 179, 83, 45, 11, 49, 97, 200, 250, 69, 196, 178, 188, 138, 106, 18, 22, 77, 169, 132, 182, 1, 201, 28, 159, 222, 54, 32, 78, 51, 149, 218, 60, 183, 55, 157, 212, 175, 147, 119, 235, 128, 136, 16, 92, 247, 195, 57, 121, 39, 9, 35, 56, 185, 248, 251, 58, 190, 177, 86, 171, 155, 258, 153, 192, 47, 91, 124, 191, 130, 240, 94, 122, 98, 206, 102, 165, 71, 76, 254, 219, 215, 118, 134, 156, 63, 131, 135, 245, 180, 242, 252, 93, 64, 85, 29, 15, 176, 31, 34, 210, 81, 194, 120, 172, 19, 150, 38, 111, 214, 123, 46, 101, 204, 27, 115, 53, 44, 8, 5, 109, 23, 59, 221, 6, 253, 96, 84, 207, 154, 187, 184, 30, 105, 2, 199]

# Function to read a pyarrow.Table object using pyarrow parquet
def read_data(file: str, sample_id: np.int32) -> pd.DataFrame:

    df = pq.read_table(
        file,
        filters=[
            ("passenger_count", ">", 0),
            ("trip_distance", ">", 0),
            ("fare_amount", ">", 0),
            ("pickup_location_id", "not in", [264, 265]),
            ("dropoff_location_id", "not in", [264, 265]),
            ("dropoff_location_id", "=", sample_id),
        ],
        columns=[
            "pickup_at",
            "dropoff_at",
            "pickup_location_id",
            "dropoff_location_id",
            "passenger_count",
            "trip_distance",
            "fare_amount",
        ],
    ).to_pandas()

    return df


# Function to transform a pandas dataframe
def transform_df(the_df: pd.DataFrame) -> pd.DataFrame:
    df = the_df.copy()
    # calculate trip_duration
    df["trip_duration"] = (df["dropoff_at"] - df["pickup_at"]).dt.seconds
    # filter trip_durations > 1 minute and less than 24 hours
    df = df[df["trip_duration"] > 60]
    df = df[df["trip_duration"] < 24 * 60 * 60]
    # keep only necessary columns
    df.drop(
        ["dropoff_at", "pickup_at", "pickup_location_id", "fare_amount"],
        axis=1,
        inplace=True,
    )
    df["dropoff_location_id"] = df["dropoff_location_id"].fillna(-1)
    return df

# %%time

# # Test reading data.
# import itertools
# my_list = itertools.product(s3_files, sample_locations)

# # [print(f[0], f[1]) for f in my_list]
# df_list = [read_data(f[0], f[1]) for f in my_list]
# df_raw = pd.concat(df_list, ignore_index=True)
# # Transform data.
# df = transform_df(df_raw)

# # Inspect the pandas dataframe.
# df.head()

Define your Ray Tune Search Space and Search Algorithm #

In this notebook, we will use Ray Tune to run parallel training jobs per dropoff location. The training jobs will be defined using a search space and simple grid search. Depending on your need, fancier search spaces and search algorithms are possible with Tune.

First, define a search space of experiment trials to run.

The typical use case for Tune search spaces are for hypterparameter tuning. In our case, we are defining a Tune search space in a way to allow for training jobs to be conducted automatically. Each training job will run on a different data partition (taxi dropoff location) and use a different model.

Common search algorithms include grid search, random search, and Bayesian optimization. For more details, see Working with Tune Search Spaces. Deciding the best combination of search space and search algorithm is part of the art of being a Data Scientist and depends on the data, algorithm, and problem being solved!

Next, define a search algorithm.

Ray Tune will use the search space and the specified search algorithm to generate multiple configurations, each of which will be evaluated in a separate Trial on a Ray Cluster. Ray Tune will take care of orchestrating those Trials automatically. Specifically, Ray Tune will pass a config dictionary to each partition and make a Trainable function call.

Below, we define our search space consists of:

  • 2 different Scikit-learn algorithms

  • Some or all NYC taxi drop-off locations.

Also below, we define our search algorithm is:

  • Grid search.

What this means is every algorithm will be applied to every NYC Taxi drop-off location.

# 1. Define a search space.
search_space = {
    "model": tune.grid_search(
        [LinearRegression(fit_intercept=True), DecisionTreeRegressor(max_depth=3)]
    ),
    "location": tune.grid_search(sample_locations),
}

Define a Trainable (callable) function #

📈 Typically when you are running Data Science experiments, you want to be able to keep track of summary metrics for each trial, so you can decide at the end which trials were best. That way, you can decide which model to deploy.

🇫 Next, we define a trainable function in order to train and evaluate a scikit-learn model on a data partition. This function will be called in parallel by every Tune trial. Inside this trainable function, we will:

  • Add detailed metrics we want to report (each model’s loss or error).

  • Checkpoint each model for easy deployment later.

📖 The metrics defined inside the trainable function will appear in the Ray Tune experiment summary table.

Tip

Ray Tune has two ways of defining a trainable, namely the Function API and the Class API. Both are valid ways of defining a trainable, but the Function API is generally recommended.

In the cell below, we define a “Trainable” function called train_model().

  • The input is a config dictionary argument.

  • The output can be a simple dictionary of metrics which will be reported back to Tune.

  • We will checkpoint save each model in addition to reporting each trial’s metrics.

    For checkpointing, we use ray.air.checkpoint.Checkpoint. Ray AIR includes integrations to popular ML libraries, including Scikit-learn. This makes it possible to use the convenient AIR API abstractions, without having to specify code details of the Scikit-learn library itself.

  • Since we are using grid search, this means train_model() will be run in parallel for every permutation in the Tune search space!

from ray.air import Checkpoint

# 2. Define a custom train function
def train_model(config: dict):

    model = config["model"]
    location_id = config["location"]

    # Load data.
    df_list = [read_data(f, location_id) for f in s3_files]
    df_raw = pd.concat(df_list, ignore_index=True)
    df = transform_df(df_raw)

    # We need at least 10 rows to create a train / test split.
    if len(df) < 10:
        print_time(
            f"Data batch for LocID {location_id} is empty or smaller than 10 rows"
        )
        session.report(dict(error=None))
        return

    # Train/valid split.
    train_df, valid_df = train_test_split(df, test_size=0.2, shuffle=True)
    train_X = train_df[["passenger_count", "trip_distance"]]
    train_y = train_df.trip_duration
    valid_X = valid_df[["passenger_count", "trip_distance"]]
    valid_y = valid_df.trip_duration

    # Train model.
    model = model.fit(train_X, train_y)
    pred_y = model.predict(valid_X)

    # Evaluate.
    error = sklearn.metrics.mean_absolute_error(valid_y, pred_y)

    # Define a model checkpoint using AIR API.
    # https://docs.ray.io/en/latest/tune/tutorials/tune-checkpoints.html
    checkpoint = Checkpoint.from_dict({"model": model, "location_id": location_id})

    # Save checkpoint and report back metrics, using ray.air.session.report()
    # The metrics you specify here will appear in Tune summary table.
    # They will also be recorded in Tune results under `metrics`.
    metrics = dict(error=error)
    session.report(metrics, checkpoint=checkpoint)

Run batch training on Ray Tune #

In the cell below, we configure the resources allocated per trial.

Tune uses this resources allocation to control the parallelism. For example, if each trial was configured to use 4 CPUs, and the cluster had only 32 CPUs, then Tune will limit the number of concurrent trials to 8 to avoid overloading the cluster. For more information, see A Guide To Parallelism and Resources.

# 3. Customize resources per trial, here we set 1 CPU each.
train_model = tune.with_resources(train_model, {"cpu": 1})

Now we are ready to kick off a Ray Tune experiment!

Recall what we are doing, high level, is training several different models per dropoff location. We are using Ray Tune so we can run all these trials in parallel. At the end, we will inspect the results of the experiment and deploy only the best model per dropoff location.

In the cell below, we use AIR configs and run the experiment using tuner.fit().

Tune will report on experiment status, and after the experiment finishes, you can inspect the results.

In the AIR config below, we have specified a local directory my_Tune_logs for logging instead of the default ~/ray_results directory. Giving your logs a project name makes them easier to find. Also giving a relative path, means you can see your logs inside the Jupyter browser. Learn more about logging Tune results at How to configure logging in Tune.

Tune can retry failed experiments automatically, as well as entire experiments. This is necessary in case a node on your remote cluster fails (when running on a cloud such as AWS or GCP).

💡 Right-click on the cell below and choose “Enable Scrolling for Outputs”! This will make it easier to view, since model training output can be very long!

Below was tested for 518 models, using 18 NYC Taxi S3 files dating from 2018/01 to 2019/06 (split into partitions approx 1GiB each), simultaneously trained on a 23-node AWS cluster of m5.4xlarges. Total data reading and train time was 37 minutes.

# Define a tuner object using Ray AIR Tuner API
tuner = tune.Tuner(
    train_model,
    param_space=search_space,
    run_config=air.RunConfig(
        # redirect logs to relative path instead of default ~/ray_results/
        local_dir="my_Tune_logs",
        name="batch_tuning",
        # Set Ray Tune verbosity.  Print summary table only with levels 2 or 3.
        verbose=2,
    ),
)

# 4. Run the experiment with Ray Tune
start = time.time()
results = tuner.fit()
total_time_taken = time.time() - start

# Print some training stats
print(f"Total number of models: {len(results)}")
print(f"TOTAL TIME TAKEN: {total_time_taken:.2f} seconds")
best_result = results.get_best_result(metric="error", mode="min").config
print(f"Best result: {best_result}")

Tune Status

Current time:2022-11-10 12:26:32
Running for: 00:37:01.91
Memory: 4.3/61.4 GiB

System Info

Using FIFO scheduling algorithm.
Resources requested: 0/368 CPUs, 0/0 GPUs, 0.0/144.18 GiB heap, 0.0/127.09 GiB objects

Trial Status

Trial name status loc locationmodel iter total time (s) error
train_model_cabb8_00000TERMINATED172.31.92.227:44554 24LinearRegression() 1 66.3002 447.234
train_model_cabb8_00001TERMINATED172.31.92.227:44592 140LinearRegression() 1 62.7181 493.912
train_model_cabb8_00002TERMINATED172.31.92.227:44594 141LinearRegression() 1 64.1264 461.717
train_model_cabb8_00003TERMINATED172.31.92.227:44596 257LinearRegression() 1 61.2023 832.879
train_model_cabb8_00004TERMINATED172.31.92.227:44598 239LinearRegression() 1 64.3973 447.743
train_model_cabb8_00005TERMINATED172.31.92.227:44600 143LinearRegression() 1 61.6934 446.824
train_model_cabb8_00006TERMINATED172.31.92.227:44603 238LinearRegression() 1 62.466 424.082
train_model_cabb8_00007TERMINATED172.31.92.227:44606 170LinearRegression() 1 65.7129 496.27
train_model_cabb8_00008TERMINATED172.31.92.227:44608 229LinearRegression() 1 62.6523 502.505
train_model_cabb8_00009TERMINATED172.31.92.227:44613 113LinearRegression() 1 62.8456 473.44
train_model_cabb8_00010TERMINATED172.31.92.227:44616 224LinearRegression() 1 62.408 468.736
train_model_cabb8_00011TERMINATED172.31.92.227:44618 79LinearRegression() 1 62.0856 536.818
train_model_cabb8_00012TERMINATED172.31.92.227:44622 100LinearRegression() 1 62.319 586.173
train_model_cabb8_00013TERMINATED172.31.92.227:44625 189LinearRegression() 1 63.223 826.569
train_model_cabb8_00014TERMINATED172.31.92.227:44629 236LinearRegression() 1 64.5983 411.649
train_model_cabb8_00015TERMINATED172.31.92.227:44634 162LinearRegression() 1 64.1532 519.815
train_model_cabb8_00016TERMINATED172.31.107.40:6554 61LinearRegression() 1 59.5367 843.189
train_model_cabb8_00017TERMINATED172.31.107.40:6585 75LinearRegression() 1 59.9344 444.058
train_model_cabb8_00018TERMINATED172.31.107.40:6617 233LinearRegression() 1 60.6699 515.764
train_model_cabb8_00019TERMINATED172.31.107.40:6620 125LinearRegression() 1 59.4999 561.376
train_model_cabb8_00020TERMINATED172.31.107.40:6672 146LinearRegression() 1 58.423 635.01
train_model_cabb8_00021TERMINATED172.31.107.40:6700 234LinearRegression() 1 60.9398 469.924
train_model_cabb8_00022TERMINATED172.31.107.40:6723 263LinearRegression() 1 60.2877 434.226
train_model_cabb8_00023TERMINATED172.31.107.40:6729 237LinearRegression() 1 61.1364 440.032
train_model_cabb8_00024TERMINATED172.31.107.40:6751 166LinearRegression() 1 59.2446 465.123
train_model_cabb8_00025TERMINATED172.31.107.40:6754 249LinearRegression() 1 59.9338 488.919
train_model_cabb8_00026TERMINATED172.31.107.40:6756 151LinearRegression() 1 59.3968 437.95
train_model_cabb8_00027TERMINATED172.31.107.40:6759 231LinearRegression() 1 60.2091 587.417
train_model_cabb8_00028TERMINATED172.31.107.40:6762 116LinearRegression() 1 58.9658 638.248
train_model_cabb8_00029TERMINATED172.31.107.40:6763 145LinearRegression() 1 58.7797 620.117
train_model_cabb8_00030TERMINATED172.31.107.40:6766 82LinearRegression() 1 57.706 746.608
train_model_cabb8_00031TERMINATED172.31.107.40:6767 164LinearRegression() 1 60.3834 573.606
train_model_cabb8_00032TERMINATED172.31.116.192:1058 68LinearRegression() 1 54.0429 530.903
train_model_cabb8_00033TERMINATED172.31.116.192:1059 161LinearRegression() 1 54.6596 528.775
train_model_cabb8_00034TERMINATED172.31.116.192:1062 48LinearRegression() 1 54.1717 582.801
train_model_cabb8_00035TERMINATED172.31.116.192:1063 90LinearRegression() 1 53.0674 476.959
train_model_cabb8_00036TERMINATED172.31.116.192:1064 13LinearRegression() 1 53.255 630.055
train_model_cabb8_00037TERMINATED172.31.116.192:1065 148LinearRegression() 1 53.1753 645.033
train_model_cabb8_00038TERMINATED172.31.116.192:1066 88LinearRegression() 1 51.1529 670.53
train_model_cabb8_00039TERMINATED172.31.116.192:1067 87LinearRegression() 1 52.334 623.395
train_model_cabb8_00040TERMINATED172.31.116.192:1068 213LinearRegression() 1 48.504 819.431
train_model_cabb8_00041TERMINATED172.31.116.192:1069 74LinearRegression() 1 51.9802 489.846
train_model_cabb8_00042TERMINATED172.31.116.192:1070 25LinearRegression() 1 50.1846 738.86
train_model_cabb8_00043TERMINATED172.31.116.192:1071 158LinearRegression() 1 53.3489 552.345
train_model_cabb8_00044TERMINATED172.31.116.192:1073 246LinearRegression() 1 53.4666 506.195
train_model_cabb8_00045TERMINATED172.31.116.192:1076 12LinearRegression() 1 51.7988 748.864
train_model_cabb8_00046TERMINATED172.31.116.192:1078 142LinearRegression() 1 54.6032 444.971
train_model_cabb8_00047TERMINATED172.31.116.192:1081 66LinearRegression() 1 52.4013 684.737
train_model_cabb8_00048TERMINATED172.31.92.227:44634 186LinearRegression() 1 38.9382 582.08
train_model_cabb8_00049TERMINATED172.31.119.49:1057 232LinearRegression() 1 52.9945 582.837
train_model_cabb8_00050TERMINATED172.31.118.86:1085 4LinearRegression() 1 50.6343 513.494
train_model_cabb8_00051TERMINATED172.31.119.49:1058 262LinearRegression() 1 54.3411 467.511
train_model_cabb8_00052TERMINATED172.31.118.86:1086 202LinearRegression() 1 49.3149 572.614
train_model_cabb8_00053TERMINATED172.31.118.86:1087 107LinearRegression() 1 52.1503 470.489
train_model_cabb8_00054TERMINATED172.31.118.86:1088 50LinearRegression() 1 51.4149 533.831
train_model_cabb8_00055TERMINATED172.31.118.86:1090 42LinearRegression() 1 50.0531 586.92
train_model_cabb8_00056TERMINATED172.31.118.86:1091 205LinearRegression() 1 48.7301 708.579
train_model_cabb8_00057TERMINATED172.31.118.86:1093 137LinearRegression() 1 51.6847 459.818
train_model_cabb8_00058TERMINATED172.31.119.49:1061 80LinearRegression() 1 51.7056 798.612
train_model_cabb8_00059TERMINATED172.31.118.86:1095 114LinearRegression() 1 51.4777 534.058
train_model_cabb8_00060TERMINATED172.31.119.49:1062 43LinearRegression() 1 51.7156 575.949
train_model_cabb8_00061TERMINATED172.31.119.49:1063 181LinearRegression() 1 51.9758 815.745
train_model_cabb8_00062TERMINATED172.31.118.86:1098 255LinearRegression() 1 50.5923 750.948
train_model_cabb8_00063TERMINATED172.31.119.49:1064 7LinearRegression() 1 54.1565 625.271
train_model_cabb8_00064TERMINATED172.31.118.86:1100 126LinearRegression() 1 48.7059 978.726
train_model_cabb8_00065TERMINATED172.31.118.86:1102 89LinearRegression() 1 46.6494 928.499
train_model_cabb8_00066TERMINATED172.31.118.86:1103 220LinearRegression() 1 48.0844 873.963
train_model_cabb8_00067TERMINATED172.31.119.49:1065 133LinearRegression() 1 53.5878 911.989
train_model_cabb8_00068TERMINATED172.31.118.86:1105 209LinearRegression() 1 50.5571 601.737
train_model_cabb8_00069TERMINATED172.31.119.49:1068 152LinearRegression() 1 53.3868 497.51
train_model_cabb8_00070TERMINATED172.31.118.86:1106 160LinearRegression() 1 50.4208 779.68
train_model_cabb8_00071TERMINATED172.31.119.49:1071 41LinearRegression() 1 54.1099 515.513
train_model_cabb8_00072TERMINATED172.31.119.49:1072 144LinearRegression() 1 54.2388 640.862
train_model_cabb8_00073TERMINATED172.31.119.49:1073 211LinearRegression() 1 54.062 595.409
train_model_cabb8_00074TERMINATED172.31.118.86:1110 203LinearRegression() 1 50.185 594.696
train_model_cabb8_00075TERMINATED172.31.119.49:1076 65LinearRegression() 1 53.976 722.254
train_model_cabb8_00076TERMINATED172.31.119.49:1077 20LinearRegression() 1 51.43471175.99
train_model_cabb8_00077TERMINATED172.31.119.49:1078 167LinearRegression() 1 53.6634 852.729
train_model_cabb8_00078TERMINATED172.31.118.86:1113 168LinearRegression() 1 47.1484 665.045
train_model_cabb8_00079TERMINATED172.31.119.49:1079 129LinearRegression() 1 53.886 715.804
train_model_cabb8_00080TERMINATED172.31.119.49:1081 139LinearRegression() 1 53.2424 793.186
train_model_cabb8_00081TERMINATED172.31.92.227:44598 223LinearRegression() 1 36.1454 679.202
train_model_cabb8_00082TERMINATED172.31.92.227:44592 33LinearRegression() 1 37.4595 696.223
train_model_cabb8_00083TERMINATED172.31.107.40:6767 227LinearRegression() 1 37.2531 925.487
train_model_cabb8_00084TERMINATED172.31.107.40:6617 26LinearRegression() 1 37.00971136.9
train_model_cabb8_00085TERMINATED172.31.107.40:6762 36LinearRegression() 1 37.9067 901.902
train_model_cabb8_00086TERMINATED172.31.107.40:6766 217LinearRegression() 1 39.37981447.78
train_model_cabb8_00087TERMINATED172.31.107.40:6554 226LinearRegression() 1 38.636 663.305
train_model_cabb8_00088TERMINATED172.31.107.40:6729 193LinearRegression() 1 38.6808 827.901
train_model_cabb8_00089TERMINATED172.31.107.40:6723 3LinearRegression() 1 37.2992 866.606
train_model_cabb8_00090TERMINATED172.31.107.40:6620 112LinearRegression() 1 38.8521 696.926
train_model_cabb8_00091TERMINATED172.31.107.40:6585 163LinearRegression() 1 41.0831 551.274
train_model_cabb8_00092TERMINATED172.31.107.40:6759 62LinearRegression() 1 36.3662 916.012
train_model_cabb8_00093TERMINATED172.31.92.227:44600 117LinearRegression() 1 49.4763 939.088
train_model_cabb8_00094TERMINATED172.31.92.227:44625 95LinearRegression() 1 46.4456 796.153
train_model_cabb8_00095TERMINATED172.31.92.227:44596 21LinearRegression() 1 51.47011083.62
train_model_cabb8_00096TERMINATED172.31.107.40:6700 261LinearRegression() 1 36.347 739.744
train_model_cabb8_00097TERMINATED172.31.92.227:44608 17LinearRegression() 1 50.6528 836.959
train_model_cabb8_00098TERMINATED172.31.92.227:44618 127LinearRegression() 1 50.5645 807.23
train_model_cabb8_00099TERMINATED172.31.92.227:44613 174LinearRegression() 1 50.753 943.097
train_model_cabb8_00100TERMINATED172.31.92.227:44629 198LinearRegression() 1 50.4779 949.485
train_model_cabb8_00101TERMINATED172.31.92.227:44554 243LinearRegression() 1 51.2144 776.939
train_model_cabb8_00102TERMINATED172.31.92.227:44594 256LinearRegression() 1 47.3136 776.681
train_model_cabb8_00103TERMINATED172.31.116.192:1067 241LinearRegression() 1 32.5447 928.545
train_model_cabb8_00104TERMINATED172.31.119.27:1079 70LinearRegression() 1 50.3317 679.397
train_model_cabb8_00105TERMINATED172.31.119.27:1080 244LinearRegression() 1 50.3704 691.016
train_model_cabb8_00106TERMINATED172.31.119.27:1081 108LinearRegression() 1 50.2333 934.768
train_model_cabb8_00107TERMINATED172.31.119.27:1083 173LinearRegression() 1 49.6945 865.874
train_model_cabb8_00108TERMINATED172.31.119.27:1094 197LinearRegression() 1 50.5883 894.195
train_model_cabb8_00109TERMINATED172.31.119.27:1096 260LinearRegression() 1 51.2387 659.628
train_model_cabb8_00110TERMINATED172.31.119.27:1097 67LinearRegression() 1 50.1553 915.975
train_model_cabb8_00111TERMINATED172.31.119.27:1098 259LinearRegression() 1 50.33111180.61
train_model_cabb8_00112TERMINATED172.31.119.27:1156 208LinearRegression() 1 49.5102 880.512
train_model_cabb8_00113TERMINATED172.31.119.27:1136 72LinearRegression() 1 48.4989 976.283
train_model_cabb8_00114TERMINATED172.31.119.27:1198 52LinearRegression() 1 50.2433 697.535
train_model_cabb8_00115TERMINATED172.31.107.40:6672 40LinearRegression() 1 32.8918 701.698
train_model_cabb8_00116TERMINATED172.31.119.27:1320 230LinearRegression() 1 52.6908 642.217
train_model_cabb8_00117TERMINATED172.31.119.27:1321 73LinearRegression() 1 48.9169 822.855
train_model_cabb8_00118TERMINATED172.31.119.27:1323 216LinearRegression() 1 48.3125 802.704
train_model_cabb8_00119TERMINATED172.31.119.27:1324 37LinearRegression() 1 49.2304 930.469
train_model_cabb8_00120TERMINATED172.31.119.27:1326 228LinearRegression() 1 48.9776 822.599
train_model_cabb8_00121TERMINATED172.31.92.227:44634 10LinearRegression() 1 40.3575 585.042
train_model_cabb8_00122TERMINATED172.31.96.248:1075 14LinearRegression() 1 47.0695 902.867
train_model_cabb8_00123TERMINATED172.31.96.248:1077 225LinearRegression() 1 48.1784 882.167
train_model_cabb8_00124TERMINATED172.31.96.248:1078 179LinearRegression() 1 46.6721 581.68
train_model_cabb8_00125TERMINATED172.31.96.248:1081 83LinearRegression() 1 48.634 689.408
train_model_cabb8_00126TERMINATED172.31.96.248:1084 45LinearRegression() 1 49.1586 644.432
train_model_cabb8_00127TERMINATED172.31.96.248:1085 11LinearRegression() 1 46.62571081.26
train_model_cabb8_00128TERMINATED172.31.96.248:1156 49LinearRegression() 1 48.0473 788.735
train_model_cabb8_00129TERMINATED172.31.96.248:1185 97LinearRegression() 1 49.3603 704.799
train_model_cabb8_00130TERMINATED172.31.96.248:1190 200LinearRegression() 1 48.0185 932.215
train_model_cabb8_00131TERMINATED172.31.96.248:1253 250LinearRegression() 1 47.0236 877.741
train_model_cabb8_00132TERMINATED172.31.96.248:1256 69LinearRegression() 1 47.6892 842.718
train_model_cabb8_00133TERMINATED172.31.109.30:1142 196LinearRegression() 1 49.805 800.332
train_model_cabb8_00134TERMINATED172.31.109.30:1143 178LinearRegression() 1 50.37451102.72
train_model_cabb8_00135TERMINATED172.31.109.30:1144 188LinearRegression() 1 49.7179 909.544
train_model_cabb8_00136TERMINATED172.31.109.30:1146 138LinearRegression() 1 51.6881 904.699
train_model_cabb8_00137TERMINATED172.31.109.30:1148 106LinearRegression() 1 51.3457 815.704
train_model_cabb8_00138TERMINATED172.31.96.248:1864 18LinearRegression() 1 45.49311013.5
train_model_cabb8_00139TERMINATED172.31.109.30:1152 22LinearRegression() 1 51.0672 769.851
train_model_cabb8_00140TERMINATED172.31.109.30:1153 77LinearRegression() 1 50.94451114.61
train_model_cabb8_00141TERMINATED172.31.109.30:1154 169LinearRegression() 1 51.2103 932.17
train_model_cabb8_00142TERMINATED172.31.109.30:1155 132LinearRegression() 1 51.40261301.97
train_model_cabb8_00143TERMINATED172.31.96.248:1866 182LinearRegression() 1 46.6261 997.36
train_model_cabb8_00144TERMINATED172.31.96.248:1867 1LinearRegression() 1 47.82741303.2
train_model_cabb8_00145TERMINATED172.31.109.30:1158 201LinearRegression() 1 49.81021149.16
train_model_cabb8_00146TERMINATED172.31.96.248:1868 28LinearRegression() 1 46.8216 809.239
train_model_cabb8_00147TERMINATED172.31.109.30:1161 159LinearRegression() 1 50.9266 812.47
train_model_cabb8_00148TERMINATED172.31.109.30:1162 222LinearRegression() 1 49.45891042.6
train_model_cabb8_00149TERMINATED172.31.109.30:1163 54LinearRegression() 1 51.0969 749.195
train_model_cabb8_00150TERMINATED172.31.109.30:1164 32LinearRegression() 1 51.2499 937.135
train_model_cabb8_00151TERMINATED172.31.109.30:1166 78LinearRegression() 1 50.74011032.17
train_model_cabb8_00152TERMINATED172.31.109.30:1167 51LinearRegression() 1 49.1364 941.59
train_model_cabb8_00153TERMINATED172.31.96.248:1871 149LinearRegression() 1 47.30031044.11
train_model_cabb8_00154TERMINATED172.31.107.40:6756 218LinearRegression() 1 32.7369 672.288
train_model_cabb8_00155TERMINATED172.31.116.192:1076 60LinearRegression() 1 33.1355 830.98
train_model_cabb8_00156TERMINATED172.31.118.86:1105 183LinearRegression() 1 32.1579 838.233
train_model_cabb8_00157TERMINATED172.31.116.192:1068 55LinearRegression() 1 32.9715 953.51
train_model_cabb8_00158TERMINATED172.31.118.86:1113 157LinearRegression() 1 32.4887 844.536
train_model_cabb8_00159TERMINATED172.31.116.192:1058 212LinearRegression() 1 33.2338 804.003
train_model_cabb8_00160TERMINATED172.31.119.49:1063 175LinearRegression() 1 33.2856 808.665
train_model_cabb8_00161TERMINATED172.31.107.40:6751 147LinearRegression() 1 31.8416 932.495
train_model_cabb8_00162TERMINATED172.31.118.86:1091 119LinearRegression() 1 32.9371 986.148
train_model_cabb8_00163TERMINATED172.31.119.49:1078 235LinearRegression() 1 32.24261023.12
train_model_cabb8_00164TERMINATED172.31.107.40:6729 128LinearRegression() 1 32.9263 651.052
train_model_cabb8_00165TERMINATED172.31.118.86:1102 136LinearRegression() 1 33.0902 890.515
train_model_cabb8_00166TERMINATED172.31.116.192:1062 16LinearRegression() 1 33.0755 945.843
train_model_cabb8_00167TERMINATED172.31.105.11:1067 92LinearRegression() 1 50.5462 798.427
train_model_cabb8_00168TERMINATED172.31.105.11:1068 247LinearRegression() 1 50.1185 871.733
train_model_cabb8_00169TERMINATED172.31.105.11:1069 195LinearRegression() 1 49.7569 794.267
train_model_cabb8_00170TERMINATED172.31.105.11:1070 57LinearRegression() 1 50.0677 904.808
train_model_cabb8_00171TERMINATED172.31.105.11:1071 121LinearRegression() 1 50.8477 768.581
train_model_cabb8_00172TERMINATED172.31.105.11:1072 39LinearRegression() 1 50.57 1017.61
train_model_cabb8_00173TERMINATED172.31.105.11:1073 9LinearRegression() 1 49.40161019.55
train_model_cabb8_00174TERMINATED172.31.105.11:1074 35LinearRegression() 1 50.9128 988.151
train_model_cabb8_00175TERMINATED172.31.105.11:1076 56LinearRegression() 1 49.2694 783.836
train_model_cabb8_00176TERMINATED172.31.105.11:1077 185LinearRegression() 1 51.39441008.99
train_model_cabb8_00177TERMINATED172.31.105.11:1079 248LinearRegression() 1 48.71441024.57
train_model_cabb8_00178TERMINATED172.31.105.11:1242 251LinearRegression() 1 47.67381005.46
train_model_cabb8_00179TERMINATED172.31.105.11:1243 58LinearRegression() 1 49.62321207.97
train_model_cabb8_00180TERMINATED172.31.105.11:1244 190LinearRegression() 1 49.4747 735.263
train_model_cabb8_00181TERMINATED172.31.105.11:1245 177LinearRegression() 1 47.8492 979.182
train_model_cabb8_00182TERMINATED172.31.105.11:1246 86LinearRegression() 1 49.9201 988.628
train_model_cabb8_00183TERMINATED172.31.119.49:1076 171LinearRegression() 1 33.2341 789.881
train_model_cabb8_00184TERMINATED172.31.106.163:934 155LinearRegression() 1 47.1859 968.959
train_model_cabb8_00185TERMINATED172.31.106.163:937 258LinearRegression() 1 47.0077 859.568
train_model_cabb8_00186TERMINATED172.31.106.163:944 153LinearRegression() 1 47.6115 763.152
train_model_cabb8_00187TERMINATED172.31.106.163:959 192LinearRegression() 1 46.4909 660.595
train_model_cabb8_00188TERMINATED172.31.106.163:976 47LinearRegression() 1 48.6412 994.283
train_model_cabb8_00189TERMINATED172.31.106.163:985 91LinearRegression() 1 48.1017 994.602
train_model_cabb8_00190TERMINATED172.31.106.163:990 124LinearRegression() 1 47.2131 881.976
train_model_cabb8_00191TERMINATED172.31.106.163:991 191LinearRegression() 1 47.247 800.369
train_model_cabb8_00192TERMINATED172.31.106.163:992 130LinearRegression() 1 47.5719 765.721
train_model_cabb8_00193TERMINATED172.31.106.163:997 240LinearRegression() 1 46.8987 975.641
train_model_cabb8_00194TERMINATED172.31.106.163:998 94LinearRegression() 1 47.0739 937.527
train_model_cabb8_00195TERMINATED172.31.106.163:1393 122LinearRegression() 1 44.52641093.04
train_model_cabb8_00196TERMINATED172.31.106.163:1397 98LinearRegression() 1 45.5733 890.415
train_model_cabb8_00197TERMINATED172.31.106.163:1401 206LinearRegression() 1 44.69091153.63
train_model_cabb8_00198TERMINATED172.31.106.163:1403 102LinearRegression() 1 45.8035 843.831
train_model_cabb8_00199TERMINATED172.31.106.163:1405 165LinearRegression() 1 45.86061030.5
train_model_cabb8_00200TERMINATED172.31.118.114:963 71LinearRegression() 1 48.3035 819.106
train_model_cabb8_00201TERMINATED172.31.118.114:968 76LinearRegression() 1 49.13321001.59
train_model_cabb8_00202TERMINATED172.31.118.114:978 254LinearRegression() 1 47.1284 932.234
train_model_cabb8_00203TERMINATED172.31.118.114:986 219LinearRegression() 1 48.1927 640.028
train_model_cabb8_00204TERMINATED172.31.118.114:1002 215LinearRegression() 1 48.1183 716.565
train_model_cabb8_00205TERMINATED172.31.118.114:1476 118LinearRegression() 1 49.12551452.28
train_model_cabb8_00206TERMINATED172.31.118.114:1477 134LinearRegression() 1 47.812 832.524
train_model_cabb8_00207TERMINATED172.31.118.114:1478 156LinearRegression() 1 48.2654 924.29
train_model_cabb8_00208TERMINATED172.31.118.114:1479 63LinearRegression() 1 48.576 994.931
train_model_cabb8_00209TERMINATED172.31.118.114:1480 131LinearRegression() 1 48.6662 741.202
train_model_cabb8_00210TERMINATED172.31.118.114:1481 135LinearRegression() 1 48.2067 742.133
train_model_cabb8_00211TERMINATED172.31.118.114:1482 245LinearRegression() 1 49.32491104.59
train_model_cabb8_00212TERMINATED172.31.118.114:1483 180LinearRegression() 1 46.1902 807.311
train_model_cabb8_00213TERMINATED172.31.118.114:1484 242LinearRegression() 1 48.7459 942.116
train_model_cabb8_00214TERMINATED172.31.118.114:1486 252LinearRegression() 1 49.4203 685.934
train_model_cabb8_00215TERMINATED172.31.118.114:1487 93LinearRegression() 1 48.4385 775.18
train_model_cabb8_00216TERMINATED172.31.119.49:1065 64LinearRegression() 1 33.11691008.35
train_model_cabb8_00217TERMINATED172.31.116.192:1078 85LinearRegression() 1 32.6723 942.897
train_model_cabb8_00218TERMINATED172.31.107.40:6585 29LinearRegression() 1 33.251 998.627
train_model_cabb8_00219TERMINATED172.31.116.192:1070 15LinearRegression() 1 32.8543 750.73
train_model_cabb8_00220TERMINATED172.31.119.49:1077 176LinearRegression() 1 32.06731162.88
train_model_cabb8_00221TERMINATED172.31.116.192:1064 31LinearRegression() 1 33.4088 861.811
train_model_cabb8_00222TERMINATED172.31.92.227:44603 34LinearRegression() 1 34.9007 788.634
train_model_cabb8_00223TERMINATED172.31.109.30:1142 210LinearRegression() 1 32.90051112.7
train_model_cabb8_00224TERMINATED172.31.92.227:44625 81LinearRegression() 1 36.5566 937.172
train_model_cabb8_00225TERMINATED172.31.107.40:6763 194LinearRegression() 1 32.9275 850.925
train_model_cabb8_00226TERMINATED172.31.111.195:1113 120LinearRegression() 1 48.4823 621.424
train_model_cabb8_00227TERMINATED172.31.111.195:1114 172LinearRegression() 1 50.418 836.129
train_model_cabb8_00228TERMINATED172.31.111.195:1115 19LinearRegression() 1 49.9108 872.502
train_model_cabb8_00229TERMINATED172.31.111.195:1116 150LinearRegression() 1 49.2143 930.933
train_model_cabb8_00230TERMINATED172.31.111.195:1118 38LinearRegression() 1 49.8773 820.518
train_model_cabb8_00231TERMINATED172.31.111.195:1120 111LinearRegression() 1 49.721 1101.47
train_model_cabb8_00232TERMINATED172.31.111.195:1125 214LinearRegression() 1 50.0559 928.928
train_model_cabb8_00233TERMINATED172.31.111.195:1126 123LinearRegression() 1 49.14341014.79
train_model_cabb8_00234TERMINATED172.31.111.195:1127 46LinearRegression() 1 49.8768 934.841
train_model_cabb8_00235TERMINATED172.31.111.195:1128 101LinearRegression() 1 49.2863 949.496
train_model_cabb8_00236TERMINATED172.31.111.195:1129 204LinearRegression() 1 50.31951895.21
train_model_cabb8_00237TERMINATED172.31.111.195:1130 27LinearRegression() 1 49.2731 990.916
train_model_cabb8_00238TERMINATED172.31.111.195:1131 115LinearRegression() 1 50.1338 858.027
train_model_cabb8_00239TERMINATED172.31.111.195:1132 53LinearRegression() 1 48.0136 959.43
train_model_cabb8_00240TERMINATED172.31.111.195:1133 44LinearRegression() 1 49.93291290.55
train_model_cabb8_00241TERMINATED172.31.111.195:1134 8LinearRegression() 1 49.5804 810.876
train_model_cabb8_00242TERMINATED172.31.92.227:44622 5LinearRegression() 1 34.77361417.66
train_model_cabb8_00243TERMINATED172.31.121.243:1279 109LinearRegression() 1 49.10191179.34
train_model_cabb8_00244TERMINATED172.31.104.216:1059 23LinearRegression() 1 48.28051262.11
train_model_cabb8_00245TERMINATED172.31.121.243:1280 59LinearRegression() 1 49.93881747.95
train_model_cabb8_00246TERMINATED172.31.104.216:1060 221LinearRegression() 1 49.84021158.98
train_model_cabb8_00247TERMINATED172.31.104.216:1063 6LinearRegression() 1 48.675 1872.14
train_model_cabb8_00248TERMINATED172.31.104.216:1064 253LinearRegression() 1 49.4454 414.218
train_model_cabb8_00249TERMINATED172.31.121.243:1283 96LinearRegression() 1 49.44871042.94
train_model_cabb8_00250TERMINATED172.31.104.216:1065 84LinearRegression() 1 48.4972 814.855
train_model_cabb8_00251TERMINATED172.31.104.216:1066 207LinearRegression() 1 49.7393 396.891
train_model_cabb8_00252TERMINATED172.31.121.243:1284 154LinearRegression() 1 49.062 1286.36
train_model_cabb8_00253TERMINATED172.31.121.243:1286 187LinearRegression() 1 48.80351612.05
train_model_cabb8_00254TERMINATED172.31.121.243:1287 184LinearRegression() 1 50.0827 623.42
train_model_cabb8_00255TERMINATED172.31.104.216:1070 30LinearRegression() 1 48.305 1730.94
train_model_cabb8_00256TERMINATED172.31.104.216:1072 105LinearRegression() 1 48.6268 425.296
train_model_cabb8_00257TERMINATED172.31.104.216:1073 2LinearRegression() 1 49.9253 661.768
train_model_cabb8_00258TERMINATED172.31.104.216:1074 199LinearRegression() 1 49.0089 256.422
train_model_cabb8_00259TERMINATED172.31.121.243:1291 24DecisionTreeReg_e430 1 51.0844 414.616
train_model_cabb8_00260TERMINATED172.31.121.243:1292 140DecisionTreeReg_a0a0 1 52.9191 444.062
train_model_cabb8_00261TERMINATED172.31.104.216:1076 141DecisionTreeReg_8f10 1 53.5299 425.023
train_model_cabb8_00262TERMINATED172.31.104.216:1078 257DecisionTreeReg_8ee0 1 50.6255 816.242
train_model_cabb8_00263TERMINATED172.31.104.216:1079 239DecisionTreeReg_8df0 1 52.3577 409.536
train_model_cabb8_00264TERMINATED172.31.121.243:1293 143DecisionTreeReg_8490 1 53.0278 409.6
train_model_cabb8_00265TERMINATED172.31.121.243:1294 238DecisionTreeReg_8580 1 51.8036 384.027
train_model_cabb8_00266TERMINATED172.31.121.243:1296 170DecisionTreeReg_a5b0 1 54.2154 469.844
train_model_cabb8_00267TERMINATED172.31.121.243:1297 229DecisionTreeReg_5040 1 53.3582 478.265
train_model_cabb8_00268TERMINATED172.31.104.216:1082 113DecisionTreeReg_5e50 1 51.8609 435.744
train_model_cabb8_00269TERMINATED172.31.104.216:1084 224DecisionTreeReg_5d90 1 50.3488 447.204
train_model_cabb8_00270TERMINATED172.31.121.243:1303 79DecisionTreeReg_5ca0 1 53.598 508.675
train_model_cabb8_00271TERMINATED172.31.121.243:1304 100DecisionTreeReg_56a0 1 53.1622 573.324
train_model_cabb8_00272TERMINATED172.31.121.243:1305 189DecisionTreeReg_5070 1 48.5575 775.215
train_model_cabb8_00273TERMINATED172.31.104.216:1089 236DecisionTreeReg_54c0 1 54.2484 369.222
train_model_cabb8_00274TERMINATED172.31.121.243:1306 162DecisionTreeReg_58b0 1 54.2835 497.537
train_model_cabb8_00275TERMINATED172.31.109.30:1152 61DecisionTreeReg_5dc0 1 32.6102 825.518
train_model_cabb8_00276TERMINATED172.31.107.40:6723 75DecisionTreeReg_5b80 1 35.1165 430.031
train_model_cabb8_00277TERMINATED172.31.118.86:1088 233DecisionTreeReg_5850 1 36.6956 490.047
train_model_cabb8_00278TERMINATED172.31.118.86:1106 125DecisionTreeReg_5af0 1 33.3176 505.643
train_model_cabb8_00279TERMINATED172.31.119.49:1071 146DecisionTreeReg_5a30 1 31.8151 628.414
train_model_cabb8_00280TERMINATED172.31.106.163:990 234DecisionTreeReg_59a0 1 36.5456 440.234
train_model_cabb8_00281TERMINATED172.31.118.86:1103 263DecisionTreeReg_5310 1 37.3298 404.523
train_model_cabb8_00282TERMINATED172.31.96.248:1864 237DecisionTreeReg_5640 1 37.3657 407.959
train_model_cabb8_00283TERMINATED172.31.107.40:6620 166DecisionTreeReg_5df0 1 34.1855 436.852
train_model_cabb8_00284TERMINATED172.31.107.40:6754 249DecisionTreeReg_8190 1 35.7017 450.778
train_model_cabb8_00285TERMINATED172.31.118.86:1090 151DecisionTreeReg_8040 1 34.2167 417.852
train_model_cabb8_00286TERMINATED172.31.118.86:1086 231DecisionTreeReg_8460 1 36.1682 548.66
train_model_cabb8_00287TERMINATED172.31.107.40:6700 116DecisionTreeReg_82e0 1 32.2298 615.046
train_model_cabb8_00288TERMINATED172.31.119.27:1321 145DecisionTreeReg_8d00 1 33.9747 589.76
train_model_cabb8_00289TERMINATED172.31.115.72:1070 82DecisionTreeReg_81f0 1 51.0629 770.287
train_model_cabb8_00290TERMINATED172.31.115.72:1071 164DecisionTreeReg_8b50 1 54.5304 547.395
train_model_cabb8_00291TERMINATED172.31.115.72:1072 68DecisionTreeReg_8ca0 1 55.5587 503.215
train_model_cabb8_00292TERMINATED172.31.115.72:1073 161DecisionTreeReg_8cd0 1 56.712 508.438
train_model_cabb8_00293TERMINATED172.31.115.72:1074 48DecisionTreeReg_8310 1 55.0343 570.651
train_model_cabb8_00294TERMINATED172.31.115.72:1075 90DecisionTreeReg_8ee0 1 54.849 452.535
train_model_cabb8_00295TERMINATED172.31.115.72:1076 13DecisionTreeReg_8580 1 51.408 605.479
train_model_cabb8_00296TERMINATED172.31.115.72:1077 148DecisionTreeReg_8730 1 50.5008 631.479
train_model_cabb8_00297TERMINATED172.31.115.72:1078 88DecisionTreeReg_8ee0 1 54.0028 691.318
train_model_cabb8_00298TERMINATED172.31.115.72:1146 87DecisionTreeReg_88e0 1 53.7769 603.513
train_model_cabb8_00299TERMINATED172.31.115.72:1172 213DecisionTreeReg_88b0 1 51.6106 959.97
train_model_cabb8_00300TERMINATED172.31.115.72:1249 74DecisionTreeReg_82b0 1 52.4914 464.584
train_model_cabb8_00301TERMINATED172.31.115.72:1206 25DecisionTreeReg_8580 1 52.037 701.172
train_model_cabb8_00302TERMINATED172.31.115.72:1230 158DecisionTreeReg_8610 1 51.4124 518.324
train_model_cabb8_00303TERMINATED172.31.115.72:1536 246DecisionTreeReg_8bb0 1 53.6803 498.436
train_model_cabb8_00304TERMINATED172.31.115.72:1544 12DecisionTreeReg_8fa0 1 50.1018 787.053
train_model_cabb8_00305TERMINATED172.31.96.248:1156 142DecisionTreeReg_8250 1 35.8701 415.325
train_model_cabb8_00306TERMINATED172.31.116.192:1076 66DecisionTreeReg_87c0 1 33.0199 662.404
train_model_cabb8_00307TERMINATED172.31.119.49:1076 186DecisionTreeReg_8b50 1 37.2903 563.372
train_model_cabb8_00308TERMINATED172.31.98.34:1031 232DecisionTreeReg_8490 1 49.5385 561.19
train_model_cabb8_00309TERMINATED172.31.98.34:1034 4DecisionTreeReg_85e0 1 49.8786 511.066
train_model_cabb8_00310TERMINATED172.31.98.34:1036 262DecisionTreeReg_8400 1 51.2088 419.804
train_model_cabb8_00311TERMINATED172.31.98.34:1040 202DecisionTreeReg_fc70 1 48.4846 610.746
train_model_cabb8_00312TERMINATED172.31.98.34:1053 107DecisionTreeReg_f4c0 1 52.0531 442.141
train_model_cabb8_00313TERMINATED172.31.98.34:1072 50DecisionTreeReg_f820 1 50.6918 512.765
train_model_cabb8_00314TERMINATED172.31.98.34:1073 42DecisionTreeReg_ff70 1 50.7785 571.675
train_model_cabb8_00315TERMINATED172.31.98.34:1075 205DecisionTreeReg_f520 1 49.5169 668.283
train_model_cabb8_00316TERMINATED172.31.98.34:1076 137DecisionTreeReg_f190 1 50.4805 441.492
train_model_cabb8_00317TERMINATED172.31.98.34:1077 80DecisionTreeReg_f4f0 1 47.4095 819.696
train_model_cabb8_00318TERMINATED172.31.98.34:1079 114DecisionTreeReg_f640 1 51.3808 481.299
train_model_cabb8_00319TERMINATED172.31.98.34:1081 43DecisionTreeReg_3130 1 50.6762 553.965
train_model_cabb8_00320TERMINATED172.31.98.34:1082 181DecisionTreeReg_3070 1 48.3471 760.58
train_model_cabb8_00321TERMINATED172.31.119.49:1068 255DecisionTreeReg_3280 1 34.1084 745.999
train_model_cabb8_00322TERMINATED172.31.125.89:1199 7DecisionTreeReg_3550 1 50.7769 599.243
train_model_cabb8_00323TERMINATED172.31.125.89:1200 126DecisionTreeReg_3490 1 51.136 927.041
train_model_cabb8_00324TERMINATED172.31.125.89:1201 89DecisionTreeReg_3910 1 50.7935 922.866
train_model_cabb8_00325TERMINATED172.31.98.34:1549 220DecisionTreeReg_3640 1 46.2663 900.319
train_model_cabb8_00326TERMINATED172.31.125.89:1202 133DecisionTreeReg_37c0 1 48.4677 910.578
train_model_cabb8_00327TERMINATED172.31.125.89:1203 209DecisionTreeReg_36a0 1 50.9914 614.539
train_model_cabb8_00328TERMINATED172.31.125.89:1205 152DecisionTreeReg_3ca0 1 50.4792 510.852
train_model_cabb8_00329TERMINATED172.31.125.89:1207 160DecisionTreeReg_3f10 1 49.8452 769.241
train_model_cabb8_00330TERMINATED172.31.125.89:1209 41DecisionTreeReg_3d30 1 51.3509 486.979
train_model_cabb8_00331TERMINATED172.31.125.89:1210 144DecisionTreeReg_3fd0 1 51.4809 600.674
train_model_cabb8_00332TERMINATED172.31.125.89:1212 211DecisionTreeReg_37f0 1 52.1561 538.313
train_model_cabb8_00333TERMINATED172.31.125.89:1214 203DecisionTreeReg_3be0 1 50.7643 632.727
train_model_cabb8_00334TERMINATED172.31.125.89:1215 65DecisionTreeReg_f040 1 50.7976 669.305
train_model_cabb8_00335TERMINATED172.31.98.34:1555 20DecisionTreeReg_f6a0 1 46.29331059.28
train_model_cabb8_00336TERMINATED172.31.125.89:1217 167DecisionTreeReg_f640 1 50.6441 968.194
train_model_cabb8_00337TERMINATED172.31.125.89:1219 168DecisionTreeReg_fb50 1 50.5313 691.806
train_model_cabb8_00338TERMINATED172.31.125.89:1221 129DecisionTreeReg_f8e0 1 49.0513 696.27
train_model_cabb8_00339TERMINATED172.31.98.34:1578 139DecisionTreeReg_fc40 1 45.6642 802.308
train_model_cabb8_00340TERMINATED172.31.125.89:1224 223DecisionTreeReg_fd60 1 51.3457 612.787
train_model_cabb8_00341TERMINATED172.31.119.49:1061 33DecisionTreeReg_f430 1 34.9701 631.292
train_model_cabb8_00342TERMINATED172.31.116.192:1078 227DecisionTreeReg_f850 1 31.8538 860.138
train_model_cabb8_00343TERMINATED172.31.107.40:6672 26DecisionTreeReg_f9d0 1 32.38941241.46
train_model_cabb8_00344TERMINATED172.31.119.49:1058 36DecisionTreeReg_fbb0 1 32.4574 909.991
train_model_cabb8_00345TERMINATED172.31.105.11:1076 217DecisionTreeReg_f5e0 1 32.23711326.45
train_model_cabb8_00346TERMINATED172.31.119.27:1324 226DecisionTreeReg_f520 1 33.7602 630.02
train_model_cabb8_00347TERMINATED172.31.106.163:992 193DecisionTreeReg_ba00 1 32.7073 491.831
train_model_cabb8_00348TERMINATED172.31.107.40:6617 3DecisionTreeReg_b400 1 32.791 898.735
train_model_cabb8_00349TERMINATED172.31.105.11:1242 112DecisionTreeReg_bbe0 1 34.2929 669.428
train_model_cabb8_00350TERMINATED172.31.119.27:1326 163DecisionTreeReg_bd30 1 36.9837 533.394
train_model_cabb8_00351TERMINATED172.31.106.228:1066 62DecisionTreeReg_bd60 1 49.9663 922.454
train_model_cabb8_00352TERMINATED172.31.106.228:1067 117DecisionTreeReg_bb20 1 50.12561048.96
train_model_cabb8_00353TERMINATED172.31.106.228:1068 95DecisionTreeReg_b5b0 1 50.1601 807.024
train_model_cabb8_00354TERMINATED172.31.106.228:1070 21DecisionTreeReg_be20 1 50.2068 937.808
train_model_cabb8_00355TERMINATED172.31.106.228:1071 261DecisionTreeReg_b0a0 1 50.1866 675.821
train_model_cabb8_00356TERMINATED172.31.106.228:1072 17DecisionTreeReg_b700 1 48.6767 811.843
train_model_cabb8_00357TERMINATED172.31.106.228:1073 127DecisionTreeReg_ba60 1 47.0807 784.407
train_model_cabb8_00358TERMINATED172.31.106.228:1074 174DecisionTreeReg_b970 1 49.973 807.162
train_model_cabb8_00359TERMINATED172.31.106.228:1075 198DecisionTreeReg_b7f0 1 50.0473 873.888
train_model_cabb8_00360TERMINATED172.31.106.228:1076 243DecisionTreeReg_ea00 1 48.1508 767.796
train_model_cabb8_00361TERMINATED172.31.106.228:1078 256DecisionTreeReg_e400 1 50.2215 743.98
train_model_cabb8_00362TERMINATED172.31.106.228:1081 241DecisionTreeReg_e820 1 50.1994 913.213
train_model_cabb8_00363TERMINATED172.31.106.228:1083 70DecisionTreeReg_e910 1 50.0698 692.288
train_model_cabb8_00364TERMINATED172.31.106.228:1085 244DecisionTreeReg_e790 1 50.3275 666.494
train_model_cabb8_00365TERMINATED172.31.106.228:1124 108DecisionTreeReg_e430 1 49.7349 972.719
train_model_cabb8_00366TERMINATED172.31.106.228:1565 173DecisionTreeReg_e7f0 1 46.8308 873.205
train_model_cabb8_00367TERMINATED172.31.92.227:44629 197DecisionTreeReg_e9d0 1 33.9419 860.099
train_model_cabb8_00368TERMINATED172.31.119.27:1083 260DecisionTreeReg_ee20 1 35.8479 627.511
train_model_cabb8_00369TERMINATED172.31.104.20:1055 67DecisionTreeReg_e760 1 53.5788 938.445
train_model_cabb8_00370TERMINATED172.31.121.30:1088 259DecisionTreeReg_e070 1 51.01271114.09
train_model_cabb8_00371TERMINATED172.31.104.20:1056 208DecisionTreeReg_efa0 1 53.495 1062.66
train_model_cabb8_00372TERMINATED172.31.121.30:1089 72DecisionTreeReg_e2b0 1 50.80331095.56
train_model_cabb8_00373TERMINATED172.31.104.20:1059 52DecisionTreeReg_ec70 1 53.1986 609.41
train_model_cabb8_00374TERMINATED172.31.104.20:1060 40DecisionTreeReg_fd00 1 53.5511 638.096
train_model_cabb8_00375TERMINATED172.31.121.30:1091 230DecisionTreeReg_fdf0 1 55.37 621.417
train_model_cabb8_00376TERMINATED172.31.104.20:1061 73DecisionTreeReg_f850 1 52.4333 849.599
train_model_cabb8_00377TERMINATED172.31.121.30:1094 216DecisionTreeReg_f880 1 49.9072 808.165
train_model_cabb8_00378TERMINATED172.31.121.30:1095 37DecisionTreeReg_f610 1 50.0965 841.435
train_model_cabb8_00379TERMINATED172.31.121.30:1097 228DecisionTreeReg_f8b0 1 50.243 780.833
train_model_cabb8_00380TERMINATED172.31.104.20:1062 10DecisionTreeReg_f4c0 1 51.5568 617.891
train_model_cabb8_00381TERMINATED172.31.121.30:1100 14DecisionTreeReg_f280 1 50.6933 862.727
train_model_cabb8_00382TERMINATED172.31.121.30:1104 225DecisionTreeReg_fa00 1 51.5366 800.617
train_model_cabb8_00383TERMINATED172.31.104.20:1066 179DecisionTreeReg_f2b0 1 52.1077 555.392
train_model_cabb8_00384TERMINATED172.31.104.20:1067 83DecisionTreeReg_2af0 1 53.0032 653.833
train_model_cabb8_00385TERMINATED172.31.121.30:1124 45DecisionTreeReg_22b0 1 50.8869 625.919
train_model_cabb8_00386TERMINATED172.31.121.30:1127 11DecisionTreeReg_20d0 1 50.36021153.89
train_model_cabb8_00387TERMINATED172.31.104.20:1069 49DecisionTreeReg_24c0 1 53.4681 729.803
train_model_cabb8_00388TERMINATED172.31.104.20:1070 97DecisionTreeReg_2e20 1 51.7646 666.928
train_model_cabb8_00389TERMINATED172.31.104.20:1072 200DecisionTreeReg_27f0 1 53.6136 946.049
train_model_cabb8_00390TERMINATED172.31.121.30:1164 250DecisionTreeReg_2340 1 50.76171024.23
train_model_cabb8_00391TERMINATED172.31.104.20:1074 69DecisionTreeReg_8130 1 53.6614 851.655
train_model_cabb8_00392TERMINATED172.31.104.20:1076 196DecisionTreeReg_8ca0 1 53.6386 842.145
train_model_cabb8_00393TERMINATED172.31.104.20:1078 178DecisionTreeReg_8b20 1 52.84381244.88
train_model_cabb8_00394TERMINATED172.31.121.30:1177 188DecisionTreeReg_8be0 1 51.2723 869.705
train_model_cabb8_00395TERMINATED172.31.121.30:1208 138DecisionTreeReg_8a90 1 53.6519 903.461
train_model_cabb8_00396TERMINATED172.31.104.20:1079 106DecisionTreeReg_8c40 1 53.026 753.516
train_model_cabb8_00397TERMINATED172.31.104.20:1080 18DecisionTreeReg_8e50 1 51.5216 991.263
train_model_cabb8_00398TERMINATED172.31.121.30:1244 22DecisionTreeReg_8d90 1 51.2588 830.333
train_model_cabb8_00399TERMINATED172.31.121.30:1312 77DecisionTreeReg_86d0 1 49.929 997.54
train_model_cabb8_00400TERMINATED172.31.121.30:1313 169DecisionTreeReg_8430 1 50.8829 921.847
train_model_cabb8_00401TERMINATED172.31.121.243:1292 132DecisionTreeReg_8880 1 35.56291296.03
train_model_cabb8_00402TERMINATED172.31.92.227:44554 182DecisionTreeReg_a700 1 37.9256 923.831
train_model_cabb8_00403TERMINATED172.31.121.243:1279 1DecisionTreeReg_a550 1 35.27491325.21
train_model_cabb8_00404TERMINATED172.31.115.72:1544 201DecisionTreeReg_a520 1 34.1216 964.887
train_model_cabb8_00405TERMINATED172.31.119.49:1063 28DecisionTreeReg_a370 1 33.2472 850.702
train_model_cabb8_00406TERMINATED172.31.107.40:6766 159DecisionTreeReg_a340 1 33.4717 724.768
train_model_cabb8_00407TERMINATED172.31.104.216:1074 222DecisionTreeReg_a190 1 32.772 1016.22
train_model_cabb8_00408TERMINATED172.31.121.243:1306 54DecisionTreeReg_a160 1 32.9669 702.536
train_model_cabb8_00409TERMINATED172.31.104.216:1064 32DecisionTreeReg_ad60 1 33.0108 957.159
train_model_cabb8_00410TERMINATED172.31.118.86:1095 78DecisionTreeReg_adf0 1 32.91851164.2
train_model_cabb8_00411TERMINATED172.31.111.195:1116 51DecisionTreeReg_ac70 1 33.4635 972.669
train_model_cabb8_00412TERMINATED172.31.109.30:1162 149DecisionTreeReg_afd0 1 34.30631067.38
train_model_cabb8_00413TERMINATED172.31.92.227:44616 218DecisionTreeReg_a970 1 34.9227 613.665
train_model_cabb8_00414TERMINATED172.31.121.243:1284 60DecisionTreeReg_aac0 1 33.0206 950.157
train_model_cabb8_00415TERMINATED172.31.96.248:1081 183DecisionTreeReg_a9d0 1 33.227 749.517
train_model_cabb8_00416TERMINATED172.31.105.11:1245 55DecisionTreeReg_3820 1 32.8945 945.049
train_model_cabb8_00417TERMINATED172.31.109.30:1155 157DecisionTreeReg_3970 1 33.9339 799.787
train_model_cabb8_00418TERMINATED172.31.106.163:997 212DecisionTreeReg_3640 1 33.6757 826.981
train_model_cabb8_00419TERMINATED172.31.92.227:44592 175DecisionTreeReg_35b0 1 36.3665 757.239
train_model_cabb8_00420TERMINATED172.31.118.86:1087 147DecisionTreeReg_36d0 1 33.5513 865.223
train_model_cabb8_00421TERMINATED172.31.98.34:1034 119DecisionTreeReg_3130 1 33.555 999.508
train_model_cabb8_00422TERMINATED172.31.104.216:1073 235DecisionTreeReg_3250 1 32.74321089.96
train_model_cabb8_00423TERMINATED172.31.96.248:1084 128DecisionTreeReg_3280 1 33.844 711.885
train_model_cabb8_00424TERMINATED172.31.107.40:6554 136DecisionTreeReg_3340 1 34.32681053.91
train_model_cabb8_00425TERMINATED172.31.111.195:1131 16DecisionTreeReg_32e0 1 32.778 827.77
train_model_cabb8_00426TERMINATED172.31.96.248:1077 92DecisionTreeReg_1ca0 1 33.8468 714.269
train_model_cabb8_00427TERMINATED172.31.96.248:1866 247DecisionTreeReg_16a0 1 32.854 801.326
train_model_cabb8_00428TERMINATED172.31.119.49:1057 195DecisionTreeReg_1580 1 33.7739 702.233
train_model_cabb8_00429TERMINATED172.31.116.192:1071 57DecisionTreeReg_19a0 1 32.8017 839.538
train_model_cabb8_00430TERMINATED172.31.119.27:1098 121DecisionTreeReg_1220 1 33.1946 753.322
train_model_cabb8_00431TERMINATED172.31.109.30:1167 39DecisionTreeReg_1070 1 34.1304 936.22
train_model_cabb8_00432TERMINATED172.31.119.49:1081 9DecisionTreeReg_1ac0 1 34.4617 832.906
train_model_cabb8_00433TERMINATED172.31.92.227:44606 35DecisionTreeReg_1cd0 1 35.4088 953.349
train_model_cabb8_00434TERMINATED172.31.92.227:44600 56DecisionTreeReg_1250 1 36.1216 849.461
train_model_cabb8_00435TERMINATED172.31.125.89:1210 185DecisionTreeReg_1df0 1 33.99141012.25
train_model_cabb8_00436TERMINATED172.31.107.40:6700 248DecisionTreeReg_1d00 1 32.21841013.71
train_model_cabb8_00437TERMINATED172.31.106.163:990 251DecisionTreeReg_1e20 1 34.0459 973.115
train_model_cabb8_00438TERMINATED172.31.115.72:1078 58DecisionTreeReg_b040 1 34.23721273.44
train_model_cabb8_00439TERMINATED172.31.119.27:1097 190DecisionTreeReg_bf40 1 33.5153 744.381
train_model_cabb8_00440TERMINATED172.31.105.11:1244 177DecisionTreeReg_bd60 1 34.14291010.4
train_model_cabb8_00441TERMINATED172.31.118.86:1105 86DecisionTreeReg_bfa0 1 32.57211139.62
train_model_cabb8_00442TERMINATED172.31.118.86:1098 171DecisionTreeReg_b160 1 32.322 771.109
train_model_cabb8_00443TERMINATED172.31.115.72:1077 155DecisionTreeReg_b190 1 32.8353 966.38
train_model_cabb8_00444TERMINATED172.31.118.114:1483 258DecisionTreeReg_bf70 1 33.4288 926.472
train_model_cabb8_00445TERMINATED172.31.118.114:978 153DecisionTreeReg_b6a0 1 32.9322 694.658
train_model_cabb8_00446TERMINATED172.31.114.169:1088 192DecisionTreeReg_b790 1 48.4651 730.849
train_model_cabb8_00447TERMINATED172.31.114.169:1090 47DecisionTreeReg_b7f0 1 48.92351068.56
train_model_cabb8_00448TERMINATED172.31.126.57:1085 91DecisionTreeReg_b820 1 48.8607 960.325
train_model_cabb8_00449TERMINATED172.31.114.169:1092 124DecisionTreeReg_b0a0 1 48.914 871.701
train_model_cabb8_00450TERMINATED172.31.110.8:1089 191DecisionTreeReg_a7f0 1 48.5716 906.066
train_model_cabb8_00451TERMINATED172.31.110.8:1090 130DecisionTreeReg_4dc0 1 48.057 745.201
train_model_cabb8_00452TERMINATED172.31.126.57:1086 240DecisionTreeReg_4fa0 1 46.815 966.075
train_model_cabb8_00453TERMINATED172.31.114.169:1094 94DecisionTreeReg_4c40 1 50.88071005.56
train_model_cabb8_00454TERMINATED172.31.110.8:1091 122DecisionTreeReg_47c0 1 48.7351 690.197
train_model_cabb8_00455TERMINATED172.31.114.169:1098 98DecisionTreeReg_49a0 1 48.8054 933.612
train_model_cabb8_00456TERMINATED172.31.114.169:1191 206DecisionTreeReg_44c0 1 49.17021150.56
train_model_cabb8_00457TERMINATED172.31.126.57:1089 102DecisionTreeReg_48b0 1 48.7229 988.922
train_model_cabb8_00458TERMINATED172.31.126.57:1090 165DecisionTreeReg_46a0 1 47.88051028.9
train_model_cabb8_00459TERMINATED172.31.110.8:1094 71DecisionTreeReg_4490 1 48.8297 828.414
train_model_cabb8_00460TERMINATED172.31.126.57:1092 76DecisionTreeReg_e370 1 48.2467 948.744
train_model_cabb8_00461TERMINATED172.31.114.169:1313 254DecisionTreeReg_efd0 1 49.324 903.878
train_model_cabb8_00462TERMINATED172.31.126.57:1096 219DecisionTreeReg_ef40 1 46.4076 628.484
train_model_cabb8_00463TERMINATED172.31.126.57:1097 215DecisionTreeReg_ed90 1 48.4685 677.17
train_model_cabb8_00464TERMINATED172.31.126.57:1099 118DecisionTreeReg_ed60 1 48.25791194.78
train_model_cabb8_00465TERMINATED172.31.110.8:1098 134DecisionTreeReg_ebb0 1 48.6911 865.457
train_model_cabb8_00466TERMINATED172.31.110.8:1099 156DecisionTreeReg_eb80 1 48.0192 952.08
train_model_cabb8_00467TERMINATED172.31.114.169:1204 63DecisionTreeReg_e9d0 1 49.4127 942.023
train_model_cabb8_00468TERMINATED172.31.126.57:1104 131DecisionTreeReg_e9a0 1 48.3871 863.691
train_model_cabb8_00469TERMINATED172.31.110.8:1195 135DecisionTreeReg_e7f0 1 47.5413 764.074
train_model_cabb8_00470TERMINATED172.31.114.169:1250 245DecisionTreeReg_e7c0 1 49.25 850.32
train_model_cabb8_00471TERMINATED172.31.110.8:1196 180DecisionTreeReg_e610 1 47.8197 901.235
train_model_cabb8_00472TERMINATED172.31.114.169:1238 242DecisionTreeReg_e5e0 1 48.9312 904.826
train_model_cabb8_00473TERMINATED172.31.126.57:1146 252DecisionTreeReg_e1f0 1 49.4064 743.634
train_model_cabb8_00474TERMINATED172.31.110.8:1231 93DecisionTreeReg_e220 1 47.8849 754.058
train_model_cabb8_00475TERMINATED172.31.126.57:1172 64DecisionTreeReg_41c0 1 47.2468 925.271
train_model_cabb8_00476TERMINATED172.31.126.57:1216 85DecisionTreeReg_4c70 1 48.0952 877.888
train_model_cabb8_00477TERMINATED172.31.126.57:1225 29DecisionTreeReg_44c0 1 49.11241057.58
train_model_cabb8_00478TERMINATED172.31.110.8:1324 15DecisionTreeReg_4880 1 47.1913 825.355
train_model_cabb8_00479TERMINATED172.31.110.8:1257 176DecisionTreeReg_4d90 1 46.75141199.29
train_model_cabb8_00480TERMINATED172.31.110.8:1276 31DecisionTreeReg_43a0 1 47.323 961.614
train_model_cabb8_00481TERMINATED172.31.114.169:1340 34DecisionTreeReg_4670 1 49.7605 730.517
train_model_cabb8_00482TERMINATED172.31.114.169:1395 210DecisionTreeReg_4760 1 48.9382 919.248
train_model_cabb8_00483TERMINATED172.31.114.169:1399 81DecisionTreeReg_4610 1 49.013 799.991
train_model_cabb8_00484TERMINATED172.31.110.8:1414 194DecisionTreeReg_4f10 1 47.4597 749.128
train_model_cabb8_00485TERMINATED172.31.126.57:1337 120DecisionTreeReg_45e0 1 46.946 874.877
train_model_cabb8_00486TERMINATED172.31.110.8:1415 172DecisionTreeReg_4d30 1 48.47471112.92
train_model_cabb8_00487TERMINATED172.31.110.8:1416 19DecisionTreeReg_4640 1 48.0182 874.38
train_model_cabb8_00488TERMINATED172.31.114.169:1412 150DecisionTreeReg_4af0 1 48.91051098.25
train_model_cabb8_00489TERMINATED172.31.114.169:1417 38DecisionTreeReg_4790 1 49.2924 919.617
train_model_cabb8_00490TERMINATED172.31.110.8:1417 111DecisionTreeReg_4ca0 1 48.026 1229.93
train_model_cabb8_00491TERMINATED172.31.114.169:1430 214DecisionTreeReg_4f40 1 48.77911159.92
train_model_cabb8_00492TERMINATED172.31.126.57:1339 123DecisionTreeReg_c130 1 48.35731072.06
train_model_cabb8_00493TERMINATED172.31.92.227:44598 46DecisionTreeReg_ca30 1 34.0773 933.92
train_model_cabb8_00494TERMINATED172.31.126.57:1966 101DecisionTreeReg_cdc0 1 46.7343 967.935
train_model_cabb8_00495TERMINATED172.31.106.228:1078 204DecisionTreeReg_cd90 1 32.9643 903.251
train_model_cabb8_00496TERMINATED172.31.116.192:1070 27DecisionTreeReg_c400 1 32.36161218.45
train_model_cabb8_00497TERMINATED172.31.119.49:1061 115DecisionTreeReg_c610 1 32.82541185.13
train_model_cabb8_00498TERMINATED172.31.116.192:1058 53DecisionTreeReg_c7c0 1 32.76711042.85
train_model_cabb8_00499TERMINATED172.31.109.30:1161 44DecisionTreeReg_c790 1 34.47 1873.76
train_model_cabb8_00500TERMINATED172.31.105.11:1072 8DecisionTreeReg_cd00 1 35.1452 411.63
train_model_cabb8_00501TERMINATED172.31.119.27:1323 5DecisionTreeReg_c5e0 1 33.93281735.59
train_model_cabb8_00502TERMINATED172.31.92.227:44618 109DecisionTreeReg_c5b0 1 36.02131960.97
train_model_cabb8_00503TERMINATED172.31.104.216:1076 23DecisionTreeReg_cb50 1 32.57441302.1
train_model_cabb8_00504TERMINATED172.31.107.40:6585 59DecisionTreeReg_c3d0 1 33.23911266.04
train_model_cabb8_00505TERMINATED172.31.96.248:1078 221DecisionTreeReg_ccd0 1 32.69831301.36
train_model_cabb8_00506TERMINATED172.31.107.40:6756 6DecisionTreeReg_6be0 1 32.29731543.5
train_model_cabb8_00507TERMINATED172.31.96.248:1871 253DecisionTreeReg_6f40 1 33.0553 416.673
train_model_cabb8_00508TERMINATED172.31.92.227:44634 96DecisionTreeReg_6460 1 36.24191092.78
train_model_cabb8_00509TERMINATED172.31.104.216:1063 84DecisionTreeReg_6700 1 32.20061062.79
train_model_cabb8_00510TERMINATED172.31.106.163:998 207DecisionTreeReg_6910 1 32.5211 611.925
train_model_cabb8_00511TERMINATED172.31.114.169:1340 154DecisionTreeReg_6370 1 32.46361485.71
train_model_cabb8_00512TERMINATED172.31.116.192:1069 187DecisionTreeReg_6c70 1 32.42011674.55
train_model_cabb8_00513TERMINATED172.31.118.114:1487 184DecisionTreeReg_63d0 1 32.9425 925.744
train_model_cabb8_00514TERMINATED172.31.119.49:1068 30DecisionTreeReg_6940 1 33.38012130.15
train_model_cabb8_00515TERMINATED172.31.96.248:1864 105DecisionTreeReg_63a0 1 33.0177 247.749
train_model_cabb8_00516TERMINATED172.31.105.11:1068 2DecisionTreeReg_6b50 1 32.8593 586.666
train_model_cabb8_00517TERMINATED172.31.106.163:944 199DecisionTreeReg_de80 1 33.0161 147.133

Trial Progress

Trial name errorshould_checkpoint
train_model_cabb8_00000 447.234True
train_model_cabb8_00001 493.912True
train_model_cabb8_00002 461.717True
train_model_cabb8_00003 832.879True
train_model_cabb8_00004 447.743True
train_model_cabb8_00005 446.824True
train_model_cabb8_00006 424.082True
train_model_cabb8_00007 496.27 True
train_model_cabb8_00008 502.505True
train_model_cabb8_00009 473.44 True
train_model_cabb8_00010 468.736True
train_model_cabb8_00011 536.818True
train_model_cabb8_00012 586.173True
train_model_cabb8_00013 826.569True
train_model_cabb8_00014 411.649True
train_model_cabb8_00015 519.815True
train_model_cabb8_00016 843.189True
train_model_cabb8_00017 444.058True
train_model_cabb8_00018 515.764True
train_model_cabb8_00019 561.376True
train_model_cabb8_00020 635.01 True
train_model_cabb8_00021 469.924True
train_model_cabb8_00022 434.226True
train_model_cabb8_00023 440.032True
train_model_cabb8_00024 465.123True
train_model_cabb8_00025 488.919True
train_model_cabb8_00026 437.95 True
train_model_cabb8_00027 587.417True
train_model_cabb8_00028 638.248True
train_model_cabb8_00029 620.117True
train_model_cabb8_00030 746.608True
train_model_cabb8_00031 573.606True
train_model_cabb8_00032 530.903True
train_model_cabb8_00033 528.775True
train_model_cabb8_00034 582.801True
train_model_cabb8_00035 476.959True
train_model_cabb8_00036 630.055True
train_model_cabb8_00037 645.033True
train_model_cabb8_00038 670.53 True
train_model_cabb8_00039 623.395True
train_model_cabb8_00040 819.431True
train_model_cabb8_00041 489.846True
train_model_cabb8_00042 738.86 True
train_model_cabb8_00043 552.345True
train_model_cabb8_00044 506.195True
train_model_cabb8_00045 748.864True
train_model_cabb8_00046 444.971True
train_model_cabb8_00047 684.737True
train_model_cabb8_00048 582.08 True
train_model_cabb8_00049 582.837True
train_model_cabb8_00050 513.494True
train_model_cabb8_00051 467.511True
train_model_cabb8_00052 572.614True
train_model_cabb8_00053 470.489True
train_model_cabb8_00054 533.831True
train_model_cabb8_00055 586.92 True
train_model_cabb8_00056 708.579True
train_model_cabb8_00057 459.818True
train_model_cabb8_00058 798.612True
train_model_cabb8_00059 534.058True
train_model_cabb8_00060 575.949True
train_model_cabb8_00061 815.745True
train_model_cabb8_00062 750.948True
train_model_cabb8_00063 625.271True
train_model_cabb8_00064 978.726True
train_model_cabb8_00065 928.499True
train_model_cabb8_00066 873.963True
train_model_cabb8_00067 911.989True
train_model_cabb8_00068 601.737True
train_model_cabb8_00069 497.51 True
train_model_cabb8_00070 779.68 True
train_model_cabb8_00071 515.513True
train_model_cabb8_00072 640.862True
train_model_cabb8_00073 595.409True
train_model_cabb8_00074 594.696True
train_model_cabb8_00075 722.254True
train_model_cabb8_000761175.99 True
train_model_cabb8_00077 852.729True
train_model_cabb8_00078 665.045True
train_model_cabb8_00079 715.804True
train_model_cabb8_00080 793.186True
train_model_cabb8_00081 679.202True
train_model_cabb8_00082 696.223True
train_model_cabb8_00083 925.487True
train_model_cabb8_000841136.9 True
train_model_cabb8_00085 901.902True
train_model_cabb8_000861447.78 True
train_model_cabb8_00087 663.305True
train_model_cabb8_00088 827.901True
train_model_cabb8_00089 866.606True
train_model_cabb8_00090 696.926True
train_model_cabb8_00091 551.274True
train_model_cabb8_00092 916.012True
train_model_cabb8_00093 939.088True
train_model_cabb8_00094 796.153True
train_model_cabb8_000951083.62 True
train_model_cabb8_00096 739.744True
train_model_cabb8_00097 836.959True
train_model_cabb8_00098 807.23 True
train_model_cabb8_00099 943.097True
train_model_cabb8_00100 949.485True
train_model_cabb8_00101 776.939True
train_model_cabb8_00102 776.681True
train_model_cabb8_00103 928.545True
train_model_cabb8_00104 679.397True
train_model_cabb8_00105 691.016True
train_model_cabb8_00106 934.768True
train_model_cabb8_00107 865.874True
train_model_cabb8_00108 894.195True
train_model_cabb8_00109 659.628True
train_model_cabb8_00110 915.975True
train_model_cabb8_001111180.61 True
train_model_cabb8_00112 880.512True
train_model_cabb8_00113 976.283True
train_model_cabb8_00114 697.535True
train_model_cabb8_00115 701.698True
train_model_cabb8_00116 642.217True
train_model_cabb8_00117 822.855True
train_model_cabb8_00118 802.704True
train_model_cabb8_00119 930.469True
train_model_cabb8_00120 822.599True
train_model_cabb8_00121 585.042True
train_model_cabb8_00122 902.867True
train_model_cabb8_00123 882.167True
train_model_cabb8_00124 581.68 True
train_model_cabb8_00125 689.408True
train_model_cabb8_00126 644.432True
train_model_cabb8_001271081.26 True
train_model_cabb8_00128 788.735True
train_model_cabb8_00129 704.799True
train_model_cabb8_00130 932.215True
train_model_cabb8_00131 877.741True
train_model_cabb8_00132 842.718True
train_model_cabb8_00133 800.332True
train_model_cabb8_001341102.72 True
train_model_cabb8_00135 909.544True
train_model_cabb8_00136 904.699True
train_model_cabb8_00137 815.704True
train_model_cabb8_001381013.5 True
train_model_cabb8_00139 769.851True
train_model_cabb8_001401114.61 True
train_model_cabb8_00141 932.17 True
train_model_cabb8_001421301.97 True
train_model_cabb8_00143 997.36 True
train_model_cabb8_001441303.2 True
train_model_cabb8_001451149.16 True
train_model_cabb8_00146 809.239True
train_model_cabb8_00147 812.47 True
train_model_cabb8_001481042.6 True
train_model_cabb8_00149 749.195True
train_model_cabb8_00150 937.135True
train_model_cabb8_001511032.17 True
train_model_cabb8_00152 941.59 True
train_model_cabb8_001531044.11 True
train_model_cabb8_00154 672.288True
train_model_cabb8_00155 830.98 True
train_model_cabb8_00156 838.233True
train_model_cabb8_00157 953.51 True
train_model_cabb8_00158 844.536True
train_model_cabb8_00159 804.003True
train_model_cabb8_00160 808.665True
train_model_cabb8_00161 932.495True
train_model_cabb8_00162 986.148True
train_model_cabb8_001631023.12 True
train_model_cabb8_00164 651.052True
train_model_cabb8_00165 890.515True
train_model_cabb8_00166 945.843True
train_model_cabb8_00167 798.427True
train_model_cabb8_00168 871.733True
train_model_cabb8_00169 794.267True
train_model_cabb8_00170 904.808True
train_model_cabb8_00171 768.581True
train_model_cabb8_001721017.61 True
train_model_cabb8_001731019.55 True
train_model_cabb8_00174 988.151True
train_model_cabb8_00175 783.836True
train_model_cabb8_001761008.99 True
train_model_cabb8_001771024.57 True
train_model_cabb8_001781005.46 True
train_model_cabb8_001791207.97 True
train_model_cabb8_00180 735.263True
train_model_cabb8_00181 979.182True
train_model_cabb8_00182 988.628True
train_model_cabb8_00183 789.881True
train_model_cabb8_00184 968.959True
train_model_cabb8_00185 859.568True
train_model_cabb8_00186 763.152True
train_model_cabb8_00187 660.595True
train_model_cabb8_00188 994.283True
train_model_cabb8_00189 994.602True
train_model_cabb8_00190 881.976True
train_model_cabb8_00191 800.369True
train_model_cabb8_00192 765.721True
train_model_cabb8_00193 975.641True
train_model_cabb8_00194 937.527True
train_model_cabb8_001951093.04 True
train_model_cabb8_00196 890.415True
train_model_cabb8_001971153.63 True
train_model_cabb8_00198 843.831True
train_model_cabb8_001991030.5 True
train_model_cabb8_00200 819.106True
train_model_cabb8_002011001.59 True
train_model_cabb8_00202 932.234True
train_model_cabb8_00203 640.028True
train_model_cabb8_00204 716.565True
train_model_cabb8_002051452.28 True
train_model_cabb8_00206 832.524True
train_model_cabb8_00207 924.29 True
train_model_cabb8_00208 994.931True
train_model_cabb8_00209 741.202True
train_model_cabb8_00210 742.133True
train_model_cabb8_002111104.59 True
train_model_cabb8_00212 807.311True
train_model_cabb8_00213 942.116True
train_model_cabb8_00214 685.934True
train_model_cabb8_00215 775.18 True
train_model_cabb8_002161008.35 True
train_model_cabb8_00217 942.897True
train_model_cabb8_00218 998.627True
train_model_cabb8_00219 750.73 True
train_model_cabb8_002201162.88 True
train_model_cabb8_00221 861.811True
train_model_cabb8_00222 788.634True
train_model_cabb8_002231112.7 True
train_model_cabb8_00224 937.172True
train_model_cabb8_00225 850.925True
train_model_cabb8_00226 621.424True
train_model_cabb8_00227 836.129True
train_model_cabb8_00228 872.502True
train_model_cabb8_00229 930.933True
train_model_cabb8_00230 820.518True
train_model_cabb8_002311101.47 True
train_model_cabb8_00232 928.928True
train_model_cabb8_002331014.79 True
train_model_cabb8_00234 934.841True
train_model_cabb8_00235 949.496True
train_model_cabb8_002361895.21 True
train_model_cabb8_00237 990.916True
train_model_cabb8_00238 858.027True
train_model_cabb8_00239 959.43 True
train_model_cabb8_002401290.55 True
train_model_cabb8_00241 810.876True
train_model_cabb8_002421417.66 True
train_model_cabb8_002431179.34 True
train_model_cabb8_002441262.11 True
train_model_cabb8_002451747.95 True
train_model_cabb8_002461158.98 True
train_model_cabb8_002471872.14 True
train_model_cabb8_00248 414.218True
train_model_cabb8_002491042.94 True
train_model_cabb8_00250 814.855True
train_model_cabb8_00251 396.891True
train_model_cabb8_002521286.36 True
train_model_cabb8_002531612.05 True
train_model_cabb8_00254 623.42 True
train_model_cabb8_002551730.94 True
train_model_cabb8_00256 425.296True
train_model_cabb8_00257 661.768True
train_model_cabb8_00258 256.422True
train_model_cabb8_00259 414.616True
train_model_cabb8_00260 444.062True
train_model_cabb8_00261 425.023True
train_model_cabb8_00262 816.242True
train_model_cabb8_00263 409.536True
train_model_cabb8_00264 409.6 True
train_model_cabb8_00265 384.027True
train_model_cabb8_00266 469.844True
train_model_cabb8_00267 478.265True
train_model_cabb8_00268 435.744True
train_model_cabb8_00269 447.204True
train_model_cabb8_00270 508.675True
train_model_cabb8_00271 573.324True
train_model_cabb8_00272 775.215True
train_model_cabb8_00273 369.222True
train_model_cabb8_00274 497.537True
train_model_cabb8_00275 825.518True
train_model_cabb8_00276 430.031True
train_model_cabb8_00277 490.047True
train_model_cabb8_00278 505.643True
train_model_cabb8_00279 628.414True
train_model_cabb8_00280 440.234True
train_model_cabb8_00281 404.523True
train_model_cabb8_00282 407.959True
train_model_cabb8_00283 436.852True
train_model_cabb8_00284 450.778True
train_model_cabb8_00285 417.852True
train_model_cabb8_00286 548.66 True
train_model_cabb8_00287 615.046True
train_model_cabb8_00288 589.76 True
train_model_cabb8_00289 770.287True
train_model_cabb8_00290 547.395True
train_model_cabb8_00291 503.215True
train_model_cabb8_00292 508.438True
train_model_cabb8_00293 570.651True
train_model_cabb8_00294 452.535True
train_model_cabb8_00295 605.479True
train_model_cabb8_00296 631.479True
train_model_cabb8_00297 691.318True
train_model_cabb8_00298 603.513True
train_model_cabb8_00299 959.97 True
train_model_cabb8_00300 464.584True
train_model_cabb8_00301 701.172True
train_model_cabb8_00302 518.324True
train_model_cabb8_00303 498.436True
train_model_cabb8_00304 787.053True
train_model_cabb8_00305 415.325True
train_model_cabb8_00306 662.404True
train_model_cabb8_00307 563.372True
train_model_cabb8_00308 561.19 True
train_model_cabb8_00309 511.066True
train_model_cabb8_00310 419.804True
train_model_cabb8_00311 610.746True
train_model_cabb8_00312 442.141True
train_model_cabb8_00313 512.765True
train_model_cabb8_00314 571.675True
train_model_cabb8_00315 668.283True
train_model_cabb8_00316 441.492True
train_model_cabb8_00317 819.696True
train_model_cabb8_00318 481.299True
train_model_cabb8_00319 553.965True
train_model_cabb8_00320 760.58 True
train_model_cabb8_00321 745.999True
train_model_cabb8_00322 599.243True
train_model_cabb8_00323 927.041True
train_model_cabb8_00324 922.866True
train_model_cabb8_00325 900.319True
train_model_cabb8_00326 910.578True
train_model_cabb8_00327 614.539True
train_model_cabb8_00328 510.852True
train_model_cabb8_00329 769.241True
train_model_cabb8_00330 486.979True
train_model_cabb8_00331 600.674True
train_model_cabb8_00332 538.313True
train_model_cabb8_00333 632.727True
train_model_cabb8_00334 669.305True
train_model_cabb8_003351059.28 True
train_model_cabb8_00336 968.194True
train_model_cabb8_00337 691.806True
train_model_cabb8_00338 696.27 True
train_model_cabb8_00339 802.308True
train_model_cabb8_00340 612.787True
train_model_cabb8_00341 631.292True
train_model_cabb8_00342 860.138True
train_model_cabb8_003431241.46 True
train_model_cabb8_00344 909.991True
train_model_cabb8_003451326.45 True
train_model_cabb8_00346 630.02 True
train_model_cabb8_00347 491.831True
train_model_cabb8_00348 898.735True
train_model_cabb8_00349 669.428True
train_model_cabb8_00350 533.394True
train_model_cabb8_00351 922.454True
train_model_cabb8_003521048.96 True
train_model_cabb8_00353 807.024True
train_model_cabb8_00354 937.808True
train_model_cabb8_00355 675.821True
train_model_cabb8_00356 811.843True
train_model_cabb8_00357 784.407True
train_model_cabb8_00358 807.162True
train_model_cabb8_00359 873.888True
train_model_cabb8_00360 767.796True
train_model_cabb8_00361 743.98 True
train_model_cabb8_00362 913.213True
train_model_cabb8_00363 692.288True
train_model_cabb8_00364 666.494True
train_model_cabb8_00365 972.719True
train_model_cabb8_00366 873.205True
train_model_cabb8_00367 860.099True
train_model_cabb8_00368 627.511True
train_model_cabb8_00369 938.445True
train_model_cabb8_003701114.09 True
train_model_cabb8_003711062.66 True
train_model_cabb8_003721095.56 True
train_model_cabb8_00373 609.41 True
train_model_cabb8_00374 638.096True
train_model_cabb8_00375 621.417True
train_model_cabb8_00376 849.599True
train_model_cabb8_00377 808.165True
train_model_cabb8_00378 841.435True
train_model_cabb8_00379 780.833True
train_model_cabb8_00380 617.891True
train_model_cabb8_00381 862.727True
train_model_cabb8_00382 800.617True
train_model_cabb8_00383 555.392True
train_model_cabb8_00384 653.833True
train_model_cabb8_00385 625.919True
train_model_cabb8_003861153.89 True
train_model_cabb8_00387 729.803True
train_model_cabb8_00388 666.928True
train_model_cabb8_00389 946.049True
train_model_cabb8_003901024.23 True
train_model_cabb8_00391 851.655True
train_model_cabb8_00392 842.145True
train_model_cabb8_003931244.88 True
train_model_cabb8_00394 869.705True
train_model_cabb8_00395 903.461True
train_model_cabb8_00396 753.516True
train_model_cabb8_00397 991.263True
train_model_cabb8_00398 830.333True
train_model_cabb8_00399 997.54 True
train_model_cabb8_00400 921.847True
train_model_cabb8_004011296.03 True
train_model_cabb8_00402 923.831True
train_model_cabb8_004031325.21 True
train_model_cabb8_00404 964.887True
train_model_cabb8_00405 850.702True
train_model_cabb8_00406 724.768True
train_model_cabb8_004071016.22 True
train_model_cabb8_00408 702.536True
train_model_cabb8_00409 957.159True
train_model_cabb8_004101164.2 True
train_model_cabb8_00411 972.669True
train_model_cabb8_004121067.38 True
train_model_cabb8_00413 613.665True
train_model_cabb8_00414 950.157True
train_model_cabb8_00415 749.517True
train_model_cabb8_00416 945.049True
train_model_cabb8_00417 799.787True
train_model_cabb8_00418 826.981True
train_model_cabb8_00419 757.239True
train_model_cabb8_00420 865.223True
train_model_cabb8_00421 999.508True
train_model_cabb8_004221089.96 True
train_model_cabb8_00423 711.885True
train_model_cabb8_004241053.91 True
train_model_cabb8_00425 827.77 True
train_model_cabb8_00426 714.269True
train_model_cabb8_00427 801.326True
train_model_cabb8_00428 702.233True
train_model_cabb8_00429 839.538True
train_model_cabb8_00430 753.322True
train_model_cabb8_00431 936.22 True
train_model_cabb8_00432 832.906True
train_model_cabb8_00433 953.349True
train_model_cabb8_00434 849.461True
train_model_cabb8_004351012.25 True
train_model_cabb8_004361013.71 True
train_model_cabb8_00437 973.115True
train_model_cabb8_004381273.44 True
train_model_cabb8_00439 744.381True
train_model_cabb8_004401010.4 True
train_model_cabb8_004411139.62 True
train_model_cabb8_00442 771.109True
train_model_cabb8_00443 966.38 True
train_model_cabb8_00444 926.472True
train_model_cabb8_00445 694.658True
train_model_cabb8_00446 730.849True
train_model_cabb8_004471068.56 True
train_model_cabb8_00448 960.325True
train_model_cabb8_00449 871.701True
train_model_cabb8_00450 906.066True
train_model_cabb8_00451 745.201True
train_model_cabb8_00452 966.075True
train_model_cabb8_004531005.56 True
train_model_cabb8_00454 690.197True
train_model_cabb8_00455 933.612True
train_model_cabb8_004561150.56 True
train_model_cabb8_00457 988.922True
train_model_cabb8_004581028.9 True
train_model_cabb8_00459 828.414True
train_model_cabb8_00460 948.744True
train_model_cabb8_00461 903.878True
train_model_cabb8_00462 628.484True
train_model_cabb8_00463 677.17 True
train_model_cabb8_004641194.78 True
train_model_cabb8_00465 865.457True
train_model_cabb8_00466 952.08 True
train_model_cabb8_00467 942.023True
train_model_cabb8_00468 863.691True
train_model_cabb8_00469 764.074True
train_model_cabb8_00470 850.32 True
train_model_cabb8_00471 901.235True
train_model_cabb8_00472 904.826True
train_model_cabb8_00473 743.634True
train_model_cabb8_00474 754.058True
train_model_cabb8_00475 925.271True
train_model_cabb8_00476 877.888True
train_model_cabb8_004771057.58 True
train_model_cabb8_00478 825.355True
train_model_cabb8_004791199.29 True
train_model_cabb8_00480 961.614True
train_model_cabb8_00481 730.517True
train_model_cabb8_00482 919.248True
train_model_cabb8_00483 799.991True
train_model_cabb8_00484 749.128True
train_model_cabb8_00485 874.877True
train_model_cabb8_004861112.92 True
train_model_cabb8_00487 874.38 True
train_model_cabb8_004881098.25 True
train_model_cabb8_00489 919.617True
train_model_cabb8_004901229.93 True
train_model_cabb8_004911159.92 True
train_model_cabb8_004921072.06 True
train_model_cabb8_00493 933.92 True
train_model_cabb8_00494 967.935True
train_model_cabb8_00495 903.251True
train_model_cabb8_004961218.45 True
train_model_cabb8_004971185.13 True
train_model_cabb8_004981042.85 True
train_model_cabb8_004991873.76 True
train_model_cabb8_00500 411.63 True
train_model_cabb8_005011735.59 True
train_model_cabb8_005021960.97 True
train_model_cabb8_005031302.1 True
train_model_cabb8_005041266.04 True
train_model_cabb8_005051301.36 True
train_model_cabb8_005061543.5 True
train_model_cabb8_00507 416.673True
train_model_cabb8_005081092.78 True
train_model_cabb8_005091062.79 True
train_model_cabb8_00510 611.925True
train_model_cabb8_005111485.71 True
train_model_cabb8_005121674.55 True
train_model_cabb8_00513 925.744True
train_model_cabb8_005142130.15 True
train_model_cabb8_00515 247.749True
train_model_cabb8_00516 586.666True
train_model_cabb8_00517 147.133True 
Total number of models: 518
TOTAL TIME TAKEN: 2223.62 seconds
Best result: {'model': DecisionTreeRegressor(max_depth=3), 'location': 199}

After the Tune experiment has run, select the best model per dropoff location.

We can assemble the Tune results (ResultGrid object) into a pandas dataframe, then sort by minimum error, to select the best model per dropoff location.

# get a list of training loss errors
errors = []
[errors.append(i.metrics.get("error", 10000.0)) for i in results]

# get a list of checkpoints
checkpoints = []
[checkpoints.append(i.checkpoint) for i in results]

# get a list of locations
locations = []
[locations.append(i.config["location"]) for i in results]

# get a list of models
models = []
[models.append(i.config["model"]) for i in results]

# Assemble a pandas dataframe from Tune results
results_df = pd.DataFrame(
    zip(locations, models, errors, checkpoints),
    columns=["location_id", "model", "error", "checkpoint"],
)
print(results_df.dtypes)
results_df.head()

location_id      int64
model           object
error          float64
checkpoint      object
dtype: object
location_id model error checkpoint
0 24 LinearRegression() 447.234058 Checkpoint(local_path=/home/ray/christy-air/my...
1 140 LinearRegression() 493.912464 Checkpoint(local_path=/home/ray/christy-air/my...
2 141 LinearRegression() 461.717301 Checkpoint(local_path=/home/ray/christy-air/my...
3 257 LinearRegression() 832.879303 Checkpoint(local_path=/home/ray/christy-air/my...
4 239 LinearRegression() 447.742891 Checkpoint(local_path=/home/ray/christy-air/my... 
# Keep only 1 model per location_id with minimum error
final_df = results_df.dropna(subset=["error"])
final_df = final_df.loc[results_df.groupby("location_id")["error"].idxmin()].copy()
final_df.sort_values(by=["error"], inplace=True)
final_df.set_index("location_id", inplace=True, drop=True)
print(final_df.dtypes)
final_df

model          object
error         float64
checkpoint     object
dtype: object
model error checkpoint
location_id
199 DecisionTreeRegressor(max_depth=3) 147.133333 Checkpoint(local_path=/home/ray/christy-air/my...
105 DecisionTreeRegressor(max_depth=3) 247.749018 Checkpoint(local_path=/home/ray/christy-air/my...
236 DecisionTreeRegressor(max_depth=3) 369.222274 Checkpoint(local_path=/home/ray/christy-air/my...
238 DecisionTreeRegressor(max_depth=3) 384.026867 Checkpoint(local_path=/home/ray/christy-air/my...
207 LinearRegression() 396.891111 Checkpoint(local_path=/home/ray/christy-air/my...
... ... ... ...
217 DecisionTreeRegressor(max_depth=3) 1326.447774 Checkpoint(local_path=/home/ray/christy-air/my...
5 LinearRegression() 1417.664071 Checkpoint(local_path=/home/ray/christy-air/my...
6 DecisionTreeRegressor(max_depth=3) 1543.497932 Checkpoint(local_path=/home/ray/christy-air/my...
187 LinearRegression() 1612.046532 Checkpoint(local_path=/home/ray/christy-air/my...
30 LinearRegression() 1730.940913 Checkpoint(local_path=/home/ray/christy-air/my...

259 rows × 3 columns

Load a model from checkpoint and perform inference #

Tip

Ray AIR Predictors make batch inference easy since they have internal logic to parallelize the inference.

In this notebook, we will restore a single scikit-learn model directly from checkpoint, and demonstrate it can be used for inference.

Below, we can easily obtain AIR Checkpoint objects from the Tune results.

# Choose a dropoff location
location_id = 90
location_id

90

# Get a checkpoint directly from the pandas dataframe of Tune results
checkpoint = final_df.checkpoint[location_id]
print(type(checkpoint))

# Restore a model from checkpoint
model = checkpoint.to_dict()["model"]
print(type(model))

<class 'ray.air.checkpoint.Checkpoint'>
<class 'sklearn.tree._classes.DecisionTreeRegressor'>

# Create some test data
df_list = [read_data(f, location_id) for f in s3_files[:1]]
df_raw = pd.concat(df_list, ignore_index=True)
df = transform_df(df_raw)

# Train/test split.
_, test_df = train_test_split(df, test_size=0.2, shuffle=True)
test_X = test_df[["passenger_count", "trip_distance"]]
test_y = np.array(test_df.trip_duration)  # actual values

# Perform inference using restored model from checkpoint
pred_y = model.predict(test_X)

# Zip together predictions and actuals to visualize
pd.DataFrame(zip(pred_y, test_y), columns=["pred_y", "trip_duration"])[0:10]
pred_y trip_duration
0 880.224200 753
1 880.224200 699
2 364.604675 408
3 600.984340 259
4 600.984340 451
5 2221.970460 1144
6 1197.244840 483
7 880.224200 564
8 880.224200 546
9 600.984340 466

Compare validation and test error.

During model training we reported error on “validation” data (random sample). Below, we will report error on a pretend “test” data set (a different random sample).

Do a quick validation that both errors are reasonably close together.

# Evaluate restored model on test data.
error = sklearn.metrics.mean_absolute_error(test_y, pred_y)
print(f"Test error: {error}")

Test error: 404.4554806162227

# Compare test error with training validation error
print(f"Validation error: {final_df.error[location_id]}")

# Validation and test errors should be reasonably close together.

Validation error: 452.53533680221614