TensorflowTrainer#
- class ray.train.tensorflow.tensorflow_trainer.TensorflowTrainer(*args, **kwargs)[source]#
Bases:
DataParallelTrainerA Trainer for data parallel Tensorflow training.
This Trainer runs the function
train_loop_per_workeron multiple Ray Actors. These actors already have the necessary TensorFlow process group already configured for distributed TensorFlow training.If
train_loop_per_workeraccepts an argument, thentrain_loop_configwill be passed in as the argument. This is useful if you want to tune the values intrain_loop_configas hyperparameters.If the
datasetsdict contains a training dataset (denoted by the “train” key), then it will be split into multiple dataset shards that can then be accessed byray.train.get_dataset_shard("train")insidetrain_loop_per_worker. All the other datasets will not be split andray.train.get_dataset_shard(...)will return the entire Dataset.Warning
Ray will not automatically set any environment variables or configuration related to local parallelism / threading aside from “OMP_NUM_THREADS”. If you desire greater control over TensorFlow threading, use the
tf.config.threadingmodule (eg.tf.config.threading.set_inter_op_parallelism_threads(num_cpus)) at the beginning of yourtrain_loop_per_workerfunction.Any returns from the
train_loop_per_workerwill be discarded and not used or persisted anywhere.Examples
The
train_loop_per_workerfunction is expected to take in either 0 or 1 arguments:>>> def train_loop_per_worker(): ... ...
>>> def train_loop_per_worker(config: Dict): ... ...
Inside the
train_loop_per_workerfunction, you can use any of the Ray Train loop methods.>>> from ray import train >>> def train_loop_per_worker(): ... # Report intermediate results for callbacks or logging and ... # checkpoint data. ... train.report(...) ... ... # Returns the last saved checkpoint, or None. ... train.get_checkpoint() ... ... # Returns the Dataset shard for the given key. ... train.get_dataset_shard("my_dataset") ... ... # Returns the total number of workers executing training. ... train.get_context().get_world_size() ... ... # Returns the rank of this worker. ... train.get_context().get_world_rank() ... ... # Returns the rank of the worker on the current node. ... train.get_context().get_local_rank()
>>> import os >>> import tempfile >>> import tensorflow as tf >>> import numpy as np >>> import ray >>> from ray import train >>> from ray.train import Checkpoint, ScalingConfig >>> from ray.train.tensorflow import TensorflowTrainer >>> def build_model(): ... # toy neural network : 1-layer ... return tf.keras.Sequential( ... [tf.keras.layers.Dense( ... 1, activation="linear", input_shape=(1,))] ... ) >>> def train_loop_per_worker(config): ... dataset_shard = train.get_dataset_shard("train") ... strategy = tf.distribute.experimental.MultiWorkerMirroredStrategy() ... with strategy.scope(): ... model = build_model() ... model.compile( ... optimizer="Adam", loss="mean_squared_error", metrics=["mse"]) ... ... tf_dataset = dataset_shard.to_tf( ... feature_columns="x", ... label_columns="y", ... batch_size=1 ... ) ... for epoch in range(config["num_epochs"]): ... model.fit(tf_dataset) ... ... # Create checkpoint. ... checkpoint_dir = tempfile.mkdtemp() ... model.save_weights( ... os.path.join(checkpoint_dir, "my_checkpoint") ... ) ... checkpoint = Checkpoint.from_directory(checkpoint_dir) ... ... train.report( ... {}, ... checkpoint=checkpoint, ... ) >>> train_dataset = ray.data.from_items([{"x": np.array([x], dtype=np.float32), "y": x + 1} for x in range(32)]) >>> trainer = TensorflowTrainer( ... train_loop_per_worker=train_loop_per_worker, ... scaling_config=ScalingConfig(num_workers=3, use_gpu=True), ... datasets={"train": train_dataset}, ... train_loop_config={"num_epochs": 2}, ... ) >>> result = trainer.fit()
- Parameters:
train_loop_per_worker (Callable[[], None] | Callable[[Dict], None]) – The training function to execute. This can either take in no arguments or a
configdict.train_loop_config (Dict | None) – Configurations to pass into
train_loop_per_workerif it accepts an argument.tensorflow_config (TensorflowConfig | None) – Configuration for setting up the TensorFlow backend. If set to None, use the default configuration. This replaces the
backend_configarg ofDataParallelTrainer.scaling_config (ScalingConfig | None) – Configuration for how to scale data parallel training.
dataset_config (DataConfig | None) – Configuration for dataset ingest.
run_config (RunConfig | None) – Configuration for the execution of the training run.
datasets (Dict[str, Dataset | Callable[[], Dataset]] | None) – Any Datasets to use for training. Use the key “train” to denote which dataset is the training dataset.
metadata (Dict[str, Any] | None) – Dict that should be made available via
ray.train.get_context().get_metadata()and incheckpoint.get_metadata()for checkpoints saved from this Trainer. Must be JSON-serializable.resume_from_checkpoint (Checkpoint | None) – A checkpoint to resume training from.
PublicAPI (beta): This API is in beta and may change before becoming stable.
Methods
Converts self to a
tune.Trainableclass.Checks whether a given directory contains a restorable Train experiment.
Runs training.
Returns a copy of this Trainer's final dataset configs.
Deprecated.
Restores a DataParallelTrainer from a previously interrupted/failed run.
Called during fit() to perform initial setup on the Trainer.