{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Fine-tune `dolly-v2-7b` with Ray Train, PyTorch Lightning and FSDP\n", "\n", "\n", " \"try-anyscale-quickstart\"\n", "\n", "

" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In this example, we demonstrate how to use Ray Train to fine-tune a [`dolly-v2-7b`](https://huggingface.co/databricks/dolly-v2-7b) model. `dolly-v2-7b` is a 7 billion parameter causal language model created by Databricks, derived from EleutherAI’s [Pythia-6.9b](https://huggingface.co/EleutherAI/pythia-6.9b), and fine-tuned on a [~15K record instruction corpus](https://github.com/databrickslabs/dolly/tree/master/data).\n", "\n", "We load the pre-trained model from the HuggingFace model hub into a LightningModule and launch an FSDP fine-tuning job across 16 T4 GPUs with the help of {class}`Ray TorchTrainer `. It is also straightforward to fine-tune other similar large language models in a similar manner as shown in this example.\n", "\n", "Before starting this example, we highly recommend reading [Ray Train Key Concepts](train-key-concepts) and [Ray Data Quickstart](data_quickstart)." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Set up ray cluster \n", "In this example, we are using a Ray cluster with a `g4dn.8xlarge` head node and 15 `g4dn.4xlarge` worker nodes. Each instance has one Tesla T4 GPU (16GiB Memory). \n", "\n", "We define a `runtime_env` to install the necessary Python libraries on each node. You can skip this step if you have already installed all the required packages in your workers' base image. We tested this example with `lightning==2.0.2` and `transformers==4.29.2`." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "tags": [] }, "outputs": [], "source": [ "import ray\n", "\n", "ray.init(\n", " runtime_env={\n", " \"pip\": [\n", " \"datasets\",\n", " \"evaluate\",\n", " \"transformers>=4.26.0\",\n", " \"torch>=1.12.0\",\n", " \"lightning>=2.0\",\n", " ]\n", " }\n", ")" ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "tags": [] }, "outputs": [], "source": [ "MODEL_NAME = \"databricks/dolly-v2-7b\"" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Prepare your data \n", "We are using tiny_shakespeare for fine-tuning, which contains 40,000 lines of Shakespeare from a variety of Shakespeare's plays. Featured in Andrej Karpathy's blog post ['The Unreasonable Effectiveness of Recurrent Neural Networks'](http://karpathy.github.io/2015/05/21/rnn-effectiveness/). \n", "\n", "Dataset samples:\n", "```\n", "BAPTISTA:\n", "I know him well: you are welcome for his sake.\n", "\n", "GREMIO:\n", "Saving your tale, Petruchio, I pray,\n", "Let us, that are poor petitioners, speak too:\n", "Baccare! you are marvellous forward.\n", "\n", "PETRUCHIO:\n", "O, pardon me, Signior Gremio; I would fain be doing.\n", "```\n", "\n", "Here, we have adopted similar pre-processing logic from another demo: {doc}`GPT-J-6B Fine-Tuning with Ray Train and DeepSpeed <../deepspeed/gptj_deepspeed_fine_tuning>`." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "tags": [] }, "outputs": [], "source": [ "import ray\n", "import pandas as pd\n", "from datasets import load_dataset\n", "from transformers import AutoTokenizer, AutoModelForCausalLM\n", "\n", "def split_text(batch: pd.DataFrame) -> pd.DataFrame:\n", " text = list(batch[\"text\"])\n", " flat_text = \"\".join(text)\n", " split_text = [\n", " x.strip()\n", " for x in flat_text.split(\"\\n\")\n", " if x.strip() and not x.strip()[-1] == \":\"\n", " ]\n", " return pd.DataFrame(split_text, columns=[\"text\"])\n", "\n", "\n", "def tokenize(batch: pd.DataFrame) -> dict:\n", " tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME, padding_side=\"left\")\n", " tokenizer.pad_token = tokenizer.eos_token\n", " ret = tokenizer(\n", " list(batch[\"text\"]),\n", " truncation=True,\n", " max_length=256,\n", " padding=\"max_length\",\n", " return_tensors=\"np\",\n", " )\n", " ret[\"labels\"] = ret[\"input_ids\"].copy()\n", " return dict(ret)\n", "\n", "hf_dataset = load_dataset(\"tiny_shakespeare\")\n", "train_ds = ray.data.from_huggingface(hf_dataset[\"train\"])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We first split the original paragraphs into multiple sentences, then tokenize them. Here are some samples:" ] }, { "cell_type": "code", "execution_count": 4, "metadata": { "tags": [] }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "2023-08-30 11:03:12,182\tINFO dataset.py:2380 -- Tip: Use `take_batch()` instead of `take() / show()` to return records in pandas or numpy batch format.\n", "2023-08-30 11:03:12,185\tINFO streaming_executor.py:93 -- Executing DAG InputDataBuffer[Input] -> TaskPoolMapOperator[MapBatches(split_text)] -> LimitOperator[limit=10]\n", "2023-08-30 11:03:12,186\tINFO streaming_executor.py:94 -- Execution config: ExecutionOptions(resource_limits=ExecutionResources(cpu=None, gpu=None, object_store_memory=None), locality_with_output=False, preserve_order=False, actor_locality_enabled=True, verbose_progress=False)\n", "2023-08-30 11:03:12,187\tINFO streaming_executor.py:96 -- Tip: For detailed progress reporting, run `ray.data.DataContext.get_current().execution_options.verbose_progress = True`\n" ] }, { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "e398e697cafb4b548fabc85020df5e87", "version_major": 2, "version_minor": 0 }, "text/plain": [ "Running 0: 0%| | 0/1 [00:00` for more details." ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [], "source": [ "num_workers = 16\n", "batch_size_per_worker = 10" ] }, { "cell_type": "code", "execution_count": 9, "metadata": { "tags": [ "remove-cell" ] }, "outputs": [], "source": [ "# To accelerate release tests\n", "train_ds = train_ds.limit(num_workers * batch_size_per_worker * 10) # each worker has 10 batches" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Additionally, remember to define a Lightning callback that saves and reports checkpoints. Ray Train offers a simple implementation, {meth}`~ray.train.lightning.RayTrainReportCallback`, which persists your checkpoint and metrics in remote storage at the end of each training epoch. \n", "\n", "Note you can also implement your own report callback with customized logics, such as saving customized checkpoint files or reporting at a different frequency." ] }, { "cell_type": "code", "execution_count": 10, "metadata": { "tags": [ "remove-cell" ] }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "2023-08-30 11:03:16,651\tINFO streaming_executor.py:93 -- Executing DAG InputDataBuffer[Input] -> TaskPoolMapOperator[MapBatches(split_text)->MapBatches(tokenize)]\n", "2023-08-30 11:03:16,652\tINFO streaming_executor.py:94 -- Execution config: ExecutionOptions(resource_limits=ExecutionResources(cpu=None, gpu=None, object_store_memory=None), locality_with_output=False, preserve_order=False, actor_locality_enabled=True, verbose_progress=False)\n", "2023-08-30 11:03:16,652\tINFO streaming_executor.py:96 -- Tip: For detailed progress reporting, run `ray.data.DataContext.get_current().execution_options.verbose_progress = True`\n" ] }, { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "a0e66d43f7d44da0a99483390e78e113", "version_major": 2, "version_minor": 0 }, "text/plain": [ "Running 0: 0%| | 0/1 [00:00\n", "
\n", "
\n", "

Tune Status

\n", " \n", "\n", "\n", "\n", "\n", "\n", "
Current time:2023-08-30 11:51:22
Running for: 00:47:57.19
Memory: 39.3/124.3 GiB
\n", "
\n", "
\n", "
\n", "

System Info

\n", " Using FIFO scheduling algorithm.
Logical resource usage: 193.0/272 CPUs, 16.0/16 GPUs (0.0/16.0 accelerator_type:None)\n", "
\n", " \n", "
\n", "
\n", "
\n", "

Trial Status

\n", " \n", "\n", "\n", "\n", "\n", "\n", "\n", "
Trial name status loc iter total time (s) train_loss epoch step
TorchTrainer_839b5_00000TERMINATED10.0.23.226:66074 1 2868.15 0.176025 0 135
\n", "
\n", "\n", "\n" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[2m\u001b[36m(TrainTrainable pid=66074)\u001b[0m StorageContext on SESSION (rank=None):\n", "\u001b[2m\u001b[36m(TrainTrainable pid=66074)\u001b[0m StorageContext<\n", "\u001b[2m\u001b[36m(TrainTrainable pid=66074)\u001b[0m storage_path=/mnt/cluster_storage\n", "\u001b[2m\u001b[36m(TrainTrainable pid=66074)\u001b[0m storage_local_path=/home/ray/ray_results\n", "\u001b[2m\u001b[36m(TrainTrainable pid=66074)\u001b[0m storage_filesystem=\n", "\u001b[2m\u001b[36m(TrainTrainable pid=66074)\u001b[0m storage_fs_path=/mnt/cluster_storage\n", "\u001b[2m\u001b[36m(TrainTrainable pid=66074)\u001b[0m experiment_dir_name=finetune_dolly-v2-7b\n", "\u001b[2m\u001b[36m(TrainTrainable pid=66074)\u001b[0m trial_dir_name=TorchTrainer_839b5_00000_0_2023-08-30_11-03-25\n", "\u001b[2m\u001b[36m(TrainTrainable pid=66074)\u001b[0m current_checkpoint_index=0\n", "\u001b[2m\u001b[36m(TrainTrainable pid=66074)\u001b[0m >\n", "\u001b[2m\u001b[36m(TorchTrainer pid=66074)\u001b[0m Starting distributed worker processes: ['66181 (10.0.23.226)', '14250 (10.0.40.16)', '13932 (10.0.2.17)', '13832 (10.0.41.56)', '14288 (10.0.53.250)', '13909 (10.0.41.152)', '13803 (10.0.14.94)', '47214 (10.0.44.99)', '13836 (10.0.58.27)', '13838 (10.0.58.206)', '13755 (10.0.62.244)', '13828 (10.0.9.99)', '13771 (10.0.43.35)', '13726 (10.0.59.245)', '13829 (10.0.58.178)', '13861 (10.0.46.116)']\n", "\u001b[2m\u001b[36m(RayTrainWorker pid=66181)\u001b[0m Setting up process group for: env:// [rank=0, world_size=16]\n", "\u001b[2m\u001b[36m(RayTrainWorker pid=66181)\u001b[0m StorageContext on SESSION (rank=0):\n", "\u001b[2m\u001b[36m(RayTrainWorker pid=14250, ip=10.0.40.16)\u001b[0m StorageContext<\u001b[32m [repeated 2x across cluster] (Ray deduplicates logs by default. Set RAY_DEDUP_LOGS=0 to disable log deduplication, or see https://docs.ray.io/en/master/ray-observability/ray-logging.html#log-deduplication for more options.)\u001b[0m\n", "\u001b[2m\u001b[36m(RayTrainWorker pid=14250, ip=10.0.40.16)\u001b[0m storage_path=/mnt/cluster_storage\u001b[32m [repeated 2x across cluster]\u001b[0m\n", "\u001b[2m\u001b[36m(RayTrainWorker pid=14250, ip=10.0.40.16)\u001b[0m storage_local_path=/home/ray/ray_results\u001b[32m [repeated 2x across cluster]\u001b[0m\n", "\u001b[2m\u001b[36m(RayTrainWorker pid=14250, ip=10.0.40.16)\u001b[0m storage_filesystem=\u001b[32m [repeated 2x across cluster]\u001b[0m\n", "\u001b[2m\u001b[36m(RayTrainWorker pid=14250, ip=10.0.40.16)\u001b[0m storage_fs_path=/mnt/cluster_storage\u001b[32m [repeated 2x across cluster]\u001b[0m\n", "\u001b[2m\u001b[36m(RayTrainWorker pid=14250, ip=10.0.40.16)\u001b[0m current_checkpoint_index=0\u001b[32m [repeated 6x across cluster]\u001b[0m\n", "\u001b[2m\u001b[36m(RayTrainWorker pid=14250, ip=10.0.40.16)\u001b[0m >\u001b[32m [repeated 2x across cluster]\u001b[0m\n", "\u001b[2m\u001b[36m(SplitCoordinator pid=66292)\u001b[0m Auto configuring locality_with_output=['5ed99d043a52f67deb150f34202c09b77bd37409502ebf6e581b0544', '8efe8198d7c04d45714ae757f298c316117405f3a8b25b87a71e0d9e', 'e3754d1e1017e68dd919b35d35ea62ed7b005ad96452f371721fc9fa', '8bd0f431ab3733c4b423c1d50db06460e3c210de47355b3b4d215c31', '73a8b9377fe9531a84eaa7b30c966fbb11bc36aff070d55c8f7acd1a', 'ef922c93f3b2fc93ebe5a521426d24fb8aae7e13c65f9fbd106aea2a', '5249cff3eab41121f840c17a79e6a3cd0af0f059def707a39e055fcf', '042b668e5553a589a4f6693c45deee0abe57a1d754812172af425acb', '9ed138bfe1f9c7dca484ee08d8311806389adb3af7a76566a6f4dfaa', '7e2fcb5dfe4ab1b572d87257f9e13bbc22b33ba968b1e67a79505589', '39b1ef4da8493a22e321a1ea9dd13387f50d9a6e2d2fbad58ad5fe9c', '9484193409a5346c0838a4a19a0a08eec122477682ea1cb0ad3e305a', '0158084645ec305bdd2ab11a6f35c44ad206405ca810e65f24b09398', 'fe5b11633900d1c437b2e3ee4ea44c18cf68f3dece546537d2090c63', '573645f42162f531a66d20776a95ba05102fae8e4b8090d48b94b233', '47e317ad5d0eb94cabb78871541160763283629d0d3f3b77b69521ae']\n", "\u001b[2m\u001b[36m(RayTrainWorker pid=66181)\u001b[0m Using 16bit Automatic Mixed Precision (AMP)\n", "\u001b[2m\u001b[36m(RayTrainWorker pid=66181)\u001b[0m GPU available: True (cuda), used: True\n", "\u001b[2m\u001b[36m(RayTrainWorker pid=66181)\u001b[0m TPU available: False, using: 0 TPU cores\n", "\u001b[2m\u001b[36m(RayTrainWorker pid=66181)\u001b[0m IPU available: False, using: 0 IPUs\n", "\u001b[2m\u001b[36m(RayTrainWorker pid=66181)\u001b[0m HPU available: False, using: 0 HPUs\n", "\u001b[2m\u001b[36m(RayTrainWorker pid=13726, ip=10.0.59.245)\u001b[0m StorageContext on SESSION (rank=13):\u001b[32m [repeated 15x across cluster]\u001b[0m\n", "\u001b[2m\u001b[36m(RayTrainWorker pid=13726, ip=10.0.59.245)\u001b[0m StorageContext<\u001b[32m [repeated 14x across cluster]\u001b[0m\n", "\u001b[2m\u001b[36m(RayTrainWorker pid=13726, ip=10.0.59.245)\u001b[0m storage_path=/mnt/cluster_storage\u001b[32m [repeated 14x across cluster]\u001b[0m\n", "\u001b[2m\u001b[36m(RayTrainWorker pid=13726, ip=10.0.59.245)\u001b[0m storage_local_path=/home/ray/ray_results\u001b[32m [repeated 14x across cluster]\u001b[0m\n", "\u001b[2m\u001b[36m(RayTrainWorker pid=13726, ip=10.0.59.245)\u001b[0m storage_filesystem=\u001b[32m [repeated 14x across cluster]\u001b[0m\n", "\u001b[2m\u001b[36m(RayTrainWorker pid=13726, ip=10.0.59.245)\u001b[0m storage_fs_path=/mnt/cluster_storage\u001b[32m [repeated 14x across cluster]\u001b[0m\n", "\u001b[2m\u001b[36m(RayTrainWorker pid=13726, ip=10.0.59.245)\u001b[0m current_checkpoint_index=0\u001b[32m [repeated 42x across cluster]\u001b[0m\n", "\u001b[2m\u001b[36m(RayTrainWorker pid=13726, ip=10.0.59.245)\u001b[0m >\u001b[32m [repeated 14x across cluster]\u001b[0m\n", "\u001b[2m\u001b[36m(RayTrainWorker pid=66181)\u001b[0m Missing logger folder: /home/ray/ray_results/finetune_dolly-v2-7b/TorchTrainer_839b5_00000_0_2023-08-30_11-03-25/lightning_logs\n", "\u001b[2m\u001b[36m(RayTrainWorker pid=13832, ip=10.0.41.56)\u001b[0m Using 16bit Automatic Mixed Precision (AMP)\n", "\u001b[2m\u001b[36m(RayTrainWorker pid=13836, ip=10.0.58.27)\u001b[0m Using 16bit Automatic Mixed Precision (AMP)\n", "\u001b[2m\u001b[36m(RayTrainWorker pid=13832, ip=10.0.41.56)\u001b[0m Missing logger folder: /home/ray/ray_results/finetune_dolly-v2-7b/TorchTrainer_839b5_00000_0_2023-08-30_11-03-25/lightning_logs\n", "\u001b[2m\u001b[36m(RayTrainWorker pid=13726, ip=10.0.59.245)\u001b[0m Using 16bit Automatic Mixed Precision (AMP)\u001b[32m [repeated 13x across cluster]\u001b[0m\n", "\u001b[2m\u001b[36m(RayTrainWorker pid=47214, ip=10.0.44.99)\u001b[0m Missing logger folder: /home/ray/ray_results/finetune_dolly-v2-7b/TorchTrainer_839b5_00000_0_2023-08-30_11-03-25/lightning_logs\u001b[32m [repeated 13x across cluster]\u001b[0m\n", "\u001b[2m\u001b[36m(RayTrainWorker pid=13909, ip=10.0.41.152)\u001b[0m LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\u001b[2m\u001b[36m(RayTrainWorker pid=66181)\u001b[0m FullyShardedDataParallel(\n", "\u001b[2m\u001b[36m(RayTrainWorker pid=66181)\u001b[0m (_fsdp_wrapped_module): _LightningModuleWrapperBase(\n", "\u001b[2m\u001b[36m(RayTrainWorker pid=66181)\u001b[0m (_forward_module): DollyV2Model(\n", "\u001b[2m\u001b[36m(RayTrainWorker pid=66181)\u001b[0m (model): GPTNeoXForCausalLM(\n", "\u001b[2m\u001b[36m(RayTrainWorker pid=66181)\u001b[0m (gpt_neox): GPTNeoXModel(\n", "\u001b[2m\u001b[36m(RayTrainWorker pid=66181)\u001b[0m (embed_in): Embedding(50280, 4096)\n", "\u001b[2m\u001b[36m(RayTrainWorker pid=66181)\u001b[0m (layers): ModuleList(\n", "\u001b[2m\u001b[36m(RayTrainWorker pid=66181)\u001b[0m (0-31): 32 x FullyShardedDataParallel(\n", "\u001b[2m\u001b[36m(RayTrainWorker pid=66181)\u001b[0m (_fsdp_wrapped_module): CheckpointWrapper(\n", "\u001b[2m\u001b[36m(RayTrainWorker pid=66181)\u001b[0m (_checkpoint_wrapped_module): GPTNeoXLayer(\n", "\u001b[2m\u001b[36m(RayTrainWorker pid=66181)\u001b[0m (input_layernorm): LayerNorm((4096,), eps=1e-05, elementwise_affine=True)\n", "\u001b[2m\u001b[36m(RayTrainWorker pid=66181)\u001b[0m (post_attention_layernorm): LayerNorm((4096,), eps=1e-05, elementwise_affine=True)\n", "\u001b[2m\u001b[36m(RayTrainWorker pid=66181)\u001b[0m (attention): GPTNeoXAttention(\n", "\u001b[2m\u001b[36m(RayTrainWorker pid=66181)\u001b[0m (rotary_emb): RotaryEmbedding()\n", "\u001b[2m\u001b[36m(RayTrainWorker pid=66181)\u001b[0m (query_key_value): Linear(in_features=4096, out_features=12288, bias=True)\n", "\u001b[2m\u001b[36m(RayTrainWorker pid=66181)\u001b[0m (dense): Linear(in_features=4096, out_features=4096, bias=True)\n", "\u001b[2m\u001b[36m(RayTrainWorker pid=66181)\u001b[0m )\n", "\u001b[2m\u001b[36m(RayTrainWorker pid=66181)\u001b[0m (mlp): GPTNeoXMLP(\n", "\u001b[2m\u001b[36m(RayTrainWorker pid=66181)\u001b[0m (dense_h_to_4h): Linear(in_features=4096, out_features=16384, bias=True)\n", "\u001b[2m\u001b[36m(RayTrainWorker pid=66181)\u001b[0m (dense_4h_to_h): Linear(in_features=16384, out_features=4096, bias=True)\n", "\u001b[2m\u001b[36m(RayTrainWorker pid=66181)\u001b[0m (act): GELUActivation()\n", "\u001b[2m\u001b[36m(RayTrainWorker pid=66181)\u001b[0m )\n", "\u001b[2m\u001b[36m(RayTrainWorker pid=66181)\u001b[0m )\n", "\u001b[2m\u001b[36m(RayTrainWorker pid=66181)\u001b[0m )\n", "\u001b[2m\u001b[36m(RayTrainWorker pid=66181)\u001b[0m )\n", "\u001b[2m\u001b[36m(RayTrainWorker pid=66181)\u001b[0m )\n", "\u001b[2m\u001b[36m(RayTrainWorker pid=66181)\u001b[0m (final_layer_norm): LayerNorm((4096,), eps=1e-05, elementwise_affine=True)\n", "\u001b[2m\u001b[36m(RayTrainWorker pid=66181)\u001b[0m )\n", "\u001b[2m\u001b[36m(RayTrainWorker pid=66181)\u001b[0m (embed_out): Linear(in_features=4096, out_features=50280, bias=False)\n", "\u001b[2m\u001b[36m(RayTrainWorker pid=66181)\u001b[0m )\n", "\u001b[2m\u001b[36m(RayTrainWorker pid=66181)\u001b[0m )\n", "\u001b[2m\u001b[36m(RayTrainWorker pid=66181)\u001b[0m )\n", "\u001b[2m\u001b[36m(RayTrainWorker pid=66181)\u001b[0m )\n", "Epoch 0: 0%| | 0/134 [00:00 TaskPoolMapOperator[MapBatches(split_text)->MapBatches(tokenize)] -> OutputSplitter[split(16, equal=True)]\n", "\u001b[2m\u001b[36m(SplitCoordinator pid=66292)\u001b[0m Execution config: ExecutionOptions(resource_limits=ExecutionResources(cpu=None, gpu=None, object_store_memory=2000000000.0), locality_with_output=['5ed99d043a52f67deb150f34202c09b77bd37409502ebf6e581b0544', '8efe8198d7c04d45714ae757f298c316117405f3a8b25b87a71e0d9e', 'e3754d1e1017e68dd919b35d35ea62ed7b005ad96452f371721fc9fa', '8bd0f431ab3733c4b423c1d50db06460e3c210de47355b3b4d215c31', '73a8b9377fe9531a84eaa7b30c966fbb11bc36aff070d55c8f7acd1a', 'ef922c93f3b2fc93ebe5a521426d24fb8aae7e13c65f9fbd106aea2a', '5249cff3eab41121f840c17a79e6a3cd0af0f059def707a39e055fcf', '042b668e5553a589a4f6693c45deee0abe57a1d754812172af425acb', '9ed138bfe1f9c7dca484ee08d8311806389adb3af7a76566a6f4dfaa', '7e2fcb5dfe4ab1b572d87257f9e13bbc22b33ba968b1e67a79505589', '39b1ef4da8493a22e321a1ea9dd13387f50d9a6e2d2fbad58ad5fe9c', '9484193409a5346c0838a4a19a0a08eec122477682ea1cb0ad3e305a', '0158084645ec305bdd2ab11a6f35c44ad206405ca810e65f24b09398', 'fe5b11633900d1c437b2e3ee4ea44c18cf68f3dece546537d2090c63', '573645f42162f531a66d20776a95ba05102fae8e4b8090d48b94b233', '47e317ad5d0eb94cabb78871541160763283629d0d3f3b77b69521ae'], preserve_order=False, actor_locality_enabled=True, verbose_progress=False)\n", "\u001b[2m\u001b[36m(SplitCoordinator pid=66292)\u001b[0m Tip: For detailed progress reporting, run `ray.data.DataContext.get_current().execution_options.verbose_progress = True`\n" ] }, { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "eb9d86f824ce4098bc4f081b87d47da4", "version_major": 2, "version_minor": 0 }, "text/plain": [ "(pid=66292) Running 0: 0%| | 0/1 [00:00