AttributeError: 'MetaLoader' object has no attribute 'dataset'

[Richar-Du]

When I try to resume a checkpoint from last training, it raises an error:

[rank12]: Traceback (most recent call last):
[rank12]:   File "/cpfs/29f69eb5e2e60f26/user/GPT/pretrain/mm_intern/duyifan/ST-LLM-temp/stllm/train/train_hf.py", line 278, in <module>
[rank12]:     train()
[rank12]:   File "/cpfs/29f69eb5e2e60f26/user/GPT/pretrain/mm_intern/duyifan/ST-LLM-temp/stllm/train/train_hf.py", line 267, in train
[rank12]:     trainer.train(resume_from_checkpoint=True)
[rank12]:   File "/cpfs/29f69eb5e2e60f26/user/GPT/pretrain/mm_intern/duyifan/miniconda3/envs/stllm/lib/python3.10/site-packages/transformers/trainer.py", line 1662, in train
[rank12]:     return inner_training_loop(
[rank12]:   File "/cpfs/29f69eb5e2e60f26/user/GPT/pretrain/mm_intern/duyifan/miniconda3/envs/stllm/lib/python3.10/site-packages/transformers/trainer.py", line 1893, in _inner_training_loop
[rank12]:     epoch_iterator = skip_first_batches(epoch_iterator, steps_trained_in_current_epoch)
[rank12]:   File "/cpfs/29f69eb5e2e60f26/user/GPT/pretrain/mm_intern/duyifan/miniconda3/envs/stllm/lib/python3.10/site-packages/accelerate/data_loader.py", line 1086, in skip_first_batches
[rank12]:     dataset = dataloader.dataset
[rank12]: AttributeError: 'MetaLoader' object has no attribute 'dataset'

[farewellthree]

Sorry, our code currently only supports resume from epoch, not resume from step. Resuming from step cannot retrieve the historical order of the dataset.