KeyError when loading AutoModelForTokenClassification

[pimverschuuren]

System Info

• transformers version: 4.29.2
• Platform: macOS-14.2.1-arm64-arm-64bit
• Python version: 3.10.14
• Huggingface_hub version: 0.23.0
• Safetensors version: 0.4.3
• PyTorch version (GPU?): 2.3.0 (False)
• Tensorflow version (GPU?): not installed (NA)
• Flax version (CPU?/GPU?/TPU?): not installed (NA)
• Jax version: not installed
• JaxLib version: not installed
• Using GPU in script?: No
• Using distributed or parallel set-up in script?: No

Who can help?

No response

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

1. Setup a python virtual environment with the command python -m venv .venv

2. Enter the virtual environment with source .venv/bin/activate

3. Install the following python dependencies:

   transformers==4.29.2
   accelerate==0.19.0
   datasets
   pysbd
   wandb
   h5py
   nltk
   spacy
   ersatz
   iso-639
   scikit-learn==1.2.2
   numpy==1.23.5
   pydantic
   torchinfo
   conllu
   pandarallel
   cohere
   replicate
   onnx
   onnxruntime
   torchinfo
   mosestokenizer
   cached_property
   tqdm
   skops
   pandas
   protobuf==3.20

4. Run the lines of python code

from transformers import AutoModelForTokenClassification
model = AutoModelForTokenClassification.from_pretrained("segment-any-text/sat-1l-sm")

Expected behavior

Expected the model to load without any issue. However, I get the following error instead:

Traceback (most recent call last):
  File "/Users/pim.jv/Documents/Code/wtpsplit/test_hf.py", line 4, in <module>
    model = AutoModelForTokenClassification.from_pretrained("segment-any-text/sat-1l-sm", force_download=False)
  File "/opt/homebrew/lib/python3.10/site-packages/transformers/models/auto/auto_factory.py", line 444, in from_pretrained
    config, kwargs = AutoConfig.from_pretrained(
  File "/opt/homebrew/lib/python3.10/site-packages/transformers/models/auto/configuration_auto.py", line 940, in from_pretrained
    config_class = CONFIG_MAPPING[config_dict["model_type"]]
  File "/opt/homebrew/lib/python3.10/site-packages/transformers/models/auto/configuration_auto.py", line 655, in __getitem__
    raise KeyError(key)
KeyError: 'xlm-token'

[VladOS95-cyber]

Hey, there is no config assignment for this model type. It is needed to add related type and config into mapping. @zucchini-nlp is it ok for you? I could try to fix it

[zucchini-nlp]

Seems like you are trying to load XLMForTokenClassification. The model type should be modified in the config.json file on the hub. XLM models can be loaded via xlm-roberta model type. If the model belongs to you you can update it yourself, otherwise open a PR on model page.

transformers/src/transformers/models/auto/modeling_auto.py

Lines 1126 to 1127 in ac5a055

	("xlm-roberta", "XLMRobertaForTokenClassification"),
	("xlm-roberta-xl", "XLMRobertaXLForTokenClassification"),

A workaround is to load directly with the correct model class, as XLMRobertaForTokenClassification.from_pretrained(model_id)

[VladOS95-cyber]

@zucchini-nlp Just one thing is weird to me and because of that, I suppose, model loading is failed, is that in config.json of this model "base_model": "xlm-roberta-base". I am not sure that it is correct, should not be a base_model is different one?