Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Generative Pretrained Transformer (GPT) example #1875

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

Sadeqk94
Copy link

This Pull Request introduces a new example demonstrating the implementation of a Generative Pretrained Transformer (GPT) using Keras. The example showcases how to build, train, and generate text with a GPT model, providing a comprehensive guide for users interested in natural language processing (NLP) and deep learning.

@alkaou
Copy link

alkaou commented Jun 23, 2024

This Pull Request introduces a new example demonstrating the implementation of a Generative Pretrained Transformer (GPT) using Keras. The example showcases how to build, train, and generate text with a GPT model, providing a comprehensive guide for users interested in natural language processing (NLP) and deep learning.

After training, once saved keep the model, it can no longer be loaded.

Hist = model.fit(
    train_data_generator,
    epochs=Epochs,
    steps_per_epoch=eval_interval,
    validation_data=val_data_generator,
    validation_steps=eval_iters,
)

model_name = "br_bigram_model.h5"
# save the model
model.save(model_name)

from keras.models import load_model

custom_objects = {
    "FeedForward": FeedForward,
    "Block": Block,
    "BigramLanguageModel": BigramLanguageModel,
}
# Load the model
bigram_model = load_model(model_name, custom_objects=custom_objects)
bigram_model .summary()

Here is the error we always return:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
File E:\Alkaou\Python Projects\ia_code\venv\lib\site-packages\keras\src\saving\serialization_lib.py:718, in deserialize_keras_object(config, custom_objects, safe_mode, **kwargs)
    717 try:
--> 718     instance = cls.from_config(inner_config)
    719 except TypeError as e:

Cell In[157], line 69, in BigramLanguageModel.from_config(cls, config)
     67 @classmethod
     68 def from_config(cls, config):
---> 69     return cls(**config)

TypeError: BigramLanguageModel.__init__() got an unexpected keyword argument 'name'

During handling of the above exception, another exception occurred:

TypeError                                 Traceback (most recent call last)
Cell In[173], line 11
      4 custom_objects = {
      5     "FeedForward": FeedForward,
      6     "Block": Block,
      7     "BigramLanguageModel": BigramLanguageModel,
      8 }
     10 # Charger le modèle
---> 11 br_bigram_model = load_model(model_name, custom_objects=custom_objects)
     12 br_bigram_model.summary()

File E:\Alkaou\Python Projects\ia_code\venv\lib\site-packages\keras\src\saving\saving_api.py:176, in load_model(filepath, custom_objects, compile, safe_mode)
    173         is_keras_zip = True
    175 if is_keras_zip:
--> 176     return saving_lib.load_model(
    177         filepath,
    178         custom_objects=custom_objects,
    179         compile=compile,
    180         safe_mode=safe_mode,
    181     )
    182 if str(filepath).endswith((".h5", ".hdf5")):
    183     return legacy_h5_format.load_model_from_hdf5(
    184         filepath, custom_objects=custom_objects, compile=compile
    185     )

File E:\Alkaou\Python Projects\ia_code\venv\lib\site-packages\keras\src\saving\saving_lib.py:152, in load_model(filepath, custom_objects, compile, safe_mode)
    147     raise ValueError(
    148         "Invalid filename: expected a `.keras` extension. "
    149         f"Received: filepath={filepath}"
    150     )
    151 with open(filepath, "rb") as f:
--> 152     return _load_model_from_fileobj(
    153         f, custom_objects, compile, safe_mode
    154     )

File E:\Alkaou\Python Projects\ia_code\venv\lib\site-packages\keras\src\saving\saving_lib.py:170, in _load_model_from_fileobj(fileobj, custom_objects, compile, safe_mode)
    168 # Construct the model from the configuration file in the archive.
    169 with ObjectSharingScope():
--> 170     model = deserialize_keras_object(
    171         config_dict, custom_objects, safe_mode=safe_mode
    172     )
    174 all_filenames = zf.namelist()
    175 if _VARS_FNAME + ".h5" in all_filenames:

File E:\Alkaou\Python Projects\ia_code\venv\lib\site-packages\keras\src\saving\serialization_lib.py:720, in deserialize_keras_object(config, custom_objects, safe_mode, **kwargs)
    718     instance = cls.from_config(inner_config)
    719 except TypeError as e:
--> 720     raise TypeError(
    721         f"{cls} could not be deserialized properly. Please"
    722         " ensure that components that are Python object"
    723         " instances (layers, models, etc.) returned by"
    724         " `get_config()` are explicitly deserialized in the"
    725         " model's `from_config()` method."
    726         f"\n\nconfig={config}.\n\nException encountered: {e}"
    727     )
    728 build_config = config.get("build_config", None)
    729 if build_config and not instance.built:

TypeError: <class '__main__.BigramLanguageModel'> could not be deserialized properly. Please ensure that components that are Python object instances (layers, models, etc.) returned by `get_config()` are explicitly deserialized in the model's `from_config()` method.

config={'module': None, 'class_name': 'BigramLanguageModel', 'config': {'name': 'bigram_language_model_7', 'trainable': True, 'dtype': 'float32', 'vocab_size': 108, 'n_embd': 64, 'block_size': 32, 'n_head': 4, 'n_layer': 4}, 'registered_name': 'BigramLanguageModel', 'build_config': {'input_shape': [16, 32]}, 'compile_config': {'optimizer': {'module': 'keras.optimizers', 'class_name': 'Adam', 'config': {'name': 'adam', 'learning_rate': 0.0010000000474974513, 'weight_decay': None, 'clipnorm': None, 'global_clipnorm': None, 'clipvalue': None, 'use_ema': False, 'ema_momentum': 0.99, 'ema_overwrite_frequency': None, 'loss_scale_factor': None, 'gradient_accumulation_steps': None, 'beta_1': 0.9, 'beta_2': 0.999, 'epsilon': 1e-07, 'amsgrad': False}, 'registered_name': None}, 'loss': None, 'loss_weights': None, 'metrics': None, 'weighted_metrics': None, 'run_eagerly': False, 'steps_per_execution': 1, 'jit_compile': False}}.

Exception encountered: BigramLanguageModel.__init__() got an unexpected keyword argument 'name'

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants