TiktokenTokenizer

[torronen]

Type of issue

Code doesn't work

Description

Tokernizer.CreateForModel("gpt-4") should be now TiktokenTokenizer.CreateForModel("gpt-4")
It's just the What's New page but this page ranks high in search engine results so users may easilly reference it.

Page URL

https://learn.microsoft.com/en-us/dotnet/machine-learning/whats-new/overview

Content source URL

https://github.com/dotnet/docs/blob/main/docs/machine-learning/whats-new/overview.md

Document Version Independent Id

87bb7009-9c3f-5d5b-1c92-456cd2188954

Article author

@luisquintanilla

Metadata

• ID: f888e18c-d07a-ea5b-6d83-4a6d37653ba2
• Service: dotnet-ml