ValueError: Tokenizer class BaichuanTokenizer does not exist or is not currently imported已解决

报错：ValueError: Tokenizer class BaichuanTokenizer does not exist or is not currently imported。环境：使用 AutoTokenizer 加载量化后的百川模型。添加 trust_remote_code=True。可以在量化时对 Tokenizer 进行保存。

XiaChan_26

995人浏览 · 2024-01-30 16:04:07

XiaChan_26 · 2024-01-30 16:04:07 发布

环境：使用 AutoTokenizer 加载量化后的百川模型
原始加载分词代码

from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained(
    model_id,
    torch_dtype=torch.float32,
    use_fast=False
)

报错：ValueError: Tokenizer class BaichuanTokenizer does not exist or is not currently imported

解决代码：

from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained(
    model_id,
    torch_dtype=torch.float32,
    use_fast=False, 
    trust_remote_code=True
)

添加 trust_remote_code=True

可以在量化时对 Tokenizer 进行保存