Can’t Import BertTokenizer

I am attempting to use the BertTokenizer part of the transformers package. First I install as below.

pip install transformers

Which says it succeeds.

When I try to import parts of the package as below I get the following.

 from transformers import BertTokenizer
Traceback (most recent call last):

  File "<ipython-input-2-89505a24ece6>", line 1, in <module>
    from transformers import BertTokenizer

  File "C:\Users\User\anaconda3\lib\site-packages\transformers\", line 22, in <module>
    from .integrations import (  # isort:skip

  File "C:\Users\User\anaconda3\lib\site-packages\transformers\", line 42, in <module>
    from .trainer_utils import PREFIX_CHECKPOINT_DIR, BestRun  # isort:skip

  File "C:\Users\User\anaconda3\lib\site-packages\transformers\", line 10, in <module>
    from .tokenization_utils_base import ExplicitEnum

  File "C:\Users\User\anaconda3\lib\site-packages\transformers\", line 31, in <module>
    from tokenizers import AddedToken

  File "C:\Users\User\anaconda3\lib\site-packages\tokenizers\", line 17, in <module>
    from .tokenizers import Tokenizer, Encoding, AddedToken

ModuleNotFoundError: No module named 'tokenizers.tokenizers'

The package is detailed here so I think it should be available

Read more here:

Content Attribution

This content was originally published by user2355903 at Recent Questions - Stack Overflow, and is syndicated here via their RSS feed. You can read the original post over there.

%d bloggers like this: