tokenizers

This package is deprecated. PyPI now publishes newer versions of this package for riscv64, and we will no longer maintain this package. Please use the version from PyPI instead.

If you need a specific version between the latest available here and the first available on PyPI, please open an Issue.

• Source Code: https://github.com/huggingface/tokenizers
• Supported versions:

0.22.2 (latest)

pip install tokenizers --index-url https://gitlab.com/api/v4/projects/56254198/packages/pypi/simple

• License: Apache 2.0
• Download files: https://gitlab.com/riseproject/python/wheel_builder/-/packages/51636794

0.22.1

pip install tokenizers==0.22.1 --index-url https://gitlab.com/api/v4/projects/56254198/packages/pypi/simple

• License: Apache 2.0
• Download files: https://gitlab.com/riseproject/python/wheel_builder/-/packages/48310436

0.22.0

pip install tokenizers==0.22.0 --index-url https://gitlab.com/api/v4/projects/56254198/packages/pypi/simple

• License: Apache 2.0
• Download files: https://gitlab.com/riseproject/python/wheel_builder/-/packages/48310477

0.21.1

pip install tokenizers==0.21.1 --index-url https://gitlab.com/api/v4/projects/56254198/packages/pypi/simple

• License: Apache 2.0
• Download files: https://gitlab.com/riseproject/python/wheel_builder/-/packages/36897711

0.21.0

pip install tokenizers==0.21.0 --index-url https://gitlab.com/api/v4/projects/56254198/packages/pypi/simple

• License: Apache 2.0
• Download files: https://gitlab.com/riseproject/python/wheel_builder/-/packages/36897719

0.20.3

pip install tokenizers==0.20.3 --index-url https://gitlab.com/api/v4/projects/56254198/packages/pypi/simple

• License: Apache 2.0
• Download files: https://gitlab.com/riseproject/python/wheel_builder/-/packages/36897733

License file included in the wheel.

test_tutorial_train_from_iterators.py disabled due to missing dependency.