Skip to content

Commit 49a0907

Browse files
authored
[docs] Add more decoders (#1849)
1 parent abee958 commit 49a0907

File tree

1 file changed

+22
-1
lines changed

1 file changed

+22
-1
lines changed

docs/source-doc-builder/api/decoders.mdx

Lines changed: 22 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,10 @@
66

77
[[autodoc]] tokenizers.decoders.BPEDecoder
88

9+
## ByteFallback
10+
11+
[[autodoc]] tokenizers.decoders.ByteFallback
12+
913
## ByteLevel
1014

1115
[[autodoc]] tokenizers.decoders.ByteLevel
@@ -14,18 +18,35 @@
1418

1519
[[autodoc]] tokenizers.decoders.CTC
1620

21+
## Fuse
22+
23+
[[autodoc]] tokenizers.decoders.Fuse
24+
1725
## Metaspace
1826

1927
[[autodoc]] tokenizers.decoders.Metaspace
2028

29+
## Replace
30+
31+
[[autodoc]] tokenizers.decoders.Replace
32+
33+
## Sequence
34+
35+
[[autodoc]] tokenizers.decoders.Sequence
36+
37+
## Strip
38+
39+
[[autodoc]] tokenizers.decoders.Strip
40+
2141
## WordPiece
2242

2343
[[autodoc]] tokenizers.decoders.WordPiece
44+
2445
</python>
2546
<rust>
2647
The Rust API Reference is available directly on the [Docs.rs](https://docs.rs/tokenizers/latest/tokenizers/) website.
2748
</rust>
2849
<node>
2950
The node API has not been documented yet.
3051
</node>
31-
</tokenizerslangcontent>
52+
</tokenizerslangcontent>

0 commit comments

Comments
 (0)