File tree Expand file tree Collapse file tree 1 file changed +22
-1
lines changed
docs/source-doc-builder/api Expand file tree Collapse file tree 1 file changed +22
-1
lines changed Original file line number Diff line number Diff line change 6
6
7
7
[[ autodoc]] tokenizers.decoders.BPEDecoder
8
8
9
+ ## ByteFallback
10
+
11
+ [[ autodoc]] tokenizers.decoders.ByteFallback
12
+
9
13
## ByteLevel
10
14
11
15
[[ autodoc]] tokenizers.decoders.ByteLevel
14
18
15
19
[[ autodoc]] tokenizers.decoders.CTC
16
20
21
+ ## Fuse
22
+
23
+ [[ autodoc]] tokenizers.decoders.Fuse
24
+
17
25
## Metaspace
18
26
19
27
[[ autodoc]] tokenizers.decoders.Metaspace
20
28
29
+ ## Replace
30
+
31
+ [[ autodoc]] tokenizers.decoders.Replace
32
+
33
+ ## Sequence
34
+
35
+ [[ autodoc]] tokenizers.decoders.Sequence
36
+
37
+ ## Strip
38
+
39
+ [[ autodoc]] tokenizers.decoders.Strip
40
+
21
41
## WordPiece
22
42
23
43
[[ autodoc]] tokenizers.decoders.WordPiece
44
+
24
45
</python >
25
46
<rust >
26
47
The Rust API Reference is available directly on the [ Docs.rs] ( https://docs.rs/tokenizers/latest/tokenizers/ ) website.
27
48
</rust >
28
49
<node >
29
50
The node API has not been documented yet.
30
51
</node >
31
- </tokenizerslangcontent >
52
+ </tokenizerslangcontent >
You can’t perform that action at this time.
0 commit comments