Use native implementation of LZMA and XZ (#405)

hasenbanck · Pr0methean · web-flow · commit aaaa26fdfa05 · 2025-09-03T20:27:32.000Z
* deps: Use native lzma-rust2 instead of liblzma

This is a native port of the XZ's liblzma. The main advantage is, that the crate is native Rust and is also used by the 7z crate.

Performance should be equal to the liblzma crate, since I spend quite a bit of time to improve the performance.

I only used the single threaded version of the reader/writer, since parallelization was not enabled for liblzma. lzma-rust2 has multithreaded reader/writer though.

I had to remove the old bug report fix, since lzma_rust2 doesn't have the reported behavior. The test case also was obviously AI generated and useless.

* fix: Properly implement LZMA decoding

The old implementation actually didn't worked. The test file uses STORE as compression. This makes sure that LZMA is properly implemented.

I remembered that I had to defer the initialization of reading the properties when implementing PPMd, so I did this here too.

* fix: Fix linter issue

* fix: Formatting of README.md

* fix: Reduce IO reads

* fix: Move XZ decoder and encoder to the heap

* fix: Fix code review issues

* fix: Fix UnwindSafe issue by using latest lzma-rust2 version

* Rephrase a comment in src/compression.rs

Signed-off-by: Chris Hennick &lt;4961925+Pr0methean@users.noreply.github.com&gt;

---------

Signed-off-by: Chris Hennick &lt;4961925+Pr0methean@users.noreply.github.com&gt;
Co-authored-by: Chris Hennick &lt;4961925+Pr0methean@users.noreply.github.com&gt;
diff --git a/Cargo.toml b/Cargo.toml
@@ -49,7 +49,7 @@ zeroize = { version = "1.8", optional = true, features = ["zeroize_derive"] }
 zstd = { version = "0.13", optional = true, default-features = false }
 zopfli = { version = "0.8", optional = true }
 deflate64 = { version = "0.1.9", optional = true }
-liblzma = { version = "0.4.1", optional = true }
+lzma-rust2 = { version = "0.13", optional = true, default-features = false, features = ["std", "encoder", "optimization", "xz"] }
 
 [target.'cfg(fuzzing)'.dependencies]
 arbitrary = { version = "1.4.1", features = ["derive"] }
@@ -78,18 +78,18 @@ deflate-flate2-zlib = ["deflate-flate2", "flate2/zlib"]
 deflate-zopfli = ["dep:zopfli", "_deflate-any"]
 jiff-02 = ["dep:jiff"]
 nt-time = ["dep:nt-time"]
-lzma = ["dep:liblzma"]
-lzma-static = ["lzma", "liblzma/static"]
+lzma = ["dep:lzma-rust2"]
+lzma-static = ["lzma"]
 ppmd = ["dep:ppmd-rust"]
 unreserved = []
-xz = ["dep:liblzma"]
-xz-static = ["xz", "liblzma/static"]
+xz = ["dep:lzma-rust2"]
+xz-static = ["lzma"]
 default = [
     "aes-crypto",
     "bzip2",
     "deflate64",
     "deflate",
-    "lzma-static",
+    "lzma",
     "ppmd",
     "time",
     "zstd",
diff --git a/README.md b/README.md
@@ -10,7 +10,7 @@ Info
 ----
 
 
-A zip library for rust which supports reading and writing of simple ZIP files. Formerly hosted at 
+A zip library for rust which supports reading and writing of simple ZIP files. Formerly hosted at
 https://github.com/zip-rs/zip2.
 
 Supported compression formats:
@@ -21,7 +21,7 @@ Supported compression formats:
 * bzip2
 * zstd
 * lzma (decompression only)
-* xz (decompression only)
+* xz
 * ppmd
 
 Currently unsupported zip extensions:
@@ -35,8 +35,8 @@ The features available are:
 
 * `aes-crypto`: Enables decryption of files which were encrypted with AES. Supports AE-1 and AE-2 methods.
 * `deflate`: Enables compressing and decompressing an unspecified implementation (that may change in future versions) of
- the deflate compression algorithm, which is the default for zip files. Supports compression quality 1..=264.
-* `deflate-flate2`: Combine this with any `flate2` feature flag that enables a back-end, to support deflate compression 
+  the deflate compression algorithm, which is the default for zip files. Supports compression quality 1..=264.
+* `deflate-flate2`: Combine this with any `flate2` feature flag that enables a back-end, to support deflate compression
   at quality 1..=9.
 * `deflate-zopfli`: Enables deflating files with the `zopfli` library (used when compression quality is 10..=264). This
   is the most effective `deflate` implementation available, but also among the slowest.
@@ -48,9 +48,10 @@ The features available are:
 * `chrono`: Enables converting last-modified `zip::DateTime` to and from `chrono::NaiveDateTime`.
 * `jiff-02`: Enables converting last-modified `zip::DateTime` to and from `jiff::civil::DateTime`.
 * `nt-time`: Enables returning timestamps stored in the NTFS extra field as `nt_time::FileTime`.
+* `xz`: Enables the XZ compression algorithm.
 * `zstd`: Enables the Zstandard compression algorithm.
 
-By default `aes-crypto`, `bzip2`, `deflate`, `deflate64`, `lzma`, `ppmd`, `time` and `zstd` are enabled.
+By default `aes-crypto`, `bzip2`, `deflate`, `deflate64`, `lzma`, `ppmd`, `time`, `xz` and `zstd` are enabled.
 
 MSRV
 ----
@@ -65,12 +66,13 @@ Examples
 --------
 
 See the [examples directory](examples) for:
-   * How to write a file to a zip.
-   * How to write a directory of files to a zip (using [walkdir](https://github.com/BurntSushi/walkdir)).
-   * How to extract a zip file.
-   * How to extract a single file from a zip.
-   * How to read a zip from the standard input.
-   * How to append a directory to an existing archive
+
+* How to write a file to a zip.
+* How to write a directory of files to a zip (using [walkdir](https://github.com/BurntSushi/walkdir)).
+* How to extract a zip file.
+* How to extract a single file from a zip.
+* How to read a zip from the standard input.
+* How to append a directory to an existing archive
 
 Fuzzing
 -------
diff --git a/examples/write_dir.rs b/examples/write_dir.rs
@@ -26,6 +26,7 @@ enum CompressionMethod {
     Stored,
     Deflated,
     Bzip2,
+    Xz,
     Zstd,
 }
 
@@ -57,6 +58,15 @@ fn real_main() -> i32 {
             #[cfg(feature = "bzip2")]
             zip::CompressionMethod::Bzip2
         }
+        CompressionMethod::Xz => {
+            #[cfg(not(feature = "xz"))]
+            {
+                println!("The `xz` feature is not enabled");
+                return 1;
+            }
+            #[cfg(feature = "xz")]
+            zip::CompressionMethod::Xz
+        }
         CompressionMethod::Zstd => {
             #[cfg(not(feature = "zstd"))]
             {
diff --git a/src/compression.rs b/src/compression.rs
@@ -214,13 +214,22 @@ pub(crate) enum Decompressor<R: io::BufRead> {
     #[cfg(feature = "zstd")]
     Zstd(zstd::Decoder<'static, R>),
     #[cfg(feature = "lzma")]
-    Lzma(liblzma::bufread::XzDecoder<R>),
+    Lzma(Lzma<R>),
     #[cfg(feature = "xz")]
-    Xz(liblzma::bufread::XzDecoder<R>),
+    Xz(Box<lzma_rust2::XzReader<R>>),
     #[cfg(feature = "ppmd")]
     Ppmd(Ppmd<R>),
 }
 
+#[cfg(feature = "lzma")]
+pub(crate) enum Lzma<R: io::BufRead> {
+    Uninitialized {
+        reader: Option<R>,
+        uncompressed_size: u64,
+    },
+    Initialized(Box<lzma_rust2::LzmaReader<R>>),
+}
+
 #[cfg(feature = "ppmd")]
 pub(crate) enum Ppmd<R: io::BufRead> {
     Uninitialized(Option<R>),
@@ -240,7 +249,50 @@ impl<R: io::BufRead> io::Read for Decompressor<R> {
             #[cfg(feature = "zstd")]
             Decompressor::Zstd(r) => r.read(buf),
             #[cfg(feature = "lzma")]
-            Decompressor::Lzma(r) => r.read(buf),
+            Decompressor::Lzma(r) => match r {
+                Lzma::Uninitialized {
+                    reader,
+                    uncompressed_size,
+                } => {
+                    let mut reader = reader.take().ok_or_else(|| {
+                        io::Error::other("Reader was not set while reading LZMA data")
+                    })?;
+
+                    // 5.8.8.1 LZMA Version Information & 5.8.8.2 LZMA Properties Size
+                    let mut header = [0; 4];
+                    reader.read_exact(&mut header)?;
+                    let _version_information = u16::from_le_bytes(header[0..2].try_into().unwrap());
+                    let properties_size = u16::from_le_bytes(header[2..4].try_into().unwrap());
+                    if properties_size != 5 {
+                        return Err(io::Error::new(
+                            io::ErrorKind::InvalidInput,
+                            format!("unexpected LZMA properties size of {properties_size}"),
+                        ));
+                    }
+
+                    let mut props_data = [0; 5];
+                    reader.read_exact(&mut props_data)?;
+                    let props = props_data[0];
+                    let dict_size = u32::from_le_bytes(props_data[1..5].try_into().unwrap());
+
+                    // We don't need to handle the end-of-stream marker here, since the LZMA reader
+                    // stops at the end-of-stream marker OR when it has decoded uncompressed_size bytes, whichever comes first.
+                    let mut decompressor = lzma_rust2::LzmaReader::new_with_props(
+                        reader,
+                        *uncompressed_size,
+                        props,
+                        dict_size,
+                        None,
+                    )?;
+
+                    let read = decompressor.read(buf)?;
+
+                    *r = Lzma::Initialized(Box::new(decompressor));
+
+                    Ok(read)
+                }
+                Lzma::Initialized(decompressor) => decompressor.read(buf),
+            },
             #[cfg(feature = "xz")]
             Decompressor::Xz(r) => r.read(buf),
             #[cfg(feature = "ppmd")]
@@ -291,7 +343,12 @@ impl<R: io::BufRead> io::Read for Decompressor<R> {
 }
 
 impl<R: io::BufRead> Decompressor<R> {
-    pub fn new(reader: R, compression_method: CompressionMethod) -> crate::result::ZipResult<Self> {
+    pub fn new(
+        reader: R,
+        compression_method: CompressionMethod,
+        #[cfg(feature = "lzma")] uncompressed_size: u64,
+        #[cfg(not(feature = "lzma"))] _uncompressed_size: u64,
+    ) -> crate::result::ZipResult<Self> {
         Ok(match compression_method {
             CompressionMethod::Stored => Decompressor::Stored(reader),
             #[cfg(feature = "deflate-flate2")]
@@ -307,15 +364,14 @@ impl<R: io::BufRead> Decompressor<R> {
             #[cfg(feature = "zstd")]
             CompressionMethod::Zstd => Decompressor::Zstd(zstd::Decoder::with_buffer(reader)?),
             #[cfg(feature = "lzma")]
-            CompressionMethod::Lzma => Decompressor::Lzma(liblzma::bufread::XzDecoder::new_stream(
-                reader,
-                // Use u64::MAX for unlimited memory usage, matching the previous behavior
-                // from lzma-rs. Using 0 would set the smallest memory limit, which is
-                // problematic in ancient liblzma versions (5.2.3 and earlier).
-                liblzma::stream::Stream::new_lzma_decoder(u64::MAX).unwrap(),
-            )),
+            CompressionMethod::Lzma => Decompressor::Lzma(Lzma::Uninitialized {
+                reader: Some(reader),
+                uncompressed_size,
+            }),
             #[cfg(feature = "xz")]
-            CompressionMethod::Xz => Decompressor::Xz(liblzma::bufread::XzDecoder::new(reader)),
+            CompressionMethod::Xz => {
+                Decompressor::Xz(Box::new(lzma_rust2::XzReader::new(reader, false)))
+            }
             #[cfg(feature = "ppmd")]
             CompressionMethod::Ppmd => Decompressor::Ppmd(Ppmd::Uninitialized(Some(reader))),
             _ => {
@@ -340,7 +396,12 @@ impl<R: io::BufRead> Decompressor<R> {
             #[cfg(feature = "zstd")]
             Decompressor::Zstd(r) => r.finish(),
             #[cfg(feature = "lzma")]
-            Decompressor::Lzma(r) => r.into_inner(),
+            Decompressor::Lzma(r) => match r {
+                Lzma::Uninitialized { mut reader, .. } => reader
+                    .take()
+                    .ok_or_else(|| io::Error::other("Reader was not set"))?,
+                Lzma::Initialized(decoder) => decoder.into_inner(),
+            },
             #[cfg(feature = "xz")]
             Decompressor::Xz(r) => r.into_inner(),
             #[cfg(feature = "ppmd")]
diff --git a/src/lib.rs b/src/lib.rs
@@ -23,7 +23,7 @@
 //! | Bzip2 | ✅ | ✅ |
 //! | ZStandard | ✅ | ✅ |
 //! | LZMA | ✅ | |
-//! | XZ | ✅ | |
+//! | XZ | ✅ | ✅ |
 //! | PPMd | ✅ | ✅ |
 //! | AES encryption | ✅ | ✅ |
 //! | ZipCrypto deprecated encryption | ✅ | ✅ |
diff --git a/src/read.rs b/src/read.rs
@@ -426,13 +426,18 @@ pub(crate) fn make_crypto_reader<'a, R: Read>(
 
 pub(crate) fn make_reader<R: Read>(
     compression_method: CompressionMethod,
+    uncompressed_size: u64,
     crc32: u32,
     reader: CryptoReader<R>,
 ) -> ZipResult<ZipFileReader<R>> {
     let ae2_encrypted = reader.is_ae2_encrypted();
 
     Ok(ZipFileReader::Compressed(Box::new(Crc32Reader::new(
-        Decompressor::new(io::BufReader::new(reader), compression_method)?,
+        Decompressor::new(
+            io::BufReader::new(reader),
+            compression_method,
+            uncompressed_size,
+        )?,
         crc32,
         ae2_encrypted,
     ))))
@@ -1134,7 +1139,12 @@ impl<R: Read + Seek> ZipArchive<R> {
 
         Ok(ZipFile {
             data: Cow::Borrowed(data),
-            reader: make_reader(data.compression_method, data.crc32, crypto_reader)?,
+            reader: make_reader(
+                data.compression_method,
+                data.uncompressed_size,
+                data.crc32,
+                crypto_reader,
+            )?,
         })
     }
 
@@ -1900,13 +1910,17 @@ pub fn read_zipfile_from_stream<R: Read>(reader: &mut R) -> ZipResult<Option<Zip
 
     let limit_reader = reader.take(result.compressed_size);
 
-    let result_crc32 = result.crc32;
-    let result_compression_method = result.compression_method;
     let crypto_reader = make_crypto_reader(&result, limit_reader, None, None)?;
+    let ZipFileData {
+        crc32,
+        uncompressed_size,
+        compression_method,
+        ..
+    } = result;
 
     Ok(Some(ZipFile {
         data: Cow::Owned(result),
-        reader: make_reader(result_compression_method, result_crc32, crypto_reader)?,
+        reader: make_reader(compression_method, uncompressed_size, crc32, crypto_reader)?,
     }))
 }
 
diff --git a/src/write.rs b/src/write.rs
@@ -98,7 +98,7 @@ enum GenericZipWriter<W: Write + Seek> {
     #[cfg(feature = "zstd")]
     Zstd(ZstdEncoder<'static, MaybeEncrypted<W>>),
     #[cfg(feature = "xz")]
-    Xz(liblzma::write::XzEncoder<MaybeEncrypted<W>>),
+    Xz(Box<lzma_rust2::XzWriter<MaybeEncrypted<W>>>),
     #[cfg(feature = "ppmd")]
     Ppmd(Box<ppmd_rust::Ppmd8Encoder<MaybeEncrypted<W>>>),
 }
@@ -121,7 +121,7 @@ impl<W: Write + Seek> Debug for GenericZipWriter<W> {
             #[cfg(feature = "zstd")]
             GenericZipWriter::Zstd(w) => f.write_fmt(format_args!("Zstd({:?})", w.get_ref())),
             #[cfg(feature = "xz")]
-            GenericZipWriter::Xz(w) => f.write_fmt(format_args!("Xz({:?})", w.get_ref())),
+            GenericZipWriter::Xz(w) => f.write_fmt(format_args!("Xz({:?})", w.inner())),
             #[cfg(feature = "ppmd")]
             GenericZipWriter::Ppmd(_) => f.write_fmt(format_args!("Ppmd8Encoder")),
         }
@@ -1801,8 +1801,12 @@ impl<W: Write + Seek> GenericZipWriter<W> {
                         .ok_or(UnsupportedArchive("Unsupported compression level"))?
                         as u32;
                     Ok(Box::new(move |bare| {
-                        Ok(GenericZipWriter::Xz(liblzma::write::XzEncoder::new(
-                            bare, level,
+                        Ok(GenericZipWriter::Xz(Box::new(
+                            lzma_rust2::XzWriter::new(
+                                bare,
+                                lzma_rust2::XzOptions::with_preset(level),
+                            )
+                            .map_err(ZipError::Io)?,
                         )))
                     }))
                 }
diff --git a/tests/bug398.rs b/tests/bug398.rs
diff --git a/tests/data/lzma.zip b/tests/data/lzma.zip
diff --git a/tests/lzma.rs b/tests/lzma.rs
@@ -10,12 +10,12 @@ fn decompress_lzma() {
     let mut archive = ZipArchive::new(io::Cursor::new(v)).expect("couldn't open test zip file");
 
     let mut file = archive
-        .by_name("hello.txt")
+        .by_name("binary.wmv")
         .expect("couldn't find file in archive");
-    assert_eq!("hello.txt", file.name());
+    assert_eq!("binary.wmv", file.name());
 
     let mut content = Vec::new();
     file.read_to_end(&mut content)
         .expect("couldn't read encrypted and compressed file");
-    assert_eq!("Hello world\n", String::from_utf8(content).unwrap());
+    assert_eq!(include_bytes!("data/folder/binary.wmv"), &content[..]);
 }
