Index v2 (#4387)

## Change
This change introduces schema 2.0 for the `SQLiteIndex`. This new major
version takes the learnings on how we actually used the index to shift
to a package centralized store. It also moves some of the data that is
not directly needed for search and correlation to intermediate manifests
per-package. The goal is that `search` and `list` functionality
(including `update` probing) should not need the intermediate files;
only operations that change the system state (write operations) should.

Using a recent index as a starting point for comparison, the 2.0 index
reduced the size of a ZIP archive (containing the 1 file and produced by
Windows Explorer) by **79%**.

This change only implements the index creation. A future change will
implement the consumption, including a correct implementation of the 2.0
search functionality. The current search is just copied/commented from a
previous schema version for now.

### Implementation
The 1.* index schema allows easy updating from individual manifest
changes, without the need to inspect any other manifests for the same
package. The final 2.* schema will not allow that. Instead, 2.0 actually
uses a 1.7 schema internally, with an additional table to track changes
so that we can produce the intermediate manifests. When
`PrepareForPackaging` is called, the schema 2.0 tables are created and
the data migrated to them. The intermediate manifest files that have
changed are also written to disk.

### Intermediate Manifests
The intermediate manifest files (`PackageVersionDataManifest`) are YAML
with shortened key values. This saves some bytes since humans are
neither authoring them nor reading them (except to debug). They are also
stored in a compressed stream, using the MSZIP compression algorithm.
The compression brings the average size down from 1236 bytes to 463
bytes, and the median down from 338 bytes to 206 bytes.

The future consumption change will cache these intermediate manifests
(and likely the version manifests as well), allowing their reuse as long
as they have not changed.

### Additional Functionality
To support our use in creating the index in our services, some
additional functionality was added to the `SQLiteIndex`.

#### Migration
A schema version migration function is added, allowing the target schema
to migrate from the existing one as it sees fit. If the migration is not
supported, it can simply return a value indicating that. Only 1.7 => 2.0
migration is implemented.

#### Properties
Properties can be set on the index object, some of which are for that
object only and some of which are persisted into the database itself. An
implicit property of the database file name is stored when appropriate.
The caller can set the directory path to output intermediate files to.
The caller can also set the time (in Unix epoch) to use as the baseline
for which intermediate files should be output [in practice, only an
empty string (for "now") and "0" (to output everything) are likely to be
used].

Author: JohnMcPMS

.github/actions/spelling/expect.txt

@@ -72,6 +72,7 @@ CODEOWNERS
 COINIT
 COMGLB
 commandline
+compressapi
 contactsupport
 contentfiles
 contoso
@@ -273,6 +274,8 @@ msftrubengu
 MSIHASH
 MSIXHASH
 msstore
+MSZIP
+mszyml
 Mta
 Mugiwara
 Multideclaration

src/AppInstallerCLITests/AppInstallerCLITests.vcxproj

@@ -232,6 +232,7 @@
     <ClCompile Include="PackageCollection.cpp" />
     <ClCompile Include="PackageDependenciesValidationUtil.cpp" />
     <ClCompile Include="PackageTrackingCatalog.cpp" />
+    <ClCompile Include="PackageVersionDataManifest.cpp" />
     <ClCompile Include="PathVariable.cpp" />
     <ClCompile Include="PinFlow.cpp" />
     <ClCompile Include="PinningIndex.cpp" />

src/AppInstallerCLITests/AppInstallerCLITests.vcxproj.filters

@@ -344,6 +344,9 @@
     <ClCompile Include="RestInterface_1_7.cpp">
       <Filter>Source Files\Repository</Filter>
     </ClCompile>
+    <ClCompile Include="PackageVersionDataManifest.cpp">
+      <Filter>Source Files\Common</Filter>
+    </ClCompile>
   </ItemGroup>
   <ItemGroup>
     <None Include="PropertySheet.props" />

src/AppInstallerCLITests/PackageVersionDataManifest.cpp

@@ -0,0 +1,103 @@
+// Copyright (c) Microsoft Corporation.
+// Licensed under the MIT License.
+#include "pch.h"
+#include "TestCommon.h"
+#include <winget/PackageVersionDataManifest.h>
+
+using namespace TestCommon;
+using namespace AppInstaller;
+using namespace AppInstaller::Manifest;
+using namespace AppInstaller::Utility;
+
+void RequireVersionDataEqual(const PackageVersionDataManifest::VersionData& first, const PackageVersionDataManifest::VersionData& second)
+{
+    REQUIRE(first.Version == second.Version);
+    REQUIRE(first.ArpMinVersion == second.ArpMinVersion);
+    REQUIRE(first.ArpMaxVersion == second.ArpMaxVersion);
+    REQUIRE(first.ManifestRelativePath == second.ManifestRelativePath);
+    REQUIRE(first.ManifestHash == second.ManifestHash);
+}
+
+TEST_CASE("PackageVersionDataManifest_Empty", "[PackageVersionDataManifest]")
+{
+    PackageVersionDataManifest original;
+
+    PackageVersionDataManifest copy;
+    copy.Deserialize(original.Serialize());
+
+    REQUIRE(original.Versions().empty());
+    REQUIRE(copy.Versions().empty());
+}
+
+TEST_CASE("PackageVersionDataManifest_Single_Simple", "[PackageVersionDataManifest]")
+{
+    PackageVersionDataManifest original;
+    original.AddVersion({ VersionAndChannel{ Version{ "1.0" }, Channel{} }, {}, {}, "path", "hash" });
+
+    PackageVersionDataManifest copy;
+    copy.Deserialize(original.Serialize());
+
+    REQUIRE(original.Versions().size() == 1);
+    REQUIRE(copy.Versions().size() == 1);
+
+    RequireVersionDataEqual(copy.Versions()[0], original.Versions()[0]);
+}
+
+TEST_CASE("PackageVersionDataManifest_Single_Complete", "[PackageVersionDataManifest]")
+{
+    PackageVersionDataManifest original;
+    original.AddVersion({ VersionAndChannel{ Version{ "1.0" }, Channel{} }, ".99", "1.01", "path", "hash"});
+
+    PackageVersionDataManifest copy;
+    copy.Deserialize(original.Serialize());
+
+    REQUIRE(original.Versions().size() == 1);
+    REQUIRE(copy.Versions().size() == 1);
+
+    RequireVersionDataEqual(copy.Versions()[0], original.Versions()[0]);
+}
+
+TEST_CASE("PackageVersionDataManifest_Multiple", "[PackageVersionDataManifest]")
+{
+    PackageVersionDataManifest original;
+    original.AddVersion({ VersionAndChannel{ Version{ "1.0" }, Channel{} }, ".99", "1.01", "path", "hash" });
+    original.AddVersion({ VersionAndChannel{ Version{ "1.1" }, Channel{} }, "1.99", "2.01", "path2", "hash2" });
+    original.AddVersion({ VersionAndChannel{ Version{ "1.2" }, Channel{} }, {}, {}, "path2", "hash2" });
+    original.AddVersion({ VersionAndChannel{ Version{ "2.0" }, Channel{} }, "3.99", "15.01", "path4", "hash4" });
+
+    PackageVersionDataManifest copy;
+    copy.Deserialize(original.Serialize());
+
+    REQUIRE(original.Versions().size() == copy.Versions().size());
+
+    for (size_t i = 0; i < original.Versions().size(); ++i)
+    {
+        INFO(i);
+        RequireVersionDataEqual(copy.Versions()[i], original.Versions()[i]);
+    }
+}
+
+TEST_CASE("PackageVersionDataManifest_CompressionRoundTrip", "[PackageVersionDataManifest]")
+{
+    PackageVersionDataManifest original;
+    original.AddVersion({ VersionAndChannel{ Version{ "1.0" }, Channel{} }, ".99", "1.01", "path", "hash" });
+    original.AddVersion({ VersionAndChannel{ Version{ "1.1" }, Channel{} }, "1.99", "2.01", "path2", "hash2" });
+    original.AddVersion({ VersionAndChannel{ Version{ "1.2" }, Channel{} }, {}, {}, "path2", "hash2" });
+    original.AddVersion({ VersionAndChannel{ Version{ "2.0" }, Channel{} }, "3.99", "15.01", "path4", "hash4" });
+
+    std::string serialized = original.Serialize();
+    auto compressed = PackageVersionDataManifest::CreateCompressor().Compress(serialized);
+
+    auto decompressed = PackageVersionDataManifest::CreateDecompressor().Decompress(compressed);
+
+    PackageVersionDataManifest copy;
+    copy.Deserialize(decompressed);
+
+    REQUIRE(original.Versions().size() == copy.Versions().size());
+
+    for (size_t i = 0; i < original.Versions().size(); ++i)
+    {
+        INFO(i);
+        RequireVersionDataEqual(copy.Versions()[i], original.Versions()[i]);
+    }
+}