Add filebundle (otioz and otiod) adapters

[ssteinbach]

overview

Introduces the idea of an OTIO "file bundle". This has two flavors: .otioz and .otiod. This PR includes documentation and implements adapters that can convert into those formats.

otioz

• Adds .otioz suffix file
• Bundles media and an otio file together into a libzip compatible zipfile (using python's zipfile module)
• only works with files that can be found through the filesystem (file: protocol target_url fields, as parsed by urllib in python)
• media files must have unique basenames (because they get put into a flat namespace)
• zipfile structure (for foo.otioz):
  • foo.otioz/
    • version.txt
    • content.otio
    • media/
      • file1
      • file2
      • file3
• the content.otio encodes the structure of the timeline and exclusively references media which is present in the media subdirectory.
• the content.otio is compressed, but the rest of the media files are not
• version.txt is only present in otioz files and exclusively encodes a file bundle version string into the file in case of future changes to the layout of file bundles

otiod

• identical to the otioz in layout, except expressed without the zipfile container as files and directories in the filesystem.
• This way a .otioz can be expanded using otioconvert through the adapter system without needing to use zip directly, into a form that any system that reads otio from the filesystem can consume.

TODO:

• make the file:// vs file: consistent when generating absolute paths
• [ ] add a file:///path/to/otioz?file=media/somefile.mov form of ExternalReference.target_url when reading OTIOZ files... this allows the otiod adapter the ability to directly decompress otioz files, and otioconvert could detect it when going from otioz to otiod. -- going to do this in a future PR/issue

[codecov-io]

Codecov Report

Merging #561 (eb0f397) into master (1a267c5) will increase coverage by 0.03%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master     #561      +/-   ##
==========================================
+ Coverage   84.33%   84.36%   +0.03%     
==========================================
  Files          74       74              
  Lines        3090     3090              
==========================================
+ Hits         2606     2607       +1     
+ Misses        484      483       -1     

Flag	Coverage Δ
unittests	84.36% <ø> (+0.03%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
...ntimelineio/opentimelineio-bindings/otio_tests.cpp	73.68% <0.00%> (+1.75%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1a267c5...eb0f397. Read the comment docs.

[jminor]

I like this a lot. It seems tremendously useful and is very simple in concept.

If you require the internal otio to have the same name as the zip file, then it will break if/when someone renames the zip file. Using a consistent name, like content.otio would avoid this.

On the reading side, how would someone extract the media files? Maybe there could be a helper function or option on the reader to unzip the bundle? That would also allow for a future switch from zip to some other bundle mechanism, if needed.

[ssteinbach]

Great note! I like content.otioz.

As far as a command to unzip the bundle, do you mean some kind of api function you can call? otio.adapters.expand_bundle() or something?

I like the idea of putting an API call that encapsulates the schema though. Also I wonder if we can put an extra magic number into the file somehow to detect which 'version' of OTIOZ it is.

[jminor]

You could put a version in OTIOZ metadata on the top level object, or wrap the content in a Bundle.1 schema type or something. I bet the IMF and DCP folks would have advice on what else would be worth tracking, though those formats tend to have a bunch of manifest details, checksums, etc. that might make this more complicated than it needs to be.

[reinecke]

This is super interesting.
I think it also may be useful to be able to selectively extract media from the otioz (i.e. give me the essence bytes for this specific media reference).
There are some nitpicky considerations here like how we uniqify filenames in the archive and what the conventions should be around constructing target_urls so it's clear how to locate them within the archive (Here is an interesting discussion I found with some interesting conventions around this: frictionlessdata/datapackage#137).

I think the idea of dropping some well-named file in the archive with version info is a good one that serves a couple purposes. Relying on files to keep their extensions all the time can be a little risky, so it's helpful to have some signal in the content of the file that can help you identify that it's a special kind of zip.

Overall, I'd love to see this in the hands of users and see what ideas they have!

[ssteinbach]

On the reading side, how would someone extract the media files? Maybe there could be a helper function or option on the reader to unzip the bundle?

What if we had two adapters: otioz and otiod. otiod (d for directory) is just a directory on the filesystem that meets the conditions for compressing into an otioz file. You could use the otiod adapter for decompressing an otioz file into the the filesystem is what I was thinking about.

[reinecke]

Just a few minor comments. This is looking really cool!

[reinecke]

This isn't a requirement, but using urlparse for operating on these urls may help add clarity:

try:
    # Python 2.7
    import urlparse
except ImportError:
    # Python 3
    import urllib.parse as urlparse

parsed_url = urlparse.urlparse(target_url)
if not parsed_url.scheme == "file":
    ...

# And for line 94:
target_file = parsed_url.path

[ssteinbach]

that is way better, good call.

[ssteinbach]

We'll version in the OTIO file rather than come up with a separate versioning scheme specifically for otioz.

[ssteinbach]

Rebased onto current master branch.

[ssteinbach]

Because we don't have a concrete need for this at the moment, and there is concern that this might expand the scope of OTIO (because this is a file format that includes media), we've decided to close this for now. The branch will remain in my fork so that if there is a desire for this in the future, this work can be brought forward and re-applied. Please let us know if you have any questions!

[jminor]

I came across this SMPTE spec for AXF which attempts to address a similar problem:
https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=7879152

It looks like AXF attempts to address issues of very large data sets, as well as a bunch of extra metadata about each item in the archive.

I'm just posting this here for future reference, since OTIOZ is likely to come up again in the future.

[meshula]

using six here is nice

[ssteinbach]

Thank you to the community for the thorough review.