Can't install locally with `pip install -e .`

[pedrolabonia]

Can't seem to install locally with pip install -e .

Main error is: ModuleNotFound

Build Problem

git clone https://github.com/pedrolabonia/OpenTimelineIO.git --recursive           
Cloning into 'OpenTimelineIO'...
remote: Enumerating objects: 6444, done.
...
...
➜  otio cd OpenTimelineIO 
➜  OpenTimelineIO git:(main) python -m venv .venv
➜  OpenTimelineIO git:(main) source .venv/bin/activate
➜  OpenTimelineIO git:(main) (.venv) python -m pip install --upgrade pip
Collecting pip
  Using cached pip-22.0.4-py3-none-any.whl (2.1 MB)
Successfully installed pip-22.0.4
➜  OpenTimelineIO git:(main) (.venv) python -m pip install -e .
Obtaining file:///Users/pedrolabonia/dev/otio/OpenTimelineIO
  Installing build dependencies ... done
  Checking if build backend supports build_editable ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Collecting pyaaf2~=1.4.0
  Using cached pyaaf2-1.4.0-py3-none-any.whl
Installing collected packages: pyaaf2, OpenTimelineIO
  Running setup.py develop for OpenTimelineIO
Successfully installed OpenTimelineIO-0.15.0.dev1 pyaaf2-1.4.0

➜  OpenTimelineIO git:(main) (.venv) python
Python 3.9.1 (default, Mar 21 2022, 17:23:00) 
[Clang 13.0.0 (clang-1300.0.29.30)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import opentimelineio
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'opentimelineio'

➜  OpenTimelineIO git:(main) (.venv) pip list
Package        Version     Editable project location
-------------- ----------- -------------------------------------------
OpenTimelineIO 0.15.0.dev1 /Users/pedrolabonia/dev/otio/OpenTimelineIO
pip            22.0.4
pyaaf2         1.4.0
setuptools     49.2.1

To Reproduce

1. Mac OS 11.6
2. Python 3.9.1
3. OTIO ea64d92
4. Compiler information:

➜  OpenTimelineIO git:(main) (.venv) clang -v
Apple clang version 13.0.0 (clang-1300.0.29.30)
Target: arm64-apple-darwin20.6.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

Expected Behavior

Expected to be able to install locally with editable mode. I can install it normally via pip install opentimelineio. I'm trying to figure out what's going on. There's a .egg-link file in site_packages which seems to be correctly pointing to the root. Contents of the .egg-link:

/Users/pedrolabonia/dev/otio/OpenTimelineIO
.

[JeanChristopheMorinPerso]

Hi @pedrolabonia , editable installs are not working for now in OTIO due to the uncommon source files layout that OTIO uses.

The problem is that the .egg-link points to the root of the repo, which doesn't contain any source files, let alone the built extensions.

I personally still haven't figured out a solution to that (except the brute force approach which is manually editing the link to point to the build/... directory) but maybe others already have.

[pedrolabonia]

Oh, I see.

Even then, installing without the -e flag still gives me the same error. Guess I should still manually alter the egg-link file?

I might be confused here, but the build directory only has the C++ modules, right? I'm trying to develop for Python only.

I'm not very familiar with C++, and building it for Python. Couldn't I "freeze" the otio C++ lib, and then use an editable install for the python bindings only? I'm trying to alter something in the adapters.

Edit:
Tried a hack, dynamically importing the .so modules into python, could not:

import importlib.util
import sys
def alternative_load_dynamic(name, path, file=None):
    spec = importlib.util.spec_from_file_location(name, path)
    module = importlib.util.module_from_spec(spec)
    sys.modules[name] = module
    spec.loader.exec_module(module)
    return sys.modules[name]



alternative_load_dynamic('opentimelineio', './build/lib.macosx-11.6-x86_64-cpython-39/opentimelineio/_otio.cpython-39-darwin.so')
alternative_load_dynamic('opentime', './build/lib.macosx-11.6-x86_64-cpython-39/opentimelineio/_opentime.cpython-39-darwin.so')

Traceback (most recent call last):
  File "/Users/pedrolabonia/dev/otio/OpenTimelineIO/t.py", line 12, in <module>
    alternative_load_dynamic('opentimelineio', './build/lib.macosx-11.6-x86_64-cpython-39/opentimelineio/_otio.cpython-39-darwin.so')
  File "/Users/pedrolabonia/dev/otio/OpenTimelineIO/t.py", line 5, in alternative_load_dynamic
    module = importlib.util.module_from_spec(spec)
  File "<frozen importlib._bootstrap>", line 565, in module_from_spec
  File "<frozen importlib._bootstrap_external>", line 1108, in create_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
ImportError: dynamic module does not define module export function (PyInit_opentimelineio)

[JeanChristopheMorinPerso]

pip install . should just work, but you'll have to pip uninstall opentimelineio to remove the artifacts installed when running pip install -e ..

I might be confused here, but the build directory only has the C++ modules, right?

No, it contains everything needed.

What you can try is this:

1. python setup.py build_ext build_py (that will create the build dir for you, with the python files and the built extensions)
2. pip install -e .
3. Modify the .egg-link to point to ./build/lib.macosx-11.6-x86_64-cpython-39

And from there, you can symlink the python files you want into /build/lib.macosx-11.6-x86_64-cpython-39 so that you don't have to re-run python setup.pu build_py everytime. It's hacky but it will work (it's what I normally do).

[JeanChristopheMorinPerso]

The other way is to simply copy the .so into src/py-opentimelineio/opentimelineio and add src/py-opentimelineio to your PYTHONPATH.

1. python setup.py build_ext
2. Copy copy the .sos into src/py-opentimelineio/opentimelineio
3. Set PYTHONPATH to src/py-opentimelineio

[JeanChristopheMorinPerso]

Obviously I'd really like to get the editable installs work, but we'll have to do research and it's possible that there is no solution for extensions right now.

[pedrolabonia]

Awesome!

pip install . did work. I guess I might have messed something up. There was an old global install I did on accident, that was probably it. The .egg-link solution didn't. Copying the .so modules and altering PYTHON_PATH works, although I used a different method, using the a .pth file in site-packages inside the venv.

For any one that encounters this problem again, here's the full walkthrough:

python -m venv .venv
python -m pip install --upgrade pip
pip install cmake
python setup.py build_ext
find build/lib.*/ -name '*.so' -exec cp "{}" src/py-opentimelineio/opentimelineio/ \;
echo "../../../../src/py-opentimelineio/" > .venv/lib/*/site-packages/opentimelineio.pth

Should we maybe include that in the Contributing guide? I know it's a temporary fix, but it might lower the barrier of entry for python developers who aren't that familiar with the more complex build paths such as myself.

I don't know if there's a way for setuptools to know if it's an editable install. If so, we could maybe try to execute those steps through the setup.py.

[JeanChristopheMorinPerso]

I personally don't think it should go into the contributing guide. It's way too hacky. I also wouldn't add this solution to the setup.py script because it wouldn't work for developers working on the extensions (_otio and _opentime).