Shared memory cache across multiple processes? #3488
Unanswered
golmschenk
asked this question in
Q&A
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
Hello! Is there an easy way in Zarr to have the in-memory cache be shared by multiple processes? I have several processes that need to load a subset of a large dataset. Caching the subset would be useful in my case. However, if each process had a separate copy of that subset, I would still run out of memory in my case. So I would like to have the multiple processes share the same underlying cache if possible. If there's a known way in which this would work using built-in Zarr capabilities, that would be ideal. I can resort manually creating the shared memory object if needed, but there are a lot of pitfalls going through that method, so I'd like to avoid it if possible. If any one knowns if this is already existing functionality or should be easy to implement using Zarr capabilities, please let me know! Or, if you're someone that is pretty confident they know that this is probably not easy to do with existing Zarr capabilities, that would be useful to know too! Thank you for you time!
Beta Was this translation helpful? Give feedback.
All reactions