Numpy with Apache Arrow Flight

A Problem

We provide

• An abstract base class for an Apache flight server
• A client class to communicate with such servers

The client provides a simple interface for sending NumPy arrays, performing computations, and retrieving results, all while handling the serialization and deserialization automatically in the background using Apache Arrow.

To create a server we expect the user to overload a function performing the calcutation based on a dictionary of numpy arrays.

Features

• Seamless conversion between NumPy arrays and Arrow Tables
• Simple interface for data transfer operations
• Type-safe operations with proper error handling

Installation

You can install this client via

pip install numpy-flight

Usage

Basic Setup

We introduce the Baseclass 'Server':

>>> from flight import Server

>>> class TestServer(Server):
...     def f(self, matrices):
...          self.logger.info(f"{matrices.keys()}")
...          # Simple implementation for testing - just return the input
...          return {key : 2*value for key, value in matrices.items()}

All complexity is hidden in the class 'Server' which is itself a child of the pyarrrow's FlightServerBase class. It is enough to implement the method 'f' which is expecting a dictionary of numpy arrays. It will also return a dictionary of numpy arrays.

The server can be started locally with

>>> server = TestServer.start(host="127.0.0.1", port=5555)

While the server is running we can use a Python client for computations

>>> import numpy as np
>>> from flight import Client

>>> with Client(location="grpc://127.0.0.1:5555") as client:
...     output = client.compute(command="compute", data={"input": np.array([1,2,3])})

>>> print(output["input"])
[2 4 6]

Clients for other languages are thinkable. We shut the server down with

server.shutdown()