Go grpc client (#204)

* Squashed commit of the following:

commit 46b65dd
Author: David Onuh <[email protected]>
Date:   Mon Aug 12 08:58:14 2024 +0100

    AIProxy Set Command Preprocessing actions for storeinputs (#77)

    * Add a new field preprocess_action for aiproxy set command

    * Add preprocess_action field to python client

    * Add new fields for aiproxy create store commands

    * Update python client after changes to create store command

    * Remove index and query types from aiproxy create store

    * Update python client with changes to aiproxy create store params

    * - Pass preproccess actions to set command
    - Create base on how to edit inputtypes and convert to storekeys
    - Move preprocessing actions
    - Tie model info into aimodel
    - Fix fmt trait on AIStoreInputTypes

    * regenerate types for python client

    * Create ai model struct to hold info about supported models, Implement additional functions on aimodel types via traits

    * Cleanup and rename aistoreinputtypes

    * Improvements to preprocessing logic, remove &mut references

commit 0325f15
Author: David Onuh <[email protected]>
Date:   Wed Jul 31 21:45:19 2024 +0100

    Rename StoreInput and MetadataValue Variant From Binary to Image (#76)

    * Change storeinput and metavalue variant from binary to image, remove store_type from aiproxy create_store

    * Fix python lib after change in types and update readme

commit 44b6181
Author: David Onuh <[email protected]>
Date:   Wed Jul 31 19:14:06 2024 +0100

    Create Rust AIProxy Client  (#75)

    * WIP: AIProxy Client -  Move client's connection send and read logic to trait

    * Create AIproxy rust client

    * add some tests to rust ai proxy client

    * update documentation for ai proxy

    * add tests for aiproxy get pred command

    * AI and DB Conn struct rename and doc update

    * Add more binary store tests, Change AIProxyServer, now owns aiproxyconfig

    * update aiclient get pred tests and remove unnessary clone

commit 6740dba
Author: Diretnan Domnan <[email protected]>
Date:   Thu Jul 25 00:54:24 2024 +0200

    Check connected clients before accepting connection (#74)

commit 277fda3
Author: David Onuh <[email protected]>
Date:   Wed Jul 24 22:44:31 2024 +0100

    cleanup golang typegen (#73)

commit ae272ff
Merge: a0766ad 6bf53d7
Author: David Onuh <[email protected]>
Date:   Wed Jul 24 22:13:28 2024 +0100

    Merge pull request #72 from deven96/diretnan/fix-persistence

    Fixing persistence issues from predicate index

commit 6bf53d7
Author: Diretnan Domnan <[email protected]>
Date:   Wed Jul 24 22:51:28 2024 +0200

    Fixing persistence issues from predicate index

commit a0766ad
Author: David Onuh <[email protected]>
Date:   Mon Jul 22 21:01:07 2024 +0100

    AI Proxy Persistence (#70)

    * Add persistence for ahnlich ai and add new command(destroy_database) for ai_proxy

    * update destroydb command to purge stores

    * add ai purge commands to python ai client and update readme

commit 20b6c63
Author: David Onuh <[email protected]>
Date:   Mon Jul 22 11:35:45 2024 +0100

    Add AI Proxy client  (#71)

    * seperate db client and tests from aiproxy

    * create ai base client and cleanup

    * default args cleanup

    * rename pool class

    * fleshout ai client commands

    * fix typespec with incomplete servertypes

    * update python client test workflow for aiproxy

    * add aiproxy tests for store commands

    * isolate db tests

    * update readme, update ai test and remove dependency on ci spinning up db and ai servers

    * fix typegen, trace AIStoreType

    * bump python client version

    * cleanup workflow

commit c3d8f32
Author: David Onuh <[email protected]>
Date:   Thu Jul 18 21:11:19 2024 +0100

    Python client connection pooling fix and add context manager capability to protocol (#69)

    * utilize connection pool properly. Add ability to use protocol as a context

    * add formatting and linting to python actions

    * Refactor: Make protocol dep internal
    - extract base class needed by clients.
    - Add ai_response, ai_query
    - Update tests and Readme

    * format

    * fix python import path and update readme

commit 5c397a1
Author: David Onuh <[email protected]>
Date:   Tue Jul 16 15:05:03 2024 +0100

    Setup Basic AI Proxy Types (#49)

    * Create ai-proxy query enum

    * Use storeinputs instead of metadatavalue

    * add ai server response for queries

    * add aiserver response

    * Trace ai db queries and server responses for clients

    * Setup entry point for ai_proxy, using db parts

    * WIP: moving ahnlich protocol to traits

    * Use traits to handle ahnlich protocol

    * Cleanup unused errors by ai

    * rename to ai instead of ahnlich_ai

    * Fleshing out commands

    * Adding dbclient to tasks

    * Move dbclient to aiproxy task and some cleanups

    * Change ai set query format

    * Add set command for ai proxy

    * Add getsimn variant

    * Update ai query commands to match db(create_pred_index, drop_pred_index)

    * Initialize reserved metadatakey

    * Add ai allocator and begin tests for ai

    * More tests for ahnlich ai proxy

    * remove test for unavailability

    * update todos and regen typespecs

    * Match portion of error in aiproxytests

commit 94eb429
Author: David Onuh <[email protected]>
Date:   Tue Jul 16 15:03:49 2024 +0100

    update typegen golang format (#68)

    * update typegen format

    * clippy fmt

    * Change golang typegen format to generate mods for each file

* feat: add publich test file

* feat: add grpc proto files

* Changes to proto and workspace

* Added tests for go client

* uncommented python generation

* uncommented python generation

* Tests working

* feat: add go test to actions

* chore: format and lint go-files

* Added warm up to tests

* Removed unwanted tests

* Removed warm up programmatically

* feat: fix benchmark with docker image cache

* feat: add readme

* chore: fix lint and formatter

* fix tests

---------

Co-authored-by: David Eje <[email protected]>
Co-authored-by: Diretnan Domnan <[email protected]>

Author: 3 people

.github/workflows/benchmark.yml

@@ -22,9 +22,16 @@ jobs:
       uses: Swatinem/rust-cache@v2
       with:
         workspaces: ahnlich
-
-    - name: Install Protoc
-      uses: arduino/setup-protoc@v3
+    
+    - name: Install protoc
+      run: |
+        sudo apt-get update
+        sudo apt-get install -y protobuf-compiler
+    
+    - name: Cache Docker images.
+      uses: ScribeMD/[email protected]
+      with:
+        key: ${{ runner.os }}-cargo-${{ hashFiles('ahnlich/Cargo.lock') }}
 
     - name: Install Cargo benchcmp
       working-directory: ./ahnlich

.github/workflows/rust_tag_and_deploy.yml

@@ -68,8 +68,10 @@ jobs:
         with:
           toolchain: 1.88.0
 
-      - name: Install Protoc
-        uses: arduino/setup-protoc@v3
+      - name: Install protoc
+        run: |
+          sudo apt-get update
+          sudo apt-get install -y protobuf-compiler
 
       - name: Deploy using tag
         working-directory: ./ahnlich

.github/workflows/test.yml

@@ -30,8 +30,10 @@ jobs:
       run: |
         which cargo-nextest || cargo install cargo-nextest
 
-    - name: Install Protoc
-      uses: arduino/setup-protoc@v3
+    - name: Install protoc
+      run: |
+        sudo apt-get update
+        sudo apt-get install -y protobuf-compiler
     
     - name: Cache Docker images.
       uses: ScribeMD/[email protected]
@@ -109,8 +111,10 @@ jobs:
         poetry run isort . -c --profile black; echo $?
         poetry run black . --check; echo $?
 
-    - name: Install Protoc
-      uses: arduino/setup-protoc@v3
+    - name: Install protoc
+      run: |
+        sudo apt-get update
+        sudo apt-get install -y protobuf-compiler
 
     - name: Complie Binaries
       working-directory: ./ahnlich
@@ -143,13 +147,51 @@ jobs:
       with:
         name: python
         path: sdk/ahnlich-client-py/python.xml
+      
 
-
+  run-go-tests:
+      if: always()
+      runs-on: ubuntu-latest
+      steps:
+        - uses: actions/checkout@v4
+        - name: Set up Rust cache
+          uses: Swatinem/rust-cache@v2
+          with:
+            workspaces: ahnlich
+        
+        - name: Install Protoc
+          uses: arduino/setup-protoc@v3
+
+        - name: Setup Go Environment
+          uses: actions/setup-go@v5
+          with:
+            go-version: 1.22
+            cache-dependency-path: ./sdk/ahnlich-client-go/go.sum
+
+        - name: Install Dependencies
+          working-directory: ./sdk/ahnlich-client-go
+          run: make install-dependencies CI=1
+
+        - name: Format and Lint Check
+          working-directory: ./sdk/ahnlich-client-go
+          run: make pre-commit-check
+
+        - name: Run Tests
+          working-directory: ./sdk/ahnlich-client-go
+          run: |
+            make test CACHE=1 REPORT=1
+
+        - name: Upload Test Results
+          uses: actions/upload-artifact@v4
+          if: always()
+          with:
+            name: go
+            path: sdk/ahnlich-client-go/go.xml
 
   upload-test-results:
     if: always()
     runs-on: ubuntu-latest
-    needs: ["run-rust-tests", "run-python-tests"]
+    needs: ["run-rust-tests", "run-python-tests", "run-go-tests"]
 
     steps:
     - name: Download Rust Test Output
@@ -163,6 +205,12 @@ jobs:
       with:
         name: python
         path: ./python
+    
+    - name: Download Go Test Output
+      uses: actions/download-artifact@v4
+      with:
+        name: go
+        path: ./go
 
     - name: Publish Test Results
       uses: EnricoMi/publish-unit-test-result-action/macos@v2
@@ -171,3 +219,4 @@ jobs:
         files: |
           rust/*.xml
           python/*.xml
+          go/*.xml

.gitignore

@@ -29,3 +29,4 @@ data/
 profiling.json*
 
 **/.idea/
+.trunk/

protos/Makefile

@@ -0,0 +1,69 @@
+# ------------------------------------------------------------------------
+# Makefile – fixed: buf format gets one source argument
+# ------------------------------------------------------------------------
+
+.PHONY: format lint lock breaking generate all help
+
+# Default target ─ run the full pipeline
+all: generate
+
+## Format code
+format:
+	@echo "➜ Formatting protobuf files"
+	# Passing '.' keeps buf happy and recursively formats every *.proto
+	buf format -w .
+
+## Lint proto files
+lint:
+	@echo "➜ Linting protobuf files"
+	buf lint .
+
+## Lock dependencies
+lock:
+	@echo "➜ Updating buf module dependencies"
+	buf mod update
+
+## Check for breaking changes (runs from parent dir)
+breaking:
+	@echo "➜ Checking for breaking changes against main branch"
+	(cd .. && buf breaking --against '.git#branch=main' protos) 
+
+## Generate code with protoc plugins
+generate: lock format lint
+	@echo "➜ Generating code"
+	buf generate
+	@echo "✓ Done"
+
+## Show available targets with descriptions
+help:
+	@grep -E '^##' -n Makefile | sed 's/^.*## //' | column -t -s ':'
+
+# Ensure protoc and plugins are installed
+PROTOC := $(shell which protoc)
+PROTOC_GEN_GO := $(shell which protoc-gen-go)
+PROTOC_GEN_GO_GRPC := $(shell which protoc-gen-go-grpc)
+BUF := $(shell which buf)
+
+.PHONY: deps
+deps:
+ifndef PROTOC
+	@echo "❌ protoc is not installed. Please install it manually:"
+	@echo "  - macOS: brew install protobuf"
+	@echo "  - Ubuntu: sudo apt install -y protobuf-compiler"
+	@echo "  - Or download from: https://github.com/protocolbuffers/protobuf/releases"
+	@exit 1
+endif
+ifndef PROTOC_GEN_GO
+	@echo "Installing protoc-gen-go..."
+	go install google.golang.org/protobuf/cmd/protoc-gen-go@latest
+endif
+ifndef PROTOC_GEN_GO_GRPC
+	@echo "Installing protoc-gen-go-grpc..."
+	go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@latest
+endif
+	@echo "✅ All dependencies installed."
+ifndef BUF
+	@echo "Installing buf..."
+	go install github.com/bufbuild/buf/cmd/buf@latest
+endif
+	@echo "✅ buf installed."

protos/ai/execution_provider.proto

@@ -1,14 +1,13 @@
-
 syntax = "proto3";
 
 package ai.execution_provider;
 
-
+option go_package = "github.com/deven96/ahnlich/sdk/ahnlich-client-go/grpc/ai/execution_provider;execution_provider";
 
 // Enum for Execution Providers
 enum ExecutionProvider {
-    TENSOR_RT = 0;
-    CUDA = 1;
-    DIRECT_ML = 2;
-    CORE_ML = 3;
-}
+  TENSOR_RT = 0;
+  CUDA = 1;
+  DIRECT_ML = 2;
+  CORE_ML = 3;
+}

protos/ai/models.proto

@@ -1,21 +1,21 @@
-
 syntax = "proto3";
 
 package ai.models;
 
+option go_package = "github.com/deven96/ahnlich/sdk/ahnlich-client-go/grpc/ai/models;models";
 
 enum AIModel {
-    ALL_MINI_LM_L6_V2 = 0;
-    ALL_MINI_LM_L12_V2 = 1;
-    BGE_BASE_EN_V15 = 2;
-    BGE_LARGE_EN_V15 = 3;
-    RESNET50 = 4;
-    CLIP_VIT_B32_IMAGE = 5;
-    CLIP_VIT_B32_TEXT = 6;
+  ALL_MINI_LM_L6_V2 = 0;
+  ALL_MINI_LM_L12_V2 = 1;
+  BGE_BASE_EN_V15 = 2;
+  BGE_LARGE_EN_V15 = 3;
+  RESNET50 = 4;
+  CLIP_VIT_B32_IMAGE = 5;
+  CLIP_VIT_B32_TEXT = 6;
 }
 
 // Enum for AI Store Input Types
 enum AIStoreInputType {
-    RAW_STRING = 0;
-    IMAGE = 1;
+  RAW_STRING = 0;
+  IMAGE = 1;
 }

protos/ai/pipeline.proto

@@ -6,6 +6,8 @@ import "ai/query.proto";
 import "ai/server.proto";
 import "shared/info.proto";
 
+option go_package = "github.com/deven96/ahnlich/sdk/ahnlich-client-go/grpc/ai/pipeline;pipeline";
+
 message AIQuery {
   oneof query {
     ai.query.CreateStore create_store = 1;

protos/ai/preprocess.proto

@@ -1,9 +1,10 @@
-
 syntax = "proto3";
 
 package ai.preprocess;
 
+option go_package = "github.com/deven96/ahnlich/sdk/ahnlich-client-go/grpc/ai/preprocess;preprocess";
+
 enum PreprocessAction {
-    NoPreprocessing = 0;
-    ModelPreprocessing = 1;
-}
+  NoPreprocessing = 0;
+  ModelPreprocessing = 1;
+}