Skip to content

Add concept of provider type in orchestrator #191

@gkumbhat

Description

@gkumbhat

Description

Currently we only support 2 providers which can be chosen to provide "text generation". However, as we are designing chat completion use-cases, we want to enable another LLM generation provider like vLLM. We would want the ability to run multiple types of providers together, but since we don't want to take responsibility of routing, we want to restrict configuration of only 1 provider per provider type..

This story is to add the concept of provider type in orchestrator that can be used in config.yaml in orchestrator.

Tasks

  1. Create ADR for provider_type parameter
  2. Implement provider_type concept in orchestrator
  3. Add provider type to existing text generation clients, i.e TGIS and NLP
  4. Add provider type support in config.yaml
  5. Add provider type as text_generation in config for TGIS / NLP.

Acceptance Criteria

  • ADR merged to repo
  • Add provider type for text-generation and NLP client
  • Add provider type in config.yaml for already supported providers.
  • Unit tests cover new/changed code

Metadata

Metadata

Assignees

Labels

documentationImprovements or additions to documentationenhancementNew feature or request

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions