-
Notifications
You must be signed in to change notification settings - Fork 36
Closed
Labels
documentationImprovements or additions to documentationImprovements or additions to documentationenhancementNew feature or requestNew feature or request
Description
Description
Currently we only support 2 providers which can be chosen to provide "text generation". However, as we are designing chat completion use-cases, we want to enable another LLM generation provider like vLLM. We would want the ability to run multiple types of providers together, but since we don't want to take responsibility of routing, we want to restrict configuration of only 1 provider per provider type..
This story is to add the concept of provider type in orchestrator that can be used in config.yaml
in orchestrator.
Tasks
- Create ADR for
provider_type
parameter - Implement
provider_type
concept in orchestrator - Add provider type to existing text generation clients, i.e TGIS and NLP
- Add provider type support in
config.yaml
- Add provider type as
text_generation
in config for TGIS / NLP.
Acceptance Criteria
- ADR merged to repo
- Add provider type for text-generation and NLP client
- Add provider type in config.yaml for already supported providers.
- Unit tests cover new/changed code
Metadata
Metadata
Assignees
Labels
documentationImprovements or additions to documentationImprovements or additions to documentationenhancementNew feature or requestNew feature or request