Configuration file for model parameter defaults

[vazirim]

Is your feature request related to a problem? Please describe.

The model parameter defaults are currently set here.

Describe the solution you'd like

It would be great to have a configuration file with these default settings, where the user can specify patterns that match model ids, giving the possibility to set parameters for a single model or entire model families at once.

[esnible]

• As a user, it is not obvious which parameter defaults will be supplied by PDL
• As a user, it is not obvious what the parameter names that are allowed are for a model
• As a user, I'd like some of my parameters to be independent concerns from my program
• As a PDL implementor, I'd like a declarative way to specify parameter names and their values

I switched from the ollama/granite-code:8b model to the ibm-granite/granite-3.0-2b-instruct model under the custom_llm_provider "text-completion-openai". When I did that, the PDL default of max_tokens of 1024 was no longer transparently applied, causing me to receive the vLLM default of max_tokens=16.

vLLM didn't like the stop_sequence parameter. It logged a warning "The following fields were present in the request but ignored: {'stop_sequence'}". That warning appeared on the vLLM console log, not as part of PDL. PDL via LiteLLM may not know what parameters are being ignored.

It might make sense to allow the user to specify model parameters. Most PDL programs have only one model invocation. Perhaps PDL could allow a --model-parameters (or --params) that contained a YAML file with parameters that are always used if supplied.

It might make sense if there were built-in "variables" _all_model_defaults and _model_defaults that PDL programs were initialized with. The structure of _all_model_defaults I'm not sure about, some kind of dict-of-dicts matcher. But after the model keyword was encountered then _model_defaults would be populated with the defaults, allowing a PDL user to print to see what PDL is going to supply to LiteLLM. Similar to an implicit

defs:
   _model_defaults: {
         "decoding_method": "greedy", 
         "max_tokens": 1024,
         "min_new_tokens": 1,
         "repetition_penalty": 1.05,
         "top_k": 50,
         "top_p": 0.85, # Note that the PDL doc-string says this is 1.0
          "roles": ...
          "final_prompt_value": ...
   }

It might be good to allow the user to override the built-in model defaults, with a file called model-config.yaml or similar, which would include _all_model_defaults.

The format of _all_model_defaults should perhaps be a dict-of-dicts, where the keys are regular expressions shell-style wildcards such as "*watsonx*" or "replicate/*granite-3.0*". The idea is that when PDL encounters a model statement it would generate _model_defaults from the union of defaults in all matches in _all_model_defaults.

The user might also override this by creating file pdl-config.yaml or similar in the same directory as the .PDL files, or by using an optional --pdl-config CLI option.