[BUG] LocalAI Function Calling Panic while using HolmesGPT

[robert-cronin]

Expected Behavior

Should work with HolmesGPT

Actual Behavior

AIKit's embedded LocalAI crashes with a panic when processing OpenAI function calling requests

1:54AM DBG guessDefaultsFromFile: NGPULayers set NGPULayers=99999999
1:54AM DBG guessDefaultsFromFile: template already set name=llama-3.2-1b-instruct
1:54AM DBG Chat endpoint configuration read: &{PredictionOptions:{BasicModelRequest:{Model:Llama-3.2-1B-Instruct.Q4_K_M.gguf} Language: Translate:false N:0 TopP:0xc000476c20 TopK:0xc000476c28 Temperature:0xc001981220 Maxtokens:0xc000476c60 Echo:false Batch:0 IgnoreEOS:false RepeatPenalty:0 RepeatLastN:0 Keep:0 FrequencyPenalty:0 PresencePenalty:0 TFZ:0xc000476c58 TypicalP:0xc000476c50 Seed:0xc000476c70 NegativePrompt: RopeFreqBase:0 RopeFreqScale:0 NegativePromptScale:0 ClipSkip:0 Tokenizer:} Name:llama-3.2-1b-instruct F16:0xc000476c08 Threads:0xc000476c10 Debug:0xc0019c0028 Roles:map[] Embeddings:0xc000476c69 Backend:llama TemplateConfig:{Chat:chat ChatMessage:chatMsg Completion:completion Edit: Functions:function UseTokenizerTemplate:false JoinChatMessagesByCharacter:<nil> Multimodal: JinjaTemplate:false ReplyPrefix:} KnownUsecaseStrings:[FLAG_COMPLETION FLAG_ANY FLAG_CHAT] KnownUsecases:<nil> Pipeline:{TTS: LLM: Transcription: VAD:} PromptStrings:[] InputStrings:[] InputToken:[] functionCallString: functionCallNameString: ResponseFormat: ResponseFormatMap:map[] FunctionsConfig:{DisableNoAction:true GrammarConfig:{ParallelCalls:false DisableParallelNewLines:false MixedMode:false NoMixedFreeString:false NoGrammar:true Prefix: ExpectStringsAfterJSON:false PropOrder: SchemaType: GrammarTriggers:[]} NoActionFunctionName: NoActionDescriptionName: ResponseRegex:[<function=(?P<name>\w+)>(?P<arguments>.*)</function>] JSONRegexMatch:[] ArgumentRegex:[] ArgumentRegexKey: ArgumentRegexValue: ReplaceFunctionResults:[] ReplaceLLMResult:[] CaptureLLMResult:[] FunctionNameKey: FunctionArgumentsKey:} FeatureFlag:map[] LLMConfig:{SystemPrompt: TensorSplit: MainGPU: RMSNormEps:0 NGQA:0 PromptCachePath: PromptCacheAll:false PromptCacheRO:false MirostatETA:0xc000476c48 MirostatTAU:0xc000476c40 Mirostat:0xc000476c38 NGPULayers:0xc001eaf718 MMap:0xc000476c68 MMlock:0xc000476c69 LowVRAM:0xc000476c69 Reranking:0xc000476c69 Grammar: StopWords:[<|im_end|> <dummy32000> <|eot_id|> <|end_of_text|>] Cutstrings:[] ExtractRegex:[] TrimSpace:[] TrimSuffix:[] ContextSize:0xc000476bf8 NUMA:false LoraAdapter: LoraBase: LoraAdapters:[] LoraScales:[] LoraScale:0 NoMulMatQ:false DraftModel: NDraft:0 Quantization: LoadFormat: GPUMemoryUtilization:0 TrustRemoteCode:false EnforceEager:false SwapSpace:0 MaxModelLen:0 TensorParallelSize:0 DisableLogStatus:false DType: LimitMMPerPrompt:{LimitImagePerPrompt:0 LimitVideoPerPrompt:0 LimitAudioPerPrompt:0} MMProj: FlashAttention:false NoKVOffloading:false CacheTypeK: CacheTypeV: RopeScaling: ModelType: YarnExtFactor:0 YarnAttnFactor:0 YarnBetaFast:0 YarnBetaSlow:0 CFGScale:0} Diffusers:{CUDA:false PipelineType: SchedulerType: EnableParameters: IMG2IMG:false ClipSkip:0 ClipModel: ClipSubFolder: ControlNet:} Step:0 GRPC:{Attempts:0 AttemptsSleepTime:0} TTSConfig:{Voice: AudioPath:} CUDA:false DownloadFiles:[] Description: Usage: Options:[] Overrides:[]}
1:54AM DBG Response needs to process functions
panic: interface conversion: interface {} is []interface {}, not string

goroutine 68 [running]:
github.com/mudler/LocalAI/pkg/functions/grammars.(*JSONSchemaConverter).visit(0xc0014cf4b0, 0xc0009b5950, {0xc0015220c0, 0x1a}, 0xc0005f4420)
        /home/sozercan/go/src/github.com/mudlar/LocalAI/pkg/functions/grammars/json_schema.go:65 +0x14a6
github.com/mudler/LocalAI/pkg/functions/grammars.(*JSONSchemaConverter).visit(0xc0014cf4b0, 0xc0009b58c0, {0xc00151d050, 0x11}, 0xc0005f4420)
        /home/sozercan/go/src/github.com/mudlar/LocalAI/pkg/functions/grammars/json_schema.go:150 +0xc45
github.com/mudler/LocalAI/pkg/functions/grammars.(*JSONSchemaConverter).visit(0xc0014cf4b0, 0xc0009b5860, {0xc001521430, 0x7}, 0xc0005f4420
....

Environment

• AIKit Version: Latest (using LocalAI commit sha-1a0d06f)
• Kubernetes: Azure Kubernetes Service (AKS)
• Model: Llama 3.2 1B (CPU inference)
• Client: HolmesGPT with LiteLLM
• Request Format: Standard OpenAI /v1/chat/completions with tools parameter

Steps To Reproduce

I am still working on a viable fix, but though I might add this issue. Repro repo can be found here: robert-cronin/aikit-holmesgpt-bug-repro

Are you willing to submit PRs to contribute to this bug fix?

• Yes, I am willing to implement it.