Crash on large input SIGSEGV: segmentation violation --13B model

[watcher60]

I'm running ubuntu 22.0.4 with 32 GB of RAM, with Auto-gpt pointed at LocalAI using model Manticore-13B.ggmlv3.q5_0.bin, I believe I'm up to date on LocalAI and associated libraries.

I have been stable previously but using this model it appears when the input is getting to a certain size I am getting a SIGSEGV: segmentation violation. Can anyone give any pointers on any adjustments I could make, or areas to look into as I'm not familiar with the error.

Error:
3:27PM DBG Parameter Config: &{OpenAIRequest:{Model:gpt-3.5-turbo File: Language: ResponseFormat: Size: Prompt: Instruction: Input: Stop: Messages:[] Stream:false Echo:false TopP:0.7 TopK:80 Temperature:0.9 Maxtokens:459 N:0 Batch:0 F16:false IgnoreEOS:false RepeatPenalty:0 Keep:0 MirostatETA:0 MirostatTAU:0 Mirostat:0 Seed:0 Mode:0 Step:0} Name: StopWords:[] Cutstrings:[] TrimSpace:[] ContextSize:512 F16:false Threads:4 Debug:true Roles:map[] Embeddings:false Backend: TemplateConfig:{Completion: Chat: Edit:} MirostatETA:0 MirostatTAU:0 Mirostat:0 NGPULayers:0 ImageGenerationAssets: PromptStrings:[] InputStrings:[] InputToken:[]}
3:27PM DBG Loading model 'gpt-3.5-turbo' greedly
3:27PM DBG Model 'gpt-3.5-turbo' already loaded
fatal error: unexpected signal during runtime execution
[signal SIGSEGV: segmentation violation code=0x80 addr=0x0 pc=0x47dc8f]

runtime stack:
runtime.throw({0xc3334d?, 0x1?})
/usr/lib/go-1.18/src/runtime/panic.go:992 +0x71
runtime.sigpanic()
/usr/lib/go-1.18/src/runtime/signal_unix.go:802 +0x3a9
runtime.casgstatus(0x3031206562206c6c, 0x4, 0x1)
/usr/lib/go-1.18/src/runtime/proc.go:976 +0x8f
runtime.(*gcControllerState).findRunnableGCWorker(0xc0000031e0?, 0xc00003ca00?)
/usr/lib/go-1.18/src/runtime/mgcpacer.go:891 +0x21e
runtime.schedule()
/usr/lib/go-1.18/src/runtime/proc.go:3166 +0x174
runtime.park_m(0xc0000031e0?)
/usr/lib/go-1.18/src/runtime/proc.go:3336 +0x14d
runtime.mcall()
/usr/lib/go-1.18/src/runtime/asm_amd64.s:425 +0x43

goroutine 1 [IO wait, 1190 minutes]:
internal/poll.runtime_pollWait(0x7fb13dfabd98, 0x72)
/usr/lib/go-1.18/src/runtime/netpoll.go:302 +0x89
internal/poll.(*pollDesc).wait(0xc00013ed00?, 0x4c?, 0x0)
/usr/lib/go-1.18/src/internal/poll/fd_poll_runtime.go:83 +0x32
internal/poll.(*pollDesc).waitRead(...)
/usr/lib/go-1.18/src/internal/poll/fd_poll_runtime.go:88
internal/poll.(*FD).Accept(0xc00013ed00)
/usr/lib/go-1.18/src/internal/poll/fd_unix.go:614 +0x22c
net.(*netFD).accept(0xc00013ed00)
/usr/lib/go-1.18/src/net/fd_unix.go:172 +0x35
net.(*TCPListener).accept(0xc0000a6978)
/usr/lib/go-1.18/src/net/tcpsock_posix.go:139 +0x28
net.(*TCPListener).Accept(0xc0000a6978)
/usr/lib/go-1.18/src/net/tcpsock.go:288 +0x3d
github.com/valyala/fasthttp.acceptConn(0xc0001b8000, {0xcf4650, 0xc0000a6978}, 0xc0000e56f8)
/home/watcher60/go/pkg/mod/github.com/valyala/[email protected]/server.go:1930 +0x62
github.com/valyala/fasthttp.(*Server).Serve(0xc0001b8000, {0xcf4650?, 0xc0000a6978})
/home/watcher60/go/pkg/mod/github.com/valyala/[email protected]/server.go:1823 +0x4d4
github.com/gofiber/fiber/v2.(*App).Listen(0xc0000f5200, {0xc15fba?, 0x7?})
/home/watcher60/go/pkg/mod/github.com/gofiber/fiber/[email protected]/listen.go:82 +0x110
main.main.func1(0xc0001029a0?)
/home/watcher60/autogpt4all/LocalAI/main.go:115 +0x568
github.com/urfave/cli/v2.(*Command).Run(0xc0001029a0, 0xc0000b0940, {0xc0000b0000, 0x4, 0x4})
/home/watcher60/go/pkg/mod/github.com/urfave/cli/[email protected]/command.go:274 +0xa42
github.com/urfave/cli/v2.(*App).RunContext(0xc0001fc000, {0xcf49e8?, 0xc0000b2000}, {0xc0000b0000, 0x4, 0x4})
/home/watcher60/go/pkg/mod/github.com/urfave/cli/[email protected]/app.go:332 +0x605
github.com/urfave/cli/v2.(*App).Run(...)
/home/watcher60/go/pkg/mod/github.com/urfave/cli/[email protected]/app.go:309
main.main()
/home/watcher60/autogpt4all/LocalAI/main.go:119 +0xce8

goroutine 19 [select, 1190 minutes]:
github.com/go-skynet/LocalAI/api.(*galleryApplier).start.func1()
/home/watcher60/autogpt4all/LocalAI/api/gallery.go:59 +0xe9
created by github.com/go-skynet/LocalAI/api.(*galleryApplier).start
/home/watcher60/autogpt4all/LocalAI/api/gallery.go:57 +0xaa

goroutine 20 [sleep]:
time.Sleep(0x2540be400)
/usr/lib/go-1.18/src/runtime/time.go:194 +0x12e
github.com/valyala/fasthttp.(*workerPool).Start.func2()
/home/watcher60/go/pkg/mod/github.com/valyala/[email protected]/workerpool.go:67 +0x56
created by github.com/valyala/fasthttp.(*workerPool).Start
/home/watcher60/go/pkg/mod/github.com/valyala/[email protected]/workerpool.go:59 +0xd9

goroutine 5 [syscall, 4 minutes]:
github.com/go-skynet/go-llama%2ecpp._Cfunc_llama_predict(0x7fb130002090, 0x17410e0, 0xc000268000, 0x1)
_cgo_gotypes.go:196 +0x4c
github.com/go-skynet/go-llama%2ecpp.(*LLama).Predict.func2(0xc000262000?, 0x1728?, {0xc000268000, 0x0?, 0x10?}, 0x7fb165c12108?)
/home/watcher60/autogpt4all/LocalAI/go-llama/llama.go:199 +0x94
github.com/go-skynet/go-llama%2ecpp.(*LLama).Predict(0xc00000e498, {0xc000262000, 0x1728}, {0xc00007e050, 0x7, 0x0?})
/home/watcher60/autogpt4all/LocalAI/go-llama/llama.go:199 +0x23f
github.com/go-skynet/LocalAI/api.ModelInference.func11()
/home/watcher60/autogpt4all/LocalAI/api/prediction.go:474 +0xc5
github.com/go-skynet/LocalAI/api.ModelInference.func12()
/home/watcher60/autogpt4all/LocalAI/api/prediction.go:499 +0x1a9
github.com/go-skynet/LocalAI/api.ComputeChoices({0xc000262000, 0x1728}, 0xc000104280, 0xc0001f2500, 0xc0000603f0?, 0xc59fc8, 0x4?)
/home/watcher60/autogpt4all/LocalAI/api/prediction.go:523 +0x275
github.com/go-skynet/LocalAI/api.chatEndpoint.func2(0xc000296000)
/home/watcher60/autogpt4all/LocalAI/api/openai.go:367 +0x710
github.com/gofiber/fiber/v2.(*App).next(0xc0000f5200, 0xc000296000)
/home/watcher60/go/pkg/mod/github.com/gofiber/fiber/[email protected]/router.go:144 +0x1b9
github.com/gofiber/fiber/v2.(*Ctx).Next(0xc0000d2f00?)
/home/watcher60/go/pkg/mod/github.com/gofiber/fiber/[email protected]/ctx.go:913 +0x53
github.com/gofiber/fiber/v2/middleware/recover.New.func1(0xb66e80?)
/home/watcher60/go/pkg/mod/github.com/gofiber/fiber/[email protected]/middleware/recover/recover.go:43 +0xcb
github.com/gofiber/fiber/v2.(*Ctx).Next(0xc0001a2b10?)
/home/watcher60/go/pkg/mod/github.com/gofiber/fiber/[email protected]/ctx.go:910 +0x43
github.com/gofiber/fiber/v2/middleware/logger.New.func3(0xc000296000)
/home/watcher60/go/pkg/mod/github.com/gofiber/fiber/[email protected]/middleware/logger/logger.go:121 +0x38b
github.com/gofiber/fiber/v2.(*App).next(0xc0000f5200, 0xc000296000)
/home/watcher60/go/pkg/mod/github.com/gofiber/fiber/[email protected]/router.go:144 +0x1b9
github.com/gofiber/fiber/v2.(*App).handler(0xc0000f5200, 0x4fca97?)
/home/watcher60/go/pkg/mod/github.com/gofiber/fiber/[email protected]/router.go:171 +0x87
github.com/valyala/fasthttp.(*Server).serveConn(0xc0001b8000, {0xcf6120?, 0xc000010018})
/home/watcher60/go/pkg/mod/github.com/valyala/[email protected]/server.go:2365 +0x120d
github.com/valyala/fasthttp.(*workerPool).workerFunc(0xc0000cdae0, 0xc00006a020)
/home/watcher60/go/pkg/mod/github.com/valyala/[email protected]/workerpool.go:224 +0xa9
github.com/valyala/fasthttp.(*workerPool).getCh.func1()
/home/watcher60/go/pkg/mod/github.com/valyala/[email protected]/workerpool.go:196 +0x38
created by github.com/valyala/fasthttp.(*workerPool).getCh
/home/watcher60/go/pkg/mod/github.com/valyala/[email protected]/workerpool.go:195 +0x1b0

goroutine 6 [sleep]:
time.Sleep(0x3b9aca00)
/usr/lib/go-1.18/src/runtime/time.go:194 +0x12e
github.com/valyala/fasthttp.updateServerDate.func1()
/home/watcher60/go/pkg/mod/github.com/valyala/[email protected]/header.go:2247 +0x1e
created by github.com/valyala/fasthttp.updateServerDate
/home/watcher60/go/pkg/mod/github.com/valyala/[email protected]/header.go:2245 +0x25

[watcher60]

Just to advise I switched models to use Wizard-Vicuna-30B-Uncensored.ggmlv3.q5_0.bin and the repeated crashes went away. Closing as presume a local issue (and I should have tried the apply model option to see if that resolved the issue)