diff --git a/README-sycl.md b/README-sycl.md deleted file mode 100644 index dd5bf9dea..000000000 --- a/README-sycl.md +++ /dev/null @@ -1,494 +0,0 @@ -# llama.cpp for SYCL - -- [Background](#background) -- [OS](#os) -- [Intel GPU](#intel-gpu) -- [Docker](#docker) -- [Linux](#linux) -- [Windows](#windows) -- [Environment Variable](#environment-variable) -- [Known Issue](#known-issue) -- [Q&A](#q&a) -- [Todo](#todo) - -## Background - -SYCL is a higher-level programming model to improve programming productivity on various hardware accelerators—such as CPUs, GPUs, and FPGAs. It is a single-source embedded domain-specific language based on pure C++17. - -oneAPI is a specification that is open and standards-based, supporting multiple architecture types including but not limited to GPU, CPU, and FPGA. The spec has both direct programming and API-based programming paradigms. - -Intel uses the SYCL as direct programming language to support CPU, GPUs and FPGAs. - -To avoid to re-invent the wheel, this code refer other code paths in llama.cpp (like OpenBLAS, cuBLAS, CLBlast). We use a open-source tool [SYCLomatic](https://github.com/oneapi-src/SYCLomatic) (Commercial release [Intel® DPC++ Compatibility Tool](https://www.intel.com/content/www/us/en/developer/tools/oneapi/dpc-compatibility-tool.html)) migrate to SYCL. - -The llama.cpp for SYCL is used to support Intel GPUs. - -For Intel CPU, recommend to use llama.cpp for X86 (Intel MKL building). - -## OS - -|OS|Status|Verified| -|-|-|-| -|Linux|Support|Ubuntu 22.04, Fedora Silverblue 39| -|Windows|Support|Windows 11| - - -## Intel GPU - -### Verified - -|Intel GPU| Status | Verified Model| -|-|-|-| -|Intel Data Center Max Series| Support| Max 1550| -|Intel Data Center Flex Series| Support| Flex 170| -|Intel Arc Series| Support| Arc 770, 730M| -|Intel built-in Arc GPU| Support| built-in Arc GPU in Meteor Lake| -|Intel iGPU| Support| iGPU in i5-1250P, i7-1260P, i7-1165G7| - -Note: If the EUs (Execution Unit) in iGPU is less than 80, the inference speed will be too slow to use. - -### Memory - -The memory is a limitation to run LLM on GPUs. - -When run llama.cpp, there is print log to show the applied memory on GPU. You could know how much memory to be used in your case. Like `llm_load_tensors: buffer size = 3577.56 MiB`. - -For iGPU, please make sure the shared memory from host memory is enough. For llama-2-7b.Q4_0, recommend the host memory is 8GB+. - -For dGPU, please make sure the device memory is enough. For llama-2-7b.Q4_0, recommend the device memory is 4GB+. - -## Docker - -Note: -- Only docker on Linux is tested. Docker on WSL may not work. -- You may need to install Intel GPU driver on the host machine (See the [Linux](#linux) section to know how to do that) - -### Build the image - -You can choose between **F16** and **F32** build. F16 is faster for long-prompt inference. - - -```sh -# For F16: -#docker build -t llama-cpp-sycl --build-arg="LLAMA_SYCL_F16=ON" -f .devops/main-intel.Dockerfile . - -# Or, for F32: -docker build -t llama-cpp-sycl -f .devops/main-intel.Dockerfile . - -# Note: you can also use the ".devops/main-server.Dockerfile", which compiles the "server" example -``` - -### Run - -```sh -# Firstly, find all the DRI cards: -ls -la /dev/dri -# Then, pick the card that you want to use. - -# For example with "/dev/dri/card1" -docker run -it --rm -v "$(pwd):/app:Z" --device /dev/dri/renderD128:/dev/dri/renderD128 --device /dev/dri/card1:/dev/dri/card1 llama-cpp-sycl -m "/app/models/YOUR_MODEL_FILE" -p "Building a website can be done in 10 simple steps:" -n 400 -e -ngl 33 -``` - -## Linux - -### Setup Environment - -1. Install Intel GPU driver. - -a. Please install Intel GPU driver by official guide: [Install GPU Drivers](https://dgpu-docs.intel.com/driver/installation.html). - -Note: for iGPU, please install the client GPU driver. - -b. Add user to group: video, render. - -```sh -sudo usermod -aG render username -sudo usermod -aG video username -``` - -Note: re-login to enable it. - -c. Check - -```sh -sudo apt install clinfo -sudo clinfo -l -``` - -Output (example): - -``` -Platform #0: Intel(R) OpenCL Graphics - `-- Device #0: Intel(R) Arc(TM) A770 Graphics - - -Platform #0: Intel(R) OpenCL HD Graphics - `-- Device #0: Intel(R) Iris(R) Xe Graphics [0x9a49] -``` - -2. Install Intel® oneAPI Base toolkit. - -a. Please follow the procedure in [Get the Intel® oneAPI Base Toolkit ](https://www.intel.com/content/www/us/en/developer/tools/oneapi/base-toolkit.html). - -Recommend to install to default folder: **/opt/intel/oneapi**. - -Following guide use the default folder as example. If you use other folder, please modify the following guide info with your folder. - -b. Check - -```sh -source /opt/intel/oneapi/setvars.sh - -sycl-ls -``` - -There should be one or more level-zero devices. Please confirm that at least one GPU is present, like **[ext_oneapi_level_zero:gpu:0]**. - -Output (example): -``` -[opencl:acc:0] Intel(R) FPGA Emulation Platform for OpenCL(TM), Intel(R) FPGA Emulation Device OpenCL 1.2 [2023.16.10.0.17_160000] -[opencl:cpu:1] Intel(R) OpenCL, 13th Gen Intel(R) Core(TM) i7-13700K OpenCL 3.0 (Build 0) [2023.16.10.0.17_160000] -[opencl:gpu:2] Intel(R) OpenCL Graphics, Intel(R) Arc(TM) A770 Graphics OpenCL 3.0 NEO [23.30.26918.50] -[ext_oneapi_level_zero:gpu:0] Intel(R) Level-Zero, Intel(R) Arc(TM) A770 Graphics 1.3 [1.3.26918] - -``` - -2. Build locally: - -Note: -- You can choose between **F16** and **F32** build. F16 is faster for long-prompt inference. -- By default, it will build for all binary files. It will take more time. To reduce the time, we recommend to build for **example/main** only. - -```sh -mkdir -p build -cd build -source /opt/intel/oneapi/setvars.sh - -# For FP16: -#cmake .. -DLLAMA_SYCL=ON -DCMAKE_C_COMPILER=icx -DCMAKE_CXX_COMPILER=icpx -DLLAMA_SYCL_F16=ON - -# Or, for FP32: -cmake .. -DLLAMA_SYCL=ON -DCMAKE_C_COMPILER=icx -DCMAKE_CXX_COMPILER=icpx - -# Build example/main only -#cmake --build . --config Release --target main - -# Or, build all binary -cmake --build . --config Release -v - -cd .. -``` - -or - -```sh -./examples/sycl/build.sh -``` - -### Run - -1. Put model file to folder **models** - -You could download [llama-2-7b.Q4_0.gguf](https://huggingface.co/TheBloke/Llama-2-7B-GGUF/blob/main/llama-2-7b.Q4_0.gguf) as example. - -2. Enable oneAPI running environment - -``` -source /opt/intel/oneapi/setvars.sh -``` - -3. List device ID - -Run without parameter: - -```sh -./build/bin/ls-sycl-device - -# or running the "main" executable and look at the output log: - -./build/bin/main -``` - -Check the ID in startup log, like: - -``` -found 4 SYCL devices: - Device 0: Intel(R) Arc(TM) A770 Graphics, compute capability 1.3, - max compute_units 512, max work group size 1024, max sub group size 32, global mem size 16225243136 - Device 1: Intel(R) FPGA Emulation Device, compute capability 1.2, - max compute_units 24, max work group size 67108864, max sub group size 64, global mem size 67065057280 - Device 2: 13th Gen Intel(R) Core(TM) i7-13700K, compute capability 3.0, - max compute_units 24, max work group size 8192, max sub group size 64, global mem size 67065057280 - Device 3: Intel(R) Arc(TM) A770 Graphics, compute capability 3.0, - max compute_units 512, max work group size 1024, max sub group size 32, global mem size 16225243136 - -``` - -|Attribute|Note| -|-|-| -|compute capability 1.3|Level-zero running time, recommended | -|compute capability 3.0|OpenCL running time, slower than level-zero in most cases| - -4. Set device ID and execute llama.cpp - -Set device ID = 0 by **GGML_SYCL_DEVICE=0** - -```sh -GGML_SYCL_DEVICE=0 ./build/bin/main -m models/llama-2-7b.Q4_0.gguf -p "Building a website can be done in 10 simple steps:" -n 400 -e -ngl 33 -``` -or run by script: - -```sh -./examples/sycl/run_llama2.sh -``` - -Note: - -- By default, mmap is used to read model file. In some cases, it leads to the hang issue. Recommend to use parameter **--no-mmap** to disable mmap() to skip this issue. - - -5. Check the device ID in output - -Like: -``` -Using device **0** (Intel(R) Arc(TM) A770 Graphics) as main device -``` - -## Windows - -### Setup Environment - -1. Install Intel GPU driver. - -Please install Intel GPU driver by official guide: [Install GPU Drivers](https://www.intel.com/content/www/us/en/products/docs/discrete-gpus/arc/software/drivers.html). - -Note: **The driver is mandatory for compute function**. - -2. Install Visual Studio. - -Please install [Visual Studio](https://visualstudio.microsoft.com/) which impact oneAPI environment enabling in Windows. - -3. Install Intel® oneAPI Base toolkit. - -a. Please follow the procedure in [Get the Intel® oneAPI Base Toolkit ](https://www.intel.com/content/www/us/en/developer/tools/oneapi/base-toolkit.html). - -Recommend to install to default folder: **C:\Program Files (x86)\Intel\oneAPI**. - -Following guide uses the default folder as example. If you use other folder, please modify the following guide info with your folder. - -b. Enable oneAPI running environment: - -- In Search, input 'oneAPI'. - -Search & open "Intel oneAPI command prompt for Intel 64 for Visual Studio 2022" - -- In Run: - -In CMD: -``` -"C:\Program Files (x86)\Intel\oneAPI\setvars.bat" intel64 -``` - -c. Check GPU - -In oneAPI command line: - -``` -sycl-ls -``` - -There should be one or more level-zero devices. Please confirm that at least one GPU is present, like **[ext_oneapi_level_zero:gpu:0]**. - -Output (example): -``` -[opencl:acc:0] Intel(R) FPGA Emulation Platform for OpenCL(TM), Intel(R) FPGA Emulation Device OpenCL 1.2 [2023.16.10.0.17_160000] -[opencl:cpu:1] Intel(R) OpenCL, 11th Gen Intel(R) Core(TM) i7-1185G7 @ 3.00GHz OpenCL 3.0 (Build 0) [2023.16.10.0.17_160000] -[opencl:gpu:2] Intel(R) OpenCL Graphics, Intel(R) Iris(R) Xe Graphics OpenCL 3.0 NEO [31.0.101.5186] -[ext_oneapi_level_zero:gpu:0] Intel(R) Level-Zero, Intel(R) Iris(R) Xe Graphics 1.3 [1.3.28044] -``` - -4. Install cmake & make - -a. Download & install cmake for Windows: https://cmake.org/download/ - -b. Download & install mingw-w64 make for Windows provided by w64devkit - -- Download the latest fortran version of [w64devkit](https://github.com/skeeto/w64devkit/releases). - -- Extract `w64devkit` on your pc. - -- Add the **bin** folder path in the Windows system PATH environment, like `C:\xxx\w64devkit\bin\`. - -### Build locally: - -In oneAPI command line window: - -``` -mkdir -p build -cd build -@call "C:\Program Files (x86)\Intel\oneAPI\setvars.bat" intel64 --force - -:: for FP16 -:: faster for long-prompt inference -:: cmake -G "MinGW Makefiles" .. -DLLAMA_SYCL=ON -DCMAKE_C_COMPILER=icx -DCMAKE_CXX_COMPILER=icx -DCMAKE_BUILD_TYPE=Release -DLLAMA_SYCL_F16=ON - -:: for FP32 -cmake -G "MinGW Makefiles" .. -DLLAMA_SYCL=ON -DCMAKE_C_COMPILER=icx -DCMAKE_CXX_COMPILER=icx -DCMAKE_BUILD_TYPE=Release - - -:: build example/main only -:: make main - -:: build all binary -make -j -cd .. -``` - -or - -``` -.\examples\sycl\win-build-sycl.bat -``` - -Note: - -- By default, it will build for all binary files. It will take more time. To reduce the time, we recommend to build for **example/main** only. - -### Run - -1. Put model file to folder **models** - -You could download [llama-2-7b.Q4_0.gguf](https://huggingface.co/TheBloke/Llama-2-7B-GGUF/blob/main/llama-2-7b.Q4_0.gguf) as example. - -2. Enable oneAPI running environment - -- In Search, input 'oneAPI'. - -Search & open "Intel oneAPI command prompt for Intel 64 for Visual Studio 2022" - -- In Run: - -In CMD: -``` -"C:\Program Files (x86)\Intel\oneAPI\setvars.bat" intel64 -``` - -3. List device ID - -Run without parameter: - -``` -build\bin\ls-sycl-device.exe - -or - -build\bin\main.exe -``` - -Check the ID in startup log, like: - -``` -found 4 SYCL devices: - Device 0: Intel(R) Arc(TM) A770 Graphics, compute capability 1.3, - max compute_units 512, max work group size 1024, max sub group size 32, global mem size 16225243136 - Device 1: Intel(R) FPGA Emulation Device, compute capability 1.2, - max compute_units 24, max work group size 67108864, max sub group size 64, global mem size 67065057280 - Device 2: 13th Gen Intel(R) Core(TM) i7-13700K, compute capability 3.0, - max compute_units 24, max work group size 8192, max sub group size 64, global mem size 67065057280 - Device 3: Intel(R) Arc(TM) A770 Graphics, compute capability 3.0, - max compute_units 512, max work group size 1024, max sub group size 32, global mem size 16225243136 - -``` - -|Attribute|Note| -|-|-| -|compute capability 1.3|Level-zero running time, recommended | -|compute capability 3.0|OpenCL running time, slower than level-zero in most cases| - -4. Set device ID and execute llama.cpp - -Set device ID = 0 by **set GGML_SYCL_DEVICE=0** - -``` -set GGML_SYCL_DEVICE=0 -build\bin\main.exe -m models\llama-2-7b.Q4_0.gguf -p "Building a website can be done in 10 simple steps:\nStep 1:" -n 400 -e -ngl 33 -s 0 -``` -or run by script: - -``` -.\examples\sycl\win-run-llama2.bat -``` - -Note: - -- By default, mmap is used to read model file. In some cases, it leads to the hang issue. Recommend to use parameter **--no-mmap** to disable mmap() to skip this issue. - - -5. Check the device ID in output - -Like: -``` -Using device **0** (Intel(R) Arc(TM) A770 Graphics) as main device -``` - -## Environment Variable - -#### Build - -|Name|Value|Function| -|-|-|-| -|LLAMA_SYCL|ON (mandatory)|Enable build with SYCL code path.
For FP32/FP16, LLAMA_SYCL=ON is mandatory.| -|LLAMA_SYCL_F16|ON (optional)|Enable FP16 build with SYCL code path. Faster for long-prompt inference.
For FP32, not set it.| -|CMAKE_C_COMPILER|icx|Use icx compiler for SYCL code path| -|CMAKE_CXX_COMPILER|icpx (Linux), icx (Windows)|use icpx/icx for SYCL code path| - -#### Running - - -|Name|Value|Function| -|-|-|-| -|GGML_SYCL_DEVICE|0 (default) or 1|Set the device id used. Check the device ids by default running output| -|GGML_SYCL_DEBUG|0 (default) or 1|Enable log function by macro: GGML_SYCL_DEBUG| - -## Known Issue - -- Hang during startup - - llama.cpp use mmap as default way to read model file and copy to GPU. In some system, memcpy will be abnormal and block. - - Solution: add **--no-mmap** or **--mmap 0**. - -## Q&A - -- Error: `error while loading shared libraries: libsycl.so.7: cannot open shared object file: No such file or directory`. - - Miss to enable oneAPI running environment. - - Install oneAPI base toolkit and enable it by: `source /opt/intel/oneapi/setvars.sh`. - -- In Windows, no result, not error. - - Miss to enable oneAPI running environment. - -- Meet compile error. - - Remove folder **build** and try again. - -- I can **not** see **[ext_oneapi_level_zero:gpu:0]** afer install GPU driver in Linux. - - Please run **sudo sycl-ls**. - - If you see it in result, please add video/render group to your ID: - - ``` - sudo usermod -aG render username - sudo usermod -aG video username - ``` - - Then **relogin**. - - If you do not see it, please check the installation GPU steps again. - -## Todo - -- Support multiple cards. diff --git a/README.md b/README.md index fe82dd749..390737616 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,6 @@ # koboldcpp -KoboldCpp is an easy-to-use AI text-generation software for GGML and GGUF models. It's a single self contained distributable from Concedo, that builds off llama.cpp, and adds a versatile Kobold API endpoint, additional format support, backward compatibility, as well as a fancy UI with persistent stories, editing tools, save formats, memory, world info, author's note, characters, scenarios and everything Kobold and Kobold Lite have to offer. +KoboldCpp is an easy-to-use AI text-generation software for GGML and GGUF models. It's a single self contained distributable from Concedo, that builds off llama.cpp, and adds a versatile Kobold API endpoint, additional format support, Stable Diffusion image generation, backward compatibility, as well as a fancy UI with persistent stories, editing tools, save formats, memory, world info, author's note, characters, scenarios and everything Kobold and Kobold Lite have to offer. ![Preview](media/preview.png) ![Preview](media/preview2.png) @@ -132,6 +132,7 @@ You can then run koboldcpp anywhere from the terminal by running `koboldcpp` to - Since v1.33, you can set the context size to be above what the model supports officially. It does increases perplexity but should still work well below 4096 even on untuned models. (For GPT-NeoX, GPT-J, and LLAMA models) Customize this with `--ropeconfig`. - Since v1.42, supports GGUF models for LLAMA and Falcon - Since v1.55, lcuda paths on Linux are hardcoded and may require manual changes to the makefile if you do not use koboldcpp.sh for the compilation. +- Since v1.60, provides native image generation with StableDiffusion.cpp, you can load any SD1.5 or SDXL .safetensors model and it will provide an A1111 compatible API to use. - **I plan to keep backwards compatibility with ALL past llama.cpp AND alpaca.cpp models**. But you are also encouraged to reconvert/update your models if possible for best results. ## License @@ -144,6 +145,7 @@ You can then run koboldcpp anywhere from the terminal by running `koboldcpp` to - I have heard of someone claiming a false AV positive report. The exe is a simple pyinstaller bundle that includes the necessary python scripts and dlls to run. If this still concerns you, you might wish to rebuild everything from source code using the makefile, and you can rebuild the exe yourself with pyinstaller by using `make_pyinstaller.bat` - API documentation available at `/api` and https://lite.koboldai.net/koboldcpp_api - Supported GGML models (Includes backward compatibility for older versions/legacy GGML models, though some newer features might be unavailable): + - All up-to-date GGUF models are supported (Mistral/Mixtral/QWEN/Gemma and more) - LLAMA and LLAMA2 (LLaMA / Alpaca / GPT4All / Vicuna / Koala / Pygmalion 7B / Metharme 7B / WizardLM and many more) - GPT-2 / Cerebras - GPT-J diff --git a/klite.embd b/klite.embd index 5c594d201..8c3bfd996 100644 --- a/klite.embd +++ b/klite.embd @@ -3379,6 +3379,8 @@ Current version: 119 const xtts_gen_endpoint = "/tts_to_audio/"; const xtts_voices_endpoint = "/speakers_list"; + const alltalk_gen_endpoint = "/api/tts-generate"; + const alltalk_voices_endpoint = "/api/voices"; //support for quick news updates const horde_news_endpoint = "https://hordenews.concedo.workers.dev" @@ -3392,8 +3394,10 @@ Current version: 119 const default_a1111_base = "http://localhost:7860"; const default_xtts_base = " http://localhost:8020"; + const default_alltalk_base = "http://localhost:7851"; const XTTS_ID = 1000; + const ALLTALK_ID = 1001; const HD_RES_PX = 400; const NO_HD_RES_PX = 256; @@ -3483,6 +3487,7 @@ Current version: 119 saved_oai_role: 0, //0=user,1=assistant,2=system saved_a1111_url: default_a1111_base, saved_xtts_url: default_xtts_base, + saved_alltalk_url: default_alltalk_base, prev_custom_endpoint_type: 0, //show a reconnect box to custom endpoint if needed. 0 is horde, otherwise its dropdown value+1 generate_images_mode: (localflag?0:1), //0=off, 1=horde, 2=a1111, 3=dalle @@ -4028,9 +4033,9 @@ Current version: 119 { fetch_image_models(); } - if(localsettings.speech_synth==XTTS_ID) + if(localsettings.speech_synth==XTTS_ID || localsettings.speech_synth==ALLTALK_ID) { - fetch_xtts_voices(true); + fetch_xtts_voices(true,localsettings.speech_synth==XTTS_ID); } if(localsettings.generate_images_mode==2) { @@ -7921,6 +7926,7 @@ Current version: 119 console.log("No speech synth available"); } ttshtml += ""; + ttshtml += ""; document.getElementById("ttsselect").innerHTML = ttshtml; document.getElementById("ttsselect").value = localsettings.speech_synth; toggle_tts_mode(); @@ -8956,15 +8962,20 @@ Current version: 119 } var xtts_is_connected = false; - function fetch_xtts_voices(silent) + function fetch_xtts_voices(silent, is_xtts) { if(!xtts_is_connected) { - fetch(localsettings.saved_xtts_url + xtts_voices_endpoint) + let endpt = (is_xtts?(localsettings.saved_xtts_url + xtts_voices_endpoint):(localsettings.saved_alltalk_url + alltalk_voices_endpoint)); + fetch(endpt) .then(x => x.json()) .then(data => { console.log(data); //repopulate our voices list + if (data && !data.length && data.voices) { + //alltalk mode + data = data.voices; + } let dropdown = document.getElementById("xtts_voices"); let selectionhtml = ``; for (var i = 0; i < data.length; ++i) { @@ -8979,7 +8990,8 @@ Current version: 119 xtts_is_connected = false; if(!silent) { - msgbox("XTTS Connect Error: " + error+"\nCheck XTTS API Server endpoint URL.\n"); + let epname = (is_xtts?"XTTS":"AllTalk"); + msgbox(epname + " Connect Error: " + error+"\nCheck "+epname+" API Server endpoint URL.\n"); } }); } @@ -8999,17 +9011,19 @@ Current version: 119 function toggle_tts_mode() { - if(document.getElementById("ttsselect").value==XTTS_ID) + if(document.getElementById("ttsselect").value==XTTS_ID || document.getElementById("ttsselect").value==ALLTALK_ID) { document.getElementById("xtts_container").classList.remove("hidden"); - fetch_xtts_voices(true); + fetch_xtts_voices(true, document.getElementById("ttsselect").value==XTTS_ID); }else{ document.getElementById("xtts_container").classList.add("hidden"); } } function set_xtts_url() { - inputBox("Enter XTTS API Server URL.","XTTS API Server URL",localsettings.saved_xtts_url,"Input XTTS API Server URL", ()=>{ + let is_xtts = (document.getElementById("ttsselect").value==XTTS_ID); + let epname = (is_xtts?"XTTS":"AllTalk"); + inputBox("Enter "+epname+" API Server URL.",epname+" API Server URL",(is_xtts?localsettings.saved_xtts_url:localsettings.saved_alltalk_url),"Input "+epname+" API Server URL", ()=>{ let userinput = getInputBoxValue(); userinput = userinput.trim(); if(userinput!="" && userinput.slice(-1)=="/") @@ -9018,12 +9032,20 @@ Current version: 119 } if(userinput=="") { - userinput = default_xtts_base; + userinput = (is_xtts?default_xtts_base:default_alltalk_base); } if (userinput != null && userinput!="") { - localsettings.saved_xtts_url = userinput.trim(); + if(is_xtts) + { + localsettings.saved_xtts_url = userinput.trim(); + } + else + { + localsettings.saved_alltalk_url = userinput.trim(); + } + xtts_is_connected = false; - fetch_xtts_voices(false); + fetch_xtts_voices(false, is_xtts); } },false); } @@ -9060,36 +9082,74 @@ Current version: 119 } } - if(ssval==XTTS_ID) //xtts api server + if(ssval==XTTS_ID || ssval==ALLTALK_ID) //xtts api server { if(xtts_is_connected) { + let is_xtts = (ssval==XTTS_ID); const audioContext = new (window.AudioContext || window.webkitAudioContext)(); - let xtts_payload = { - "text": text, - "speaker_wav": document.getElementById("xtts_voices").value, - "language": "EN" - }; - fetch(localsettings.saved_xtts_url + xtts_gen_endpoint, { - method: 'POST', - headers: { - 'Content-Type': 'application/json' - }, - body: JSON.stringify(xtts_payload), - }) - .then(response => response.arrayBuffer()) - .then(data => { - return audioContext.decodeAudioData(data); - }) - .then(decodedData => { - const playSound = audioContext.createBufferSource(); - playSound.buffer = decodedData; - playSound.connect(audioContext.destination); - playSound.start(audioContext.currentTime); - }).catch((error) => { - console.log("XTTS Speak Error: " + error); - }); + if(is_xtts) + { + let xtts_payload = { + "text": text, + "speaker_wav": document.getElementById("xtts_voices").value, + "language": "EN" + }; + fetch(localsettings.saved_xtts_url + xtts_gen_endpoint, { + method: 'POST', + headers: { + 'Content-Type': 'application/json' + }, + body: JSON.stringify(xtts_payload), + }) + .then(response => response.arrayBuffer()) + .then(data => { + return audioContext.decodeAudioData(data); + }) + .then(decodedData => { + const playSound = audioContext.createBufferSource(); + playSound.buffer = decodedData; + playSound.connect(audioContext.destination); + playSound.start(audioContext.currentTime); + }).catch((error) => { + console.log("XTTS Speak Error: " + error); + }); + } + else + { + //alltalk + const formData = new FormData(); + formData.append("text_input", text); // max 2000 chars + formData.append("text_filtering", "none"); // (none|standard|html) + formData.append("character_voice_gen", document.getElementById("xtts_voices").value); + formData.append("narrator_enabled", false); + formData.append("narrator_voice_gen", document.getElementById("xtts_voices").value); + formData.append("text_not_inside", "character"); // character or narrator, determines which to use + formData.append("language", "en"); + formData.append("output_file_name", "audiofile"); // NOTE: file name only, with no extension and no dashes! + formData.append("output_file_timestamp", true); + formData.append("autoplay", false); //to play in browser + formData.append("autoplay_volume", 1.0); // (0.1..2.0) + formData.append("streaming", true); // unknown why + + fetch(localsettings.saved_alltalk_url + alltalk_gen_endpoint, { + method: 'POST', + body: formData, // send payload as FormData + }) + .then(response => response.arrayBuffer()) + .then(data => { + return audioContext.decodeAudioData(data); + }) + .then(decodedData => { + const playSound = audioContext.createBufferSource(); + playSound.buffer = decodedData; + playSound.connect(audioContext.destination); + playSound.start(audioContext.currentTime); + }).catch((error) => { + console.log("AllTalk Speak Error: " + error); + }); + } } } else @@ -13475,12 +13535,11 @@ Current version: 119
Model Choice:
diff --git a/koboldcpp.py b/koboldcpp.py index 63d4c6128..9940dcd14 100644 --- a/koboldcpp.py +++ b/koboldcpp.py @@ -513,6 +513,8 @@ def sd_generate(genparams): sample_method = genparams.get("sampler_name", "euler a") #clean vars + width = width - (width%64) + height = height - (height%64) cfg_scale = (1 if cfg_scale < 1 else (25 if cfg_scale > 25 else cfg_scale)) sample_steps = (1 if sample_steps < 1 else (80 if sample_steps > 80 else sample_steps)) width = (128 if width < 128 else (1024 if width > 1024 else width))