mirror of
https://github.com/LostRuins/koboldcpp.git
synced 2026-09-09 22:39:13 +02:00
Ankit Khandelwal
64e9bceb2c
* vulkan : fuse UNARY(SIGMOID|SILU|SOFTPLUS) + MUL
* vulkan : fuse UNARY(SIGMOID|SILU|SOFTPLUS) + MUL
- implement fusion in unary.comp behind UNARY_MUL_FUSION ifdef,
specialized pipelines per op instead of runtime branching
- fuse adjacent nodes only, ordering handled by graph_optimize
- drop runtime consumer scan and pending_unary_mul deferral
* vulkan : fuse UNARY(GELU|SIGMOID|SILU|SOFTPLUS) + MUL
1. GELU: gelu_mul_f32/f16 pipelines registered, CREATE_UNARY_MUL(gelu), GELU in dispatch + fuse gate + perf fusion name
2. Renamed/moved: gate is now ggml_vk_can_fuse_unary_mul(cgraph, unary_idx, mul_idx), placed with the other can-fuse helpers
3. norepeat both variants: each op gets plain (spec {0}) + _norepeat (spec {1}) pipelines from the same SPIR-V, selected via ggml_are_same_shape(src0, src1); the shape gate now allows broadcast (other dims equal-or-1)
4. graph_optimize: lambda deleted; standard "// UNARY + MUL: pull the consuming MUL forward" block added alongside the SSM_CONV/ROPE/MUL_MAT reorderings, with the same "other src must be weights or already processed" readiness check
* vulkan : align unary_mul fusion with binary kernel layout, relax gelu test tolerance
- schedule the fused kernel like mul.comp (256 threads x 2 unrolled
iterations), recovering a 10-18% prompt-processing regression
- allow 5e-7 f32 error for gelu_mul: the shader evaluates gelu with an
exp-based tanh identity while the CPU reference uses tanhf (~1 ulp)
* vulkan : use ggml_can_repeat in UNARY+MUL fusion shape check
The fused kernel indexes src1 via per-dim fastmod (generic_binary_head.glsl),
which is exact whenever the other operand tiles into the unary result -- not
just when its dims are equal or 1. Replace the hand-rolled loop with
ggml_can_repeat(other, unary) so the check matches the kernel's actual
capability and reuses the standard helper. Argument order matters: reversed,
it would wrongly admit graphs where the unary result is mul->src[1] and the
other operand is larger, producing truncated output.
Also add a rep_ne0 layout to the fused unary+mul backend tests covering a
non-1 repeat factor along dim 0.
* vulkan : fuse UNARY+MUL pairs separated by zero-compute nodes
gemma4's per-layer embedding gating builds gelu -> view_2d_slice -> mul,
where the intervening view is a zero-compute node aliasing an input that
was computed much earlier. Strict adjacency requirements meant neither
CUDA nor the vulkan unary+mul fusion handled this pattern.
Extend ggml_vk_graph_optimize to detect a UNARY whose consuming MUL is
separated only by unscheduled zero-compute nodes (GGML_OP_NONE, VIEW,
RESHAPE, TRANSPOSE, PERMUTE) and schedule those nodes ahead of the pair,
making it adjacent so the existing fusion applies. The reorder is guarded
by ggml_vk_can_fuse_unary_mul, a source-availability check for every
interleaved node, and the protected fusion patterns (topk_moe*, snake);
if fusion is later rejected the reordered graph still executes correctly,
just unfused.
Add a view_mid layout to the fused unary+mul backend tests replicating
the gemma4 pattern.
* vulkan : support OP-on-B in UNARY+MUL fusion
Some models apply the unary activation to the smaller MUL operand, e.g.
qwen3next/qwen35moe shared-expert gating builds ffn_shexp * sigmoid(gate)
with a [1,n_tokens] gate tensor. This shape was correctly rejected before:
the fused kernel derives its iteration extent from the unary tensor and
would leave most of the destination unwritten, and the generic same-shape
requirement in ggml_can_fuse blocked the pair outright.
Add UNARY_MUL_B_FUSION shader variants computing dst = src0 * OP(src1):
the OP operand rides the existing per-dim fastmod indexing, while the
iteration extent now comes from mul. Route {UNARY, MUL} pairs through a
local can-fuse variant that drops the generic same-shape rule and instead
requires the unary result to tile into mul->src[0] (ggml_can_repeat);
pairs with the unary as src0 keep the previous direction check, and
equal-shape pairs keep using the original pipelines.
Add a "gate" layout to the fused unary+mul backend tests covering the
shared-expert gate shape for gelu/sigmoid/silu/softplus in f32 and f16.
* vulkan : fold unary+mul view-hoisting into graph_optimize dep checks
Replace the dedicated UNARY + EMPTY* + MUL scanning block with two small
extensions to the existing scheduling logic:
- a consuming MUL may now join its in-set UNARY across a gap of unused
zero-compute nodes (NONE/VIEW/RESHAPE/TRANSPOSE/PERMUTE), instead of
requiring strict adjacency
- while doing so, such zero-compute blockers are ignored for this pair
Fusion validity is still decided later by ggml_vk_can_fuse at dispatch
time, so a rejected pair simply executes adjacent-but-unfused. Note the
relaxation must stay scoped to this pattern: exempting zero-compute
blockers globally reproduces silent output corruption on gemma3n.
* vulkan : select unary_mul OP-on-B via specialization constant
Replace the UNARY_MUL_B_FUSION compile-time shader variants with an
op_on_b specialization constant on the existing unary_mul SPIR-V,
mirroring how the norepeat flag is handled. The four {op}_mul_b_{f32,f16}
shader artifacts are gone - the OP-on-B pipelines reuse the base SPIR-V
with two-entry {norepeat, op_on_b} spec lists - and the duplicated store
expression is collapsed into a single runtime branch that the driver
prunes per specialization.
The constant is declared only under UNARY_MUL_FUSION so every other
binary pipeline keeps its single-entry specialization list.
* vulkan : replace unary_mul pipeline switches with a lookup table
Collapse the four nested selection switches in ggml_vk_unary_mul into a
single indexed lookup against a pipeline_unary_mul[4][2][2][2] table
([unary op][f16][norepeat][op_on_b]), whose trailing dims mirror the
{norepeat, op_on_b} spec constant list. The op axis uses a small shared
index helper that also replaces the switch in ggml_vk_can_fuse_unary_mul,
making it the only place that maps ops to the table.
Pipeline names are unchanged. Adding another supported op now requires
one macro invocation line and one helper case instead of edits in four
separate switches.
* vulkan : use ggml_can_fuse_subgraph for unary_mul pairs
Replace the hand-rolled pair validation in ggml_vk_can_fuse_unary_mul_pair
(bounds, op match, compute flags, single-use elision) with the shared
ggml_can_fuse_subgraph helper; backend-specific shape/type rules remain in
ggml_vk_can_fuse_unary_mul. Unlike ggml_can_fuse, the subgraph helper has
no same-shape requirement, so it covers both operand slots including
OP-on-B gates, and additionally rejects intermediates flagged as graph
outputs and validates view-source confinement.
The outputs parameter takes absolute node indices into the cgraph.
* Fix Whitespace
* vulkan : drop redundant unary_mul gap check in graph_optimize
The zero-compute nodes separating a UNARY from its consuming MUL are
already scheduled ahead of the pair by pass 2 of an earlier
optimization window, so the scoped gap tolerance added for this pattern
is unreachable in practice - disabling it leaves gemma-3n dispatch
counts unchanged (841 GELU_MUL per pass). Remove the flag, the empty
blocker exemption, and the now-unused gap helper, restoring the strict
adjacency requirement of the UNARY -> MUL pull-forward.
Keep the relaxation scoped out entirely: generalizing "zero-compute
nodes never block" beyond this pattern previously reproduced silent
output corruption on gemma3n.
* vulkan: fix whitespace (tab in indent)
* vulkan: fix whitespace (extra blank line)
* vulkan : move op_on_b spec constant to unary.comp
op_on_b is only used by the fused unary*mul path. Keep
generic_binary_head.glsl generic by defining it in unary.comp
instead. Same constant_id=1 and guard, no functional change.
* vulkan : make RMS_NORM/UNARY fusion gap-tolerant for views
Strict j==c+1 blocked RMS_NORM->MUL and UNARY->MUL when a
VIEW sits between (e.g. rms_norm -> view -> mul). Allow
c==back() with an empty-or-scheduled gap, matching the
review suggestion to check src linkage instead of adjacency.
Scoped to the two blessed pairs; safe because gaps can only
contain zero-compute nodes.
* vulkan : trim comments in UNARY+MUL fusion
Assisted-by: Muse Spark
1385 lines
78 KiB
C++
1385 lines
78 KiB
C++
#include <iostream>
|
|
#include <fstream>
|
|
#include <sstream>
|
|
#include <string>
|
|
#include <stdexcept>
|
|
#include <array>
|
|
#include <vector>
|
|
#include <map>
|
|
#include <thread>
|
|
#include <mutex>
|
|
#include <future>
|
|
#include <queue>
|
|
#include <condition_variable>
|
|
#include <atomic>
|
|
#include <cstdio>
|
|
#include <cstring>
|
|
#include <cstdlib>
|
|
#include <cassert>
|
|
#include <algorithm>
|
|
#include <sys/stat.h>
|
|
#include <sys/types.h>
|
|
#include <filesystem>
|
|
|
|
#ifdef _WIN32
|
|
#define NOMINMAX
|
|
#include <windows.h>
|
|
#include <direct.h> // For _mkdir on Windows
|
|
#else
|
|
#include <unistd.h>
|
|
#include <sys/wait.h>
|
|
#include <fcntl.h>
|
|
#endif
|
|
|
|
#define ASYNCIO_CONCURRENCY 64
|
|
|
|
std::mutex lock;
|
|
std::vector<std::pair<std::string, std::string>> shader_fnames;
|
|
// Set when any shader subprocess fails (non-zero exit / stderr / launch failure) so the
|
|
// build is stopped instead of silently producing a broken libggml-vulkan. (issue #24393)
|
|
static std::atomic<bool> compile_failed{false};
|
|
std::locale c_locale("C");
|
|
|
|
std::string GLSLC = "glslc";
|
|
std::string input_filepath = "";
|
|
std::string output_dir = "/tmp";
|
|
std::string target_hpp = "";
|
|
std::string target_cpp = "";
|
|
|
|
const std::vector<std::string> type_names = {
|
|
"f32",
|
|
"f16",
|
|
"q1_0",
|
|
"q2_0",
|
|
"q4_0",
|
|
"q4_1",
|
|
"q5_0",
|
|
"q5_1",
|
|
"q8_0",
|
|
"q2_k",
|
|
"q3_k",
|
|
"q4_k",
|
|
"q5_k",
|
|
"q6_k",
|
|
"iq1_s",
|
|
"iq1_m",
|
|
"iq2_xxs",
|
|
"iq2_xs",
|
|
"iq2_s",
|
|
"iq3_xxs",
|
|
"iq3_s",
|
|
"iq4_xs",
|
|
"iq4_nl",
|
|
"mxfp4",
|
|
"nvfp4",
|
|
"tq1_0",
|
|
"tq2_0",
|
|
"bf16",
|
|
};
|
|
|
|
enum MatMulIdType {
|
|
NONE,
|
|
DEFAULT,
|
|
SUBGROUP,
|
|
};
|
|
|
|
namespace {
|
|
|
|
int execute_command(std::vector<std::string>& command, std::string& stdout_str, std::string& stderr_str) {
|
|
#ifdef _WIN32
|
|
HANDLE stdout_read, stdout_write;
|
|
HANDLE stderr_read, stderr_write;
|
|
SECURITY_ATTRIBUTES sa = { sizeof(SECURITY_ATTRIBUTES), NULL, TRUE };
|
|
|
|
if (!CreatePipe(&stdout_read, &stdout_write, &sa, 0) ||
|
|
!SetHandleInformation(stdout_read, HANDLE_FLAG_INHERIT, 0)) {
|
|
throw std::runtime_error("Failed to create stdout pipe");
|
|
}
|
|
|
|
if (!CreatePipe(&stderr_read, &stderr_write, &sa, 0) ||
|
|
!SetHandleInformation(stderr_read, HANDLE_FLAG_INHERIT, 0)) {
|
|
throw std::runtime_error("Failed to create stderr pipe");
|
|
}
|
|
|
|
PROCESS_INFORMATION pi;
|
|
STARTUPINFOA si = {};
|
|
si.cb = sizeof(STARTUPINFOA);
|
|
si.dwFlags = STARTF_USESTDHANDLES;
|
|
si.hStdOutput = stdout_write;
|
|
si.hStdError = stderr_write;
|
|
|
|
std::string cmd;
|
|
for (const auto& part : command) {
|
|
cmd += part + " ";
|
|
}
|
|
|
|
if (!CreateProcessA(NULL, cmd.data(), NULL, NULL, TRUE, 0, NULL, NULL, &si, &pi)) {
|
|
throw std::runtime_error("Failed to create process");
|
|
}
|
|
|
|
CloseHandle(stdout_write);
|
|
CloseHandle(stderr_write);
|
|
|
|
std::array<char, 128> buffer;
|
|
DWORD bytes_read;
|
|
|
|
while (ReadFile(stdout_read, buffer.data(), (DWORD)buffer.size(), &bytes_read, NULL) && bytes_read > 0) {
|
|
stdout_str.append(buffer.data(), bytes_read);
|
|
}
|
|
|
|
while (ReadFile(stderr_read, buffer.data(), (DWORD)buffer.size(), &bytes_read, NULL) && bytes_read > 0) {
|
|
stderr_str.append(buffer.data(), bytes_read);
|
|
}
|
|
|
|
CloseHandle(stdout_read);
|
|
CloseHandle(stderr_read);
|
|
WaitForSingleObject(pi.hProcess, INFINITE);
|
|
DWORD exit_code = 1;
|
|
GetExitCodeProcess(pi.hProcess, &exit_code);
|
|
CloseHandle(pi.hProcess);
|
|
CloseHandle(pi.hThread);
|
|
return (int)exit_code;
|
|
#else
|
|
int stdout_pipe[2];
|
|
int stderr_pipe[2];
|
|
|
|
if (pipe(stdout_pipe) != 0 || pipe(stderr_pipe) != 0) {
|
|
throw std::runtime_error("Failed to create pipes");
|
|
}
|
|
|
|
pid_t pid = fork();
|
|
if (pid < 0) {
|
|
std::cerr << strerror(errno) << "\n";
|
|
throw std::runtime_error("Failed to fork process");
|
|
}
|
|
|
|
std::vector<char*> argv;
|
|
for (std::string& part : command) {
|
|
argv.push_back(part.data());
|
|
}
|
|
argv.push_back(nullptr);
|
|
|
|
if (pid == 0) {
|
|
close(stdout_pipe[0]);
|
|
close(stderr_pipe[0]);
|
|
dup2(stdout_pipe[1], STDOUT_FILENO);
|
|
dup2(stderr_pipe[1], STDERR_FILENO);
|
|
close(stdout_pipe[1]);
|
|
close(stderr_pipe[1]);
|
|
execvp(argv[0], argv.data());
|
|
_exit(EXIT_FAILURE);
|
|
} else {
|
|
close(stdout_pipe[1]);
|
|
close(stderr_pipe[1]);
|
|
|
|
std::array<char, 128> buffer;
|
|
ssize_t bytes_read;
|
|
|
|
while ((bytes_read = read(stdout_pipe[0], buffer.data(), buffer.size())) > 0) {
|
|
stdout_str.append(buffer.data(), bytes_read);
|
|
}
|
|
|
|
while ((bytes_read = read(stderr_pipe[0], buffer.data(), buffer.size())) > 0) {
|
|
stderr_str.append(buffer.data(), bytes_read);
|
|
}
|
|
|
|
close(stdout_pipe[0]);
|
|
close(stderr_pipe[0]);
|
|
int status = 0;
|
|
waitpid(pid, &status, 0);
|
|
return WIFEXITED(status) ? WEXITSTATUS(status) : -1;
|
|
}
|
|
#endif
|
|
}
|
|
|
|
bool directory_exists(const std::string& path) {
|
|
struct stat info;
|
|
if (stat(path.c_str(), &info) != 0) {
|
|
return false; // Path doesn't exist or can't be accessed
|
|
}
|
|
return (info.st_mode & S_IFDIR) != 0; // Check if it is a directory
|
|
}
|
|
|
|
bool create_directory(const std::string& path) {
|
|
#ifdef _WIN32
|
|
return _mkdir(path.c_str()) == 0 || errno == EEXIST; // EEXIST means the directory already exists
|
|
#else
|
|
return mkdir(path.c_str(), 0755) == 0 || errno == EEXIST; // 0755 is the directory permissions
|
|
#endif
|
|
}
|
|
|
|
std::string to_uppercase(const std::string& input) {
|
|
std::string result = input;
|
|
for (char& c : result) {
|
|
c = std::toupper(c);
|
|
}
|
|
return result;
|
|
}
|
|
|
|
bool string_starts_with(const std::string& str, const std::string& prefix) {
|
|
if (prefix.size() > str.size()) {
|
|
return false;
|
|
}
|
|
return std::equal(prefix.begin(), prefix.end(), str.begin());
|
|
}
|
|
|
|
bool string_ends_with(const std::string& str, const std::string& suffix) {
|
|
if (suffix.size() > str.size()) {
|
|
return false;
|
|
}
|
|
return std::equal(suffix.rbegin(), suffix.rend(), str.rbegin());
|
|
}
|
|
|
|
bool is_quantized_type(const std::string& type_name) {
|
|
return type_name != "f32" && type_name != "f16" && type_name != "bf16";
|
|
}
|
|
|
|
bool is_legacy_quant(const std::string& type_name) {
|
|
return type_name == "q2_0" || type_name == "q4_0" || type_name == "q4_1" || type_name == "q5_0" || type_name == "q5_1" || type_name == "q8_0";
|
|
}
|
|
|
|
bool is_k_quant(const std::string& type_name) {
|
|
return string_ends_with(type_name, "_k");
|
|
}
|
|
|
|
bool is_iq_quant(const std::string& type_name) {
|
|
return string_starts_with(type_name, "iq");
|
|
}
|
|
|
|
static const char path_separator = '/';
|
|
|
|
std::string join_paths(const std::string& path1, const std::string& path2) {
|
|
return path1 + path_separator + path2;
|
|
}
|
|
|
|
std::string basename(const std::string &path) {
|
|
return path.substr(path.find_last_of("/\\") + 1);
|
|
}
|
|
|
|
std::stringstream make_generic_stringstream() {
|
|
std::stringstream ss;
|
|
ss.imbue(c_locale);
|
|
return ss;
|
|
}
|
|
|
|
std::string read_binary_file(const std::string& path, bool may_not_exist = false) {
|
|
FILE* f = fopen(path.c_str(), "rb");
|
|
if (!f) {
|
|
if (!may_not_exist) {
|
|
std::cerr << "Error opening file: " << path << " (" << strerror(errno) << ")\n";
|
|
}
|
|
return {};
|
|
}
|
|
|
|
fseek(f, 0, SEEK_END);
|
|
size_t size = ftell(f);
|
|
fseek(f, 0, SEEK_SET);
|
|
|
|
std::string data(size, '\0');
|
|
size_t read_size = fread(data.data(), 1, size, f);
|
|
fclose(f);
|
|
if (read_size != size) {
|
|
std::cerr << "Error reading file: " << path << " (" << strerror(errno) << ")\n";
|
|
return {};
|
|
}
|
|
|
|
return data;
|
|
}
|
|
|
|
void write_binary_file(const std::string& path, const std::string& content) {
|
|
FILE* f = fopen(path.c_str(), "wb");
|
|
if (!f) {
|
|
std::cerr << "Error opening file for writing: " << path << " (" << strerror(errno) << ")\n";
|
|
return;
|
|
}
|
|
|
|
size_t write_size = fwrite(content.data(), 1, content.size(), f);
|
|
fclose(f);
|
|
if (write_size != content.size()) {
|
|
std::cerr << "Error writing file: " << path << " (" << strerror(errno) << ")\n";
|
|
return;
|
|
}
|
|
}
|
|
|
|
void write_file_if_changed(const std::string& path, const std::string& content) {
|
|
std::string existing = read_binary_file(path, true);
|
|
if (existing != content) {
|
|
write_binary_file(path, content);
|
|
}
|
|
}
|
|
|
|
|
|
// variables to track number of compiles in progress
|
|
static uint32_t compile_count = 0;
|
|
static std::mutex compile_count_mutex;
|
|
static std::condition_variable compile_count_cond;
|
|
static bool generate_dep_file = true;
|
|
|
|
void decrement_compile_count(uint32_t * count) {
|
|
if (count) {
|
|
std::lock_guard<std::mutex> guard(compile_count_mutex);
|
|
assert(compile_count > 0);
|
|
compile_count--;
|
|
compile_count_cond.notify_all();
|
|
}
|
|
}
|
|
|
|
using compile_count_guard = std::unique_ptr<uint32_t, decltype(&decrement_compile_count)>;
|
|
|
|
compile_count_guard acquire_compile_slot() {
|
|
// wait until fewer than N compiles are in progress.
|
|
// 16 is an arbitrary limit, the goal is to avoid "failed to create pipe" errors.
|
|
uint32_t N = std::max(1u, std::min(16u, std::thread::hardware_concurrency()));
|
|
std::unique_lock<std::mutex> guard(compile_count_mutex);
|
|
compile_count_cond.wait(guard, [N] { return compile_count < N; });
|
|
compile_count++;
|
|
return compile_count_guard(&compile_count, &decrement_compile_count);
|
|
}
|
|
|
|
void string_to_spv_func(std::string name, std::string in_path, std::string out_path, std::map<std::string, std::string> defines, bool coopmat, bool dep_file, compile_count_guard slot) {
|
|
std::string target_env = (name.find("_cm2") != std::string::npos) ? "--target-env=vulkan1.3" : "--target-env=vulkan1.2";
|
|
|
|
#ifdef _WIN32
|
|
std::vector<std::string> cmd = {GLSLC, "-fshader-stage=compute", target_env, "\"" + in_path + "\"", "-o", "\"" + out_path + "\""};
|
|
#else
|
|
std::vector<std::string> cmd = {GLSLC, "-fshader-stage=compute", target_env, in_path, "-o", out_path};
|
|
#endif
|
|
|
|
// disable spirv-opt for coopmat shaders for https://github.com/ggml-org/llama.cpp/issues/10734
|
|
// disable spirv-opt for bf16 shaders for https://github.com/ggml-org/llama.cpp/issues/15344
|
|
// disable spirv-opt for rope shaders for https://github.com/ggml-org/llama.cpp/issues/16860
|
|
// disable spirv-opt for dot2 shaders (spirv-opt doesn't recognize SPV_VALVE_mixed_float_dot_product capability)
|
|
if (!coopmat && name.find("bf16") == std::string::npos && name.find("rope") == std::string::npos && name.find("_dot2") == std::string::npos) {
|
|
cmd.push_back("-O");
|
|
}
|
|
|
|
if (dep_file) {
|
|
cmd.push_back("-MD");
|
|
cmd.push_back("-MF");
|
|
#ifdef _WIN32
|
|
cmd.push_back("\"" + target_cpp + ".d\"");
|
|
#else
|
|
cmd.push_back(target_cpp + ".d");
|
|
#endif
|
|
}
|
|
|
|
#ifdef GGML_VULKAN_SHADER_DEBUG_INFO
|
|
cmd.push_back("-g");
|
|
#endif
|
|
|
|
for (const auto& define : defines) {
|
|
cmd.push_back("-D" + define.first + "=" + define.second);
|
|
}
|
|
|
|
std::string command;
|
|
for (const auto& part : cmd) {
|
|
command += part + " ";
|
|
}
|
|
|
|
std::string stdout_str, stderr_str;
|
|
try {
|
|
// std::cout << "Executing command: ";
|
|
// for (const auto& part : cmd) {
|
|
// std::cout << part << " ";
|
|
// }
|
|
// std::cout << std::endl;
|
|
|
|
int exit_code = execute_command(cmd, stdout_str, stderr_str);
|
|
if (exit_code != 0 || !stderr_str.empty()) {
|
|
std::cerr << "cannot compile " << name << " (exit code " << exit_code << ")\n\n";
|
|
for (const auto& part : cmd) {
|
|
std::cerr << part << " ";
|
|
}
|
|
std::cerr << "\n\n" << stderr_str << std::endl;
|
|
compile_failed = true;
|
|
return;
|
|
}
|
|
|
|
if (dep_file) {
|
|
// replace .spv output path with the embed .cpp path which is used as output in CMakeLists.txt
|
|
std::string dep = read_binary_file(target_cpp + ".d", true);
|
|
if (!dep.empty()) {
|
|
size_t pos = dep.find(out_path);
|
|
if (pos != std::string::npos) {
|
|
dep.replace(pos, out_path.length(), target_cpp);
|
|
}
|
|
write_binary_file(target_cpp + ".d", dep);
|
|
}
|
|
}
|
|
|
|
std::lock_guard<std::mutex> guard(lock);
|
|
shader_fnames.push_back(std::make_pair(name, out_path));
|
|
} catch (const std::exception& e) {
|
|
std::cerr << "Error executing command for " << name << ": " << e.what() << std::endl;
|
|
compile_failed = true;
|
|
}
|
|
}
|
|
|
|
std::map<std::string, std::string> merge_maps(const std::map<std::string, std::string>& a, const std::map<std::string, std::string>& b) {
|
|
std::map<std::string, std::string> result = a;
|
|
result.insert(b.begin(), b.end());
|
|
return result;
|
|
}
|
|
|
|
static std::deque<std::future<void>> compiles;
|
|
void string_to_spv(std::string name, const std::string& source, const std::map<std::string, std::string>& defines, bool fp16 = true, bool coopmat = false, bool coopmat2 = false, bool f16acc = false, const std::string& suffix = "") {
|
|
name = name + (f16acc ? "_f16acc" : "") + (coopmat ? "_cm1" : "") + (coopmat2 ? "_cm2" : (fp16 ? "" : "_fp32")) + suffix;
|
|
std::string out_path = join_paths(output_dir, name + ".spv");
|
|
|
|
if (input_filepath == "") {
|
|
// No input source to compile, only generate header for all shaders
|
|
shader_fnames.push_back(std::pair(name, out_path));
|
|
return;
|
|
} else if (basename(input_filepath) != source) {
|
|
// Only compile shader variants matching the input filename
|
|
return;
|
|
}
|
|
|
|
compile_count_guard slot = acquire_compile_slot();
|
|
compiles.push_back(std::async(
|
|
string_to_spv_func, name, input_filepath, out_path, defines, coopmat, generate_dep_file, std::move(slot)));
|
|
// Don't write the same dep file from multiple processes
|
|
generate_dep_file = false;
|
|
|
|
// Clean up completed futures - don't accumulate virtual memory for completed threads' stacks.
|
|
while (!compiles.empty() && compiles.front().wait_for(std::chrono::seconds(0)) == std::future_status::ready) {
|
|
compiles.pop_front();
|
|
}
|
|
}
|
|
|
|
void matmul_shaders(bool fp16, MatMulIdType matmul_id_type, bool coopmat, bool coopmat2, bool f16acc, bool dot2 = false) {
|
|
std::string load_vec = coopmat2 ? "1" : fp16 ? "8" : "4";
|
|
std::string aligned_b_type_f32 = coopmat2 ? "float" : fp16 ? "mat2x4" : "vec4";
|
|
std::string aligned_b_type_f16 = coopmat2 ? "float16_t" : fp16 ? "f16mat2x4" : "f16vec4";
|
|
std::string dot2_sfx = dot2 ? "_dot2" : "";
|
|
|
|
std::map<std::string, std::string> base_dict;
|
|
std::string shader_name = "matmul";
|
|
|
|
if (matmul_id_type == MatMulIdType::DEFAULT) {
|
|
base_dict["MUL_MAT_ID"] = "1";
|
|
shader_name = "matmul_id";
|
|
} else if (matmul_id_type == MatMulIdType::SUBGROUP) {
|
|
base_dict["MUL_MAT_ID"] = "1";
|
|
base_dict["MUL_MAT_ID_USE_SUBGROUPS"] = "1";
|
|
shader_name = "matmul_id_subgroup";
|
|
}
|
|
|
|
if (fp16) {
|
|
base_dict["FLOAT16"] = "1";
|
|
}
|
|
|
|
base_dict["ACC_TYPE" ] = f16acc ? "float16_t" : "float";
|
|
base_dict["ACC_TYPEV2"] = f16acc ? "f16vec2" : "vec2";
|
|
if (f16acc) {
|
|
base_dict["ACC_TYPE_MAX"] = "float16_t(65504.0)";
|
|
}
|
|
|
|
if (coopmat) {
|
|
base_dict["COOPMAT"] = "1";
|
|
}
|
|
#if defined(GGML_VULKAN_COOPMAT2_DECODE_VECTOR_GLSLC_SUPPORT)
|
|
if (coopmat2) {
|
|
base_dict["GGML_VULKAN_COOPMAT2_DECODE_VECTOR"] = "1";
|
|
}
|
|
#endif
|
|
|
|
if (dot2) {
|
|
base_dict["DOT2_F16"] = "1";
|
|
}
|
|
|
|
const std::string source_name = coopmat2 ? "mul_mm_cm2.comp" : "mul_mm.comp";
|
|
|
|
auto const &FLOAT_TYPE = [&](int vec, const std::string &t) -> std::string {
|
|
switch (vec) {
|
|
case 1:
|
|
if (t == "bf16") {
|
|
// scalar path promotes to float
|
|
if (!coopmat && !coopmat2) {
|
|
return "float";
|
|
}
|
|
return "bfloat16_t";
|
|
}
|
|
if (coopmat2 || fp16) {
|
|
return "float16_t";
|
|
}
|
|
return "float";
|
|
case 2:
|
|
if (t == "bf16") {
|
|
// scalar path promotes to float
|
|
if (!coopmat && !coopmat2) {
|
|
return "vec2";
|
|
}
|
|
return "bf16vec2";
|
|
}
|
|
if (coopmat2 || fp16) {
|
|
return "f16vec2";
|
|
}
|
|
return "vec2";
|
|
case 4:
|
|
if (t == "bf16") {
|
|
// scalar path promotes to float
|
|
if (!coopmat && !coopmat2) {
|
|
return "vec4";
|
|
}
|
|
return "bf16vec4";
|
|
}
|
|
if (coopmat2 || fp16) {
|
|
return "f16vec4";
|
|
}
|
|
return "vec4";
|
|
case 8:
|
|
if (t == "bf16") {
|
|
// scalar path promotes to float
|
|
if (!coopmat && !coopmat2) {
|
|
return "mat2x4";
|
|
}
|
|
throw std::runtime_error("bf16 vec8 not supported");
|
|
}
|
|
if (coopmat2 || fp16) {
|
|
return "f16mat2x4";
|
|
}
|
|
return "mat2x4";
|
|
default:
|
|
throw std::runtime_error("invalid vector size");
|
|
}
|
|
};
|
|
|
|
const std::map<std::string, std::string> float_type_dict_f16 = {
|
|
{"FLOAT_TYPE", FLOAT_TYPE(1, "f16")},
|
|
{"FLOAT_TYPEV2", FLOAT_TYPE(2, "f16")},
|
|
{"FLOAT_TYPEV4", FLOAT_TYPE(4, "f16")},
|
|
{"FLOAT_TYPEV8", FLOAT_TYPE(8, "f16")},
|
|
};
|
|
|
|
// Shaders with f16 B_TYPE
|
|
string_to_spv(shader_name + "_f32_f16" + dot2_sfx, source_name, merge_maps(merge_maps(base_dict, float_type_dict_f16), {{"DATA_A_F32", "1"}, {"LOAD_VEC_A", load_vec}, {"LOAD_VEC_B", load_vec}, {"B_TYPE", aligned_b_type_f16}, {"B_TYPE_SCALAR", "float16_t"}, {"B_TYPEV4", "f16vec4"}, {"D_TYPE", "float"}}), fp16, coopmat, coopmat2, f16acc);
|
|
|
|
string_to_spv(shader_name + "_f16" + dot2_sfx, source_name, merge_maps(merge_maps(base_dict, float_type_dict_f16), {{"DATA_A_F16", "1"}, {"LOAD_VEC_A", load_vec}, {"LOAD_VEC_B", load_vec}, {"B_TYPE", aligned_b_type_f16}, {"B_TYPE_SCALAR", "float16_t"}, {"B_TYPEV4", "f16vec4"}, {"D_TYPE", "float"}}), fp16, coopmat, coopmat2, f16acc);
|
|
|
|
// bf16
|
|
{
|
|
// For aligned matmul loads
|
|
std::string load_vec_a = coopmat2 ? "1" : "4";
|
|
|
|
// scalar path promotes to float
|
|
std::string to_float_type = (coopmat || coopmat2) ? "uintBitsToBFloat16EXT" : "bf16_to_fp32";
|
|
|
|
const std::map<std::string, std::string> float_type_dict_bf16 = {
|
|
{"FLOAT_TYPE", FLOAT_TYPE(1, "bf16")},
|
|
{"FLOAT_TYPEV2", FLOAT_TYPE(2, "bf16")},
|
|
{"FLOAT_TYPEV4", FLOAT_TYPE(4, "bf16")},
|
|
};
|
|
|
|
// If bfloat16 is not supported, then only compile the scalar (promote to fp32) shader
|
|
#if !defined(GGML_VULKAN_BFLOAT16_GLSLC_SUPPORT)
|
|
if (!(coopmat || coopmat2))
|
|
#endif
|
|
{
|
|
if (!dot2) {
|
|
string_to_spv(shader_name + "_bf16", source_name, merge_maps(merge_maps(base_dict, float_type_dict_bf16), {{"TO_FLOAT_TYPE", to_float_type}, {"DATA_A_BF16", "1"}, {"LOAD_VEC_A", load_vec_a}, {"LOAD_VEC_B", "4"}, {"B_TYPE", coopmat2 ? "bfloat16_t" : "u16vec4"}, {"B_TYPE_SCALAR", coopmat2 ? "bfloat16_t" : "uint16_t"}, {"B_TYPEV4", "bf16vec4"}, {"D_TYPE", "float"}, {"B_IS_FLOAT", "1"}, {"DATA_B_BF16", "1"}}), fp16, coopmat, coopmat2, f16acc);
|
|
}
|
|
}
|
|
}
|
|
|
|
for (const auto& tname : type_names) {
|
|
std::string load_vec_quant = "2";
|
|
if ((tname == "q1_0") || (tname == "q4_0") || (tname == "q4_1") || (tname == "q5_1") || (tname == "iq1_s") || (tname == "iq1_m") || (tname == "iq2_xxs") || (tname == "iq2_xs") || (tname == "iq2_s"))
|
|
load_vec_quant = "8";
|
|
else if ((tname == "q2_0") || (tname == "q5_0") || (tname == "q8_0") || (tname == "q2_k") || (tname == "q4_k") || (tname == "q5_k") || (tname == "iq3_xxs") || (tname == "iq3_s") || (tname == "iq4_xs") || (tname == "iq4_nl") || (tname == "mxfp4") || (tname == "nvfp4"))
|
|
load_vec_quant = "4";
|
|
|
|
if (tname == "bf16") {
|
|
continue;
|
|
}
|
|
|
|
std::string data_a_key = "DATA_A_" + to_uppercase(tname);
|
|
// For aligned matmul loads
|
|
std::string load_vec_a = (coopmat2 || tname == "f32" || tname == "f16" || tname == "bf16") ? load_vec : load_vec_quant;
|
|
|
|
const std::map<std::string, std::string> float_type_dict = {
|
|
{"FLOAT_TYPE", FLOAT_TYPE(1, tname)},
|
|
{"FLOAT_TYPEV2", FLOAT_TYPE(2, tname)},
|
|
{"FLOAT_TYPEV4", FLOAT_TYPE(4, tname)},
|
|
{"FLOAT_TYPEV8", FLOAT_TYPE(8, tname)},
|
|
};
|
|
|
|
// don't generate f32 variants for coopmat2
|
|
if (!coopmat2) {
|
|
string_to_spv(shader_name + "_" + tname + "_f32" + dot2_sfx, source_name, merge_maps(merge_maps(base_dict, float_type_dict), {{data_a_key, "1"}, {"LOAD_VEC_A", load_vec_a}, {"LOAD_VEC_B", load_vec}, {"B_TYPE", aligned_b_type_f32}, {"B_TYPE_SCALAR", "float"}, {"B_TYPEV4", "vec4"}, {"D_TYPE", "float"}}), fp16, coopmat, coopmat2, f16acc);
|
|
}
|
|
|
|
if (tname != "f16" && tname != "f32") {
|
|
string_to_spv(shader_name + "_" + tname + "_f16" + dot2_sfx, source_name, merge_maps(merge_maps(base_dict, float_type_dict), {{data_a_key, "1"}, {"LOAD_VEC_A", load_vec_a}, {"LOAD_VEC_B", load_vec}, {"B_TYPE", aligned_b_type_f16}, {"B_TYPE_SCALAR", "float16_t"}, {"B_TYPEV4", "f16vec4"}, {"D_TYPE", "float"}}), fp16, coopmat, coopmat2, f16acc);
|
|
}
|
|
|
|
#if defined(GGML_VULKAN_FLOAT_E2M1_GLSLC_SUPPORT) && defined(GGML_VULKAN_FLOAT_E4M3_GLSLC_SUPPORT)
|
|
if ((coopmat || coopmat2) && (tname == "mxfp4" || tname == "nvfp4")) {
|
|
if (!coopmat2) {
|
|
string_to_spv(shader_name + "_" + tname + "_f32_ocp" + dot2_sfx, source_name, merge_maps(merge_maps(base_dict, float_type_dict), {{data_a_key, "1"}, {"USE_OCP_FP4", "1"}, {"LOAD_VEC_A", load_vec_a}, {"LOAD_VEC_B", load_vec}, {"B_TYPE", aligned_b_type_f32}, {"B_TYPE_SCALAR", "float"}, {"B_TYPEV4", "vec4"}, {"D_TYPE", "float"}}), fp16, coopmat, coopmat2, f16acc);
|
|
}
|
|
string_to_spv(shader_name + "_" + tname + "_f16_ocp" + dot2_sfx, source_name, merge_maps(merge_maps(base_dict, float_type_dict), {{data_a_key, "1"}, {"USE_OCP_FP4", "1"}, {"LOAD_VEC_A", load_vec_a}, {"LOAD_VEC_B", load_vec}, {"B_TYPE", aligned_b_type_f16}, {"B_TYPE_SCALAR", "float16_t"}, {"B_TYPEV4", "f16vec4"}, {"D_TYPE", "float"}}), fp16, coopmat, coopmat2, f16acc);
|
|
}
|
|
#endif
|
|
|
|
#if defined(GGML_VULKAN_INTEGER_DOT_GLSLC_SUPPORT)
|
|
// Integer dot mmq performs better with f32 accumulators (different shader, skip for dot2)
|
|
if (!f16acc && !coopmat && !coopmat2 && !dot2 && (is_legacy_quant(tname) || is_k_quant(tname) || tname == "mxfp4")) {
|
|
string_to_spv(shader_name + "_" + tname + "_q8_1", "mul_mmq.comp", merge_maps(merge_maps(base_dict, float_type_dict), {{data_a_key, "1"}, {"D_TYPE", "float"},}), fp16, coopmat, coopmat2, f16acc);
|
|
}
|
|
#endif
|
|
}
|
|
}
|
|
|
|
void process_shaders() {
|
|
// matmul
|
|
for (const MatMulIdType& matmul_id_type : {MatMulIdType::NONE, MatMulIdType::DEFAULT, MatMulIdType::SUBGROUP}) {
|
|
// No coopmats
|
|
// fp32
|
|
matmul_shaders(false, matmul_id_type, false, false, false);
|
|
|
|
// fp16, fp32acc and fp16acc
|
|
matmul_shaders(true, matmul_id_type, false, false, false);
|
|
matmul_shaders(true, matmul_id_type, false, false, true);
|
|
|
|
// dot2 variants (scalar fp16 only)
|
|
matmul_shaders(true, matmul_id_type, false, false, false, true);
|
|
matmul_shaders(true, matmul_id_type, false, false, true, true);
|
|
|
|
if (matmul_id_type != MatMulIdType::DEFAULT) {
|
|
#if defined(GGML_VULKAN_COOPMAT_GLSLC_SUPPORT)
|
|
// Coopmat, fp32acc and fp16acc
|
|
matmul_shaders(true, matmul_id_type, true, false, false);
|
|
matmul_shaders(true, matmul_id_type, true, false, true);
|
|
#endif
|
|
|
|
#if defined(GGML_VULKAN_COOPMAT2_GLSLC_SUPPORT)
|
|
// Coopmat2, fp32acc and fp16acc
|
|
matmul_shaders(true, matmul_id_type, false, true, false);
|
|
matmul_shaders(true, matmul_id_type, false, true, true);
|
|
#endif
|
|
}
|
|
}
|
|
|
|
for (const bool& fp16 : {false, true}) {
|
|
std::map<std::string, std::string> base_dict;
|
|
if (fp16) {
|
|
base_dict = {{"FLOAT_TYPE", "float16_t"}, {"FLOAT_TYPEV2", "f16vec2"}, {"FLOAT_TYPEV4", "f16vec4"}, {"FLOAT16", "1"}, {"FLOAT_TYPE_MAX", "float16_t(65504.0)"}};
|
|
} else {
|
|
base_dict = {{"FLOAT_TYPE", "float"}, {"FLOAT_TYPEV2", "vec2"}, {"FLOAT_TYPEV4", "vec4"}};
|
|
}
|
|
|
|
// flash attention
|
|
for (const bool& f16acc : {false, true}) {
|
|
std::map<std::string, std::string> fa_base_dict = base_dict;
|
|
fa_base_dict["ACC_TYPE"] = fp16 && f16acc ? "float16_t" : "float";
|
|
fa_base_dict["ACC_TYPEV2"] = fp16 && f16acc ? "f16vec2" : "vec2";
|
|
fa_base_dict["ACC_TYPEV4"] = fp16 && f16acc ? "f16vec4" : "vec4";
|
|
// Compile IQ4_NL support into all FA variants so its shared LUT is available when K or V uses it.
|
|
fa_base_dict["DATA_A_IQ4_NL"] = "1";
|
|
if (fp16 && f16acc) {
|
|
fa_base_dict["ACC_TYPE_MAX"] = "float16_t(65504.0)";
|
|
}
|
|
|
|
if (fp16) {
|
|
#if defined(GGML_VULKAN_COOPMAT2_GLSLC_SUPPORT)
|
|
string_to_spv("flash_attn_f32_f16", "flash_attn_cm2.comp",
|
|
merge_maps(fa_base_dict, {{"Q_TYPE", "float"}, {"D_TYPE", "float"}, {"D_TYPEV4", "vec4"}}), fp16, false, true, f16acc);
|
|
#endif
|
|
|
|
#if defined(GGML_VULKAN_COOPMAT_GLSLC_SUPPORT)
|
|
string_to_spv("flash_attn_f32_f16", "flash_attn_cm1.comp",
|
|
merge_maps(fa_base_dict, {{"Q_TYPE", "float"}, {"D_TYPE", "float"}, {"D_TYPEV4", "vec4"}, {"COOPMAT", "1"}}), fp16, true, false, f16acc);
|
|
#endif
|
|
}
|
|
|
|
string_to_spv("flash_attn_f32_f16", "flash_attn.comp",
|
|
merge_maps(fa_base_dict, {{"Q_TYPE", "float"}, {"D_TYPE", "float"}, {"D_TYPEV4", "vec4"}}), fp16, false, false, f16acc);
|
|
|
|
if (fp16) {
|
|
string_to_spv("flash_attn_f32_f16_dot2", "flash_attn.comp",
|
|
merge_maps(fa_base_dict, {{"Q_TYPE", "float"}, {"D_TYPE", "float"}, {"D_TYPEV4", "vec4"}, {"DOT2_F16", "1"}}), fp16, false, false, f16acc);
|
|
}
|
|
|
|
#if defined(GGML_VULKAN_INTEGER_DOT_GLSLC_SUPPORT)
|
|
string_to_spv("flash_attn_f32_f16", "flash_attn.comp",
|
|
merge_maps(fa_base_dict, {{"Q_TYPE", "float"}, {"D_TYPE", "float"}, {"D_TYPEV4", "vec4"}, {"MMQ", "1"}, {"FA_MMQ_MIXED", "1"}}), fp16, false, false, f16acc, "_int8");
|
|
#endif
|
|
}
|
|
}
|
|
|
|
const std::map<std::string, std::string> fa_bf16_dict = {
|
|
{"FLOAT_TYPE", "bfloat16_t"},
|
|
{"FLOAT_TYPEV2", "bf16vec2"},
|
|
{"FLOAT_TYPEV4", "bf16vec4"},
|
|
{"ACC_TYPE", "float"},
|
|
{"ACC_TYPEV2", "vec2"},
|
|
{"ACC_TYPEV4", "vec4"},
|
|
{"BFLOAT16", "1"},
|
|
};
|
|
|
|
#if defined(GGML_VULKAN_BFLOAT16_GLSLC_SUPPORT) && defined(GGML_VULKAN_COOPMAT_GLSLC_SUPPORT)
|
|
string_to_spv("flash_attn_f32_f16_bf16", "flash_attn_cm1.comp",
|
|
merge_maps(fa_bf16_dict, {{"Q_TYPE", "float"}, {"D_TYPE", "float"}, {"D_TYPEV4", "vec4"}, {"COOPMAT", "1"}}),
|
|
true, true, false, false);
|
|
#endif
|
|
|
|
#if defined(GGML_VULKAN_BFLOAT16_GLSLC_SUPPORT) && defined(GGML_VULKAN_COOPMAT2_GLSLC_SUPPORT)
|
|
string_to_spv("flash_attn_f32_f16_bf16", "flash_attn_cm2.comp",
|
|
merge_maps(fa_bf16_dict, {{"Q_TYPE", "float"}, {"D_TYPE", "float"}, {"D_TYPEV4", "vec4"}}),
|
|
true, false, true, false);
|
|
#endif
|
|
|
|
std::map<std::string, std::string> base_dict = {{"FLOAT_TYPE", "float"}, {"FLOAT_TYPEV2", "vec2"}};
|
|
|
|
for (const auto& tname : type_names) {
|
|
// mul mat vec
|
|
std::string data_a_key = "DATA_A_" + to_uppercase(tname);
|
|
std::string shader = (string_ends_with(tname, "_k") || string_starts_with(tname, "iq1_") || string_starts_with(tname, "iq2_") || string_starts_with(tname, "iq3_") || tname == "tq2_0" || tname == "tq1_0") ? "mul_mat_vec_" + tname + ".comp" : "mul_mat_vec.comp";
|
|
|
|
string_to_spv("mul_mat_vec_" + tname + "_f32_f32", shader, merge_maps(base_dict, {{data_a_key, "1"}, {"B_TYPE", "float"}, {"B_TYPEV2", "vec2"}, {"B_TYPEV4", "vec4"}, {"D_TYPE", "float"}}));
|
|
string_to_spv("mul_mat_vec_" + tname + "_f16_f32", shader, merge_maps(base_dict, {{data_a_key, "1"}, {"B_TYPE", "float16_t"}, {"B_TYPEV2", "f16vec2"}, {"B_TYPEV4", "f16vec4"}, {"D_TYPE", "float"}}));
|
|
|
|
string_to_spv("mul_mat_vec_" + tname + "_f32_f32_subgroup", shader, merge_maps(base_dict, {{data_a_key, "1"}, {"B_TYPE", "float"}, {"B_TYPEV2", "vec2"}, {"B_TYPEV4", "vec4"}, {"D_TYPE", "float"}, {"USE_SUBGROUP_ADD", "1"}}));
|
|
string_to_spv("mul_mat_vec_" + tname + "_f16_f32_subgroup", shader, merge_maps(base_dict, {{data_a_key, "1"}, {"B_TYPE", "float16_t"}, {"B_TYPEV2", "f16vec2"}, {"B_TYPEV4", "f16vec4"}, {"D_TYPE", "float"}, {"USE_SUBGROUP_ADD", "1"}}));
|
|
|
|
string_to_spv("mul_mat_vec_" + tname + "_f32_f32_subgroup_no_shmem", shader, merge_maps(base_dict, {{data_a_key, "1"}, {"B_TYPE", "float"}, {"B_TYPEV2", "vec2"}, {"B_TYPEV4", "vec4"}, {"D_TYPE", "float"}, {"USE_SUBGROUP_ADD_NO_SHMEM", "1"}}));
|
|
string_to_spv("mul_mat_vec_" + tname + "_f16_f32_subgroup_no_shmem", shader, merge_maps(base_dict, {{data_a_key, "1"}, {"B_TYPE", "float16_t"}, {"B_TYPEV2", "f16vec2"}, {"B_TYPEV4", "f16vec4"}, {"D_TYPE", "float"}, {"USE_SUBGROUP_ADD_NO_SHMEM", "1"}}));
|
|
|
|
#if defined(GGML_VULKAN_FLOAT_E2M1_GLSLC_SUPPORT) && defined(GGML_VULKAN_FLOAT_E4M3_GLSLC_SUPPORT)
|
|
if (tname == "mxfp4" || tname == "nvfp4") {
|
|
string_to_spv("mul_mat_vec_" + tname + "_f32_f32_ocp", shader, merge_maps(base_dict, {{data_a_key, "1"}, {"USE_OCP_FP4", "1"}, {"B_TYPE", "float"}, {"B_TYPEV2", "vec2"}, {"B_TYPEV4", "vec4"}, {"D_TYPE", "float"}}));
|
|
string_to_spv("mul_mat_vec_" + tname + "_f16_f32_ocp", shader, merge_maps(base_dict, {{data_a_key, "1"}, {"USE_OCP_FP4", "1"}, {"B_TYPE", "float16_t"}, {"B_TYPEV2", "f16vec2"}, {"B_TYPEV4", "f16vec4"}, {"D_TYPE", "float"}}));
|
|
string_to_spv("mul_mat_vec_" + tname + "_f32_f32_ocp_subgroup", shader, merge_maps(base_dict, {{data_a_key, "1"}, {"USE_OCP_FP4", "1"}, {"B_TYPE", "float"}, {"B_TYPEV2", "vec2"}, {"B_TYPEV4", "vec4"}, {"D_TYPE", "float"}, {"USE_SUBGROUP_ADD", "1"}}));
|
|
string_to_spv("mul_mat_vec_" + tname + "_f16_f32_ocp_subgroup", shader, merge_maps(base_dict, {{data_a_key, "1"}, {"USE_OCP_FP4", "1"}, {"B_TYPE", "float16_t"}, {"B_TYPEV2", "f16vec2"}, {"B_TYPEV4", "f16vec4"}, {"D_TYPE", "float"}, {"USE_SUBGROUP_ADD", "1"}}));
|
|
string_to_spv("mul_mat_vec_" + tname + "_f32_f32_ocp_subgroup_no_shmem", shader, merge_maps(base_dict, {{data_a_key, "1"}, {"USE_OCP_FP4", "1"}, {"B_TYPE", "float"}, {"B_TYPEV2", "vec2"}, {"B_TYPEV4", "vec4"}, {"D_TYPE", "float"}, {"USE_SUBGROUP_ADD_NO_SHMEM", "1"}}));
|
|
string_to_spv("mul_mat_vec_" + tname + "_f16_f32_ocp_subgroup_no_shmem", shader, merge_maps(base_dict, {{data_a_key, "1"}, {"USE_OCP_FP4", "1"}, {"B_TYPE", "float16_t"}, {"B_TYPEV2", "f16vec2"}, {"B_TYPEV4", "f16vec4"}, {"D_TYPE", "float"}, {"USE_SUBGROUP_ADD_NO_SHMEM", "1"}}));
|
|
string_to_spv("mul_mat_vec_id_" + tname + "_f32_f32_ocp", shader, merge_maps(base_dict, {{"MUL_MAT_ID", "1"}, {data_a_key, "1"}, {"USE_OCP_FP4", "1"}, {"B_TYPE", "float"}, {"B_TYPEV2", "vec2"}, {"B_TYPEV4", "vec4"}, {"D_TYPE", "float"}}));
|
|
string_to_spv("mul_mat_vec_id_" + tname + "_f32_f32_ocp_subgroup", shader, merge_maps(base_dict, {{"MUL_MAT_ID", "1"}, {data_a_key, "1"}, {"USE_OCP_FP4", "1"}, {"B_TYPE", "float"}, {"B_TYPEV2", "vec2"}, {"B_TYPEV4", "vec4"}, {"D_TYPE", "float"}, {"USE_SUBGROUP_ADD", "1"}}));
|
|
string_to_spv("mul_mat_vec_id_" + tname + "_f32_f32_ocp_subgroup_no_shmem", shader, merge_maps(base_dict, {{"MUL_MAT_ID", "1"}, {data_a_key, "1"}, {"USE_OCP_FP4", "1"}, {"B_TYPE", "float"}, {"B_TYPEV2", "vec2"}, {"B_TYPEV4", "vec4"}, {"D_TYPE", "float"}, {"USE_SUBGROUP_ADD_NO_SHMEM", "1"}}));
|
|
}
|
|
#endif
|
|
|
|
string_to_spv("mul_mat_vec_id_" + tname + "_f32_f32", shader, merge_maps(base_dict, {{"MUL_MAT_ID", "1"}, {data_a_key, "1"}, {"B_TYPE", "float"}, {"B_TYPEV2", "vec2"}, {"B_TYPEV4", "vec4"}, {"D_TYPE", "float"}}));
|
|
string_to_spv("mul_mat_vec_id_" + tname + "_f32_f32_subgroup", shader, merge_maps(base_dict, {{"MUL_MAT_ID", "1"}, {data_a_key, "1"}, {"B_TYPE", "float"}, {"B_TYPEV2", "vec2"}, {"B_TYPEV4", "vec4"}, {"D_TYPE", "float"}, {"USE_SUBGROUP_ADD", "1"}}));
|
|
string_to_spv("mul_mat_vec_id_" + tname + "_f32_f32_subgroup_no_shmem", shader, merge_maps(base_dict, {{"MUL_MAT_ID", "1"}, {data_a_key, "1"}, {"B_TYPE", "float"}, {"B_TYPEV2", "vec2"}, {"B_TYPEV4", "vec4"}, {"D_TYPE", "float"}, {"USE_SUBGROUP_ADD_NO_SHMEM", "1"}}));
|
|
|
|
// mul mat vec with integer dot product
|
|
#if defined(GGML_VULKAN_INTEGER_DOT_GLSLC_SUPPORT)
|
|
if (is_legacy_quant(tname) || tname == "mxfp4" || is_k_quant(tname) || tname == "iq1_s" || tname == "iq1_m") {
|
|
string_to_spv("mul_mat_vec_" + tname + "_q8_1_f32", "mul_mat_vecq.comp", merge_maps(base_dict, {{data_a_key, "1"}, {"D_TYPE", "float"}, {"FLOAT_TYPE", "float"}, {"FLOAT_TYPEV2", "vec2"}, {"ACC_TYPE", "float"}}));
|
|
string_to_spv("mul_mat_vec_" + tname + "_q8_1_f32_subgroup", "mul_mat_vecq.comp", merge_maps(base_dict, {{data_a_key, "1"}, {"D_TYPE", "float"}, {"FLOAT_TYPE", "float"}, {"FLOAT_TYPEV2", "vec2"}, {"ACC_TYPE", "float"}, {"USE_SUBGROUP_ADD", "1"}}));
|
|
string_to_spv("mul_mat_vec_" + tname + "_q8_1_f32_subgroup_no_shmem", "mul_mat_vecq.comp", merge_maps(base_dict, {{data_a_key, "1"}, {"D_TYPE", "float"}, {"FLOAT_TYPE", "float"}, {"FLOAT_TYPEV2", "vec2"}, {"ACC_TYPE", "float"}, {"USE_SUBGROUP_ADD_NO_SHMEM", "1"}}));
|
|
|
|
string_to_spv("mul_mat_vec_id_" + tname + "_q8_1_f32", "mul_mat_vecq.comp", merge_maps(base_dict, {{"MUL_MAT_ID", "1"}, {data_a_key, "1"}, {"D_TYPE", "float"}, {"FLOAT_TYPE", "float"}, {"FLOAT_TYPEV2", "vec2"}, {"ACC_TYPE", "float"}}));
|
|
string_to_spv("mul_mat_vec_id_" + tname + "_q8_1_f32_subgroup", "mul_mat_vecq.comp", merge_maps(base_dict, {{"MUL_MAT_ID", "1"}, {data_a_key, "1"}, {"D_TYPE", "float"}, {"FLOAT_TYPE", "float"}, {"FLOAT_TYPEV2", "vec2"}, {"ACC_TYPE", "float"}, {"USE_SUBGROUP_ADD", "1"}}));
|
|
string_to_spv("mul_mat_vec_id_" + tname + "_q8_1_f32_subgroup_no_shmem", "mul_mat_vecq.comp", merge_maps(base_dict, {{"MUL_MAT_ID", "1"}, {data_a_key, "1"}, {"D_TYPE", "float"}, {"FLOAT_TYPE", "float"}, {"FLOAT_TYPEV2", "vec2"}, {"ACC_TYPE", "float"}, {"USE_SUBGROUP_ADD_NO_SHMEM", "1"}}));
|
|
}
|
|
#endif
|
|
|
|
// Dequant shaders
|
|
if (tname != "f16" && tname != "bf16") {
|
|
string_to_spv("dequant_" + tname, "dequant_" + tname + ".comp", merge_maps(base_dict, {{data_a_key, "1"}, {"D_TYPE", "float16_t"}}));
|
|
}
|
|
// Fused dequant+transpose variant for FA quant-KV (per-head-contiguous f16 scratch).
|
|
if (tname == "q8_0") {
|
|
string_to_spv("dequant_" + tname + "_transpose", "dequant_" + tname + ".comp", merge_maps(base_dict, {{data_a_key, "1"}, {"D_TYPE", "float16_t"}, {"DEQUANT_TRANSPOSE", "1"}}));
|
|
}
|
|
|
|
shader = (tname == "f32" || tname == "f16" || tname == "bf16") ? "get_rows.comp" : "get_rows_quant.comp";
|
|
|
|
if (tname == "f16") {
|
|
string_to_spv("get_rows_" + tname, shader, merge_maps(base_dict, {{"TEMP_TYPE", "FLOAT_TYPE"}, {data_a_key, "1"}, {"B_TYPE", "int"}, {"D_TYPE", "float16_t"}, {"OPTIMIZATION_ERROR_WORKAROUND", "1"}}));
|
|
} else {
|
|
string_to_spv("get_rows_" + tname, shader, merge_maps(base_dict, {{"TEMP_TYPE", "FLOAT_TYPE"}, {data_a_key, "1"}, {"B_TYPE", "int"}, {"D_TYPE", "float16_t"}}));
|
|
}
|
|
string_to_spv("get_rows_" + tname + "_f32", shader, merge_maps(base_dict, {{"TEMP_TYPE", "FLOAT_TYPE"}, {data_a_key, "1"}, {"B_TYPE", "int"}, {"D_TYPE", "float"}}));
|
|
}
|
|
|
|
string_to_spv("get_rows_i32", "get_rows.comp", {{"TEMP_TYPE", "uint"}, {"A_TYPE", "uint"}, {"B_TYPE", "int"}, {"D_TYPE", "uint"}});
|
|
|
|
string_to_spv("mul_mat_vec_p021_f16_f32_subgroup_add", "mul_mat_vec_p021.comp", {{"A_TYPE", "float16_t"}, {"A_TYPEV4", "f16vec4"}, {"B_TYPE", "float"}, {"B_TYPEV4", "vec4"}, {"D_TYPE", "float"}, {"USE_SUBGROUP_ADD", "1"}});
|
|
string_to_spv("mul_mat_vec_p021_f16_f32", "mul_mat_vec_p021.comp", {{"A_TYPE", "float16_t"}, {"A_TYPEV4", "f16vec4"}, {"B_TYPE", "float"}, {"B_TYPEV4", "vec4"}, {"D_TYPE", "float"}});
|
|
string_to_spv("mul_mat_vec_nc_f16_f32", "mul_mat_vec_nc.comp", {{"A_TYPE", "float16_t"}, {"A_TYPEV4", "f16vec4"}, {"B_TYPE", "float"}, {"B_TYPEV4", "vec4"}, {"D_TYPE", "float"}});
|
|
|
|
// Norms
|
|
string_to_spv("norm_f32", "norm.comp", merge_maps(base_dict, {{"A_TYPE", "float"}, {"D_TYPE", "float"}}));
|
|
string_to_spv("group_norm_f32", "group_norm.comp", merge_maps(base_dict, {{"A_TYPE", "float"}, {"D_TYPE", "float"}}));
|
|
string_to_spv("rms_norm_f32", "rms_norm.comp", merge_maps(base_dict, {{"A_TYPE", "float"}, {"B_TYPE", "float"}, {"D_TYPE", "float"}}));
|
|
string_to_spv("rms_norm_mul_add_f32", "rms_norm.comp", merge_maps(base_dict, {{"A_TYPE", "float"}, {"B_TYPE", "float"}, {"D_TYPE", "float"}, {"RMS_NORM_ADD_FUSION", "1"}}));
|
|
string_to_spv("rms_norm_mul_add_partials_f32", "rms_norm_partials.comp", merge_maps(base_dict, {{"A_TYPE", "float"}, {"B_TYPE", "float"}, {"D_TYPE", "float"}, {"RMS_NORM_ADD_FUSION", "1"}}));
|
|
string_to_spv("rms_norm_set_rows_f32_f32", "rms_norm.comp", merge_maps(base_dict, {{"A_TYPE", "float"}, {"B_TYPE", "float"}, {"D_TYPE", "float"}, {"RMS_NORM_SET_ROWS_FUSION", "1"}}));
|
|
string_to_spv("rms_norm_set_rows_f32_f16", "rms_norm.comp", merge_maps(base_dict, {{"A_TYPE", "float"}, {"B_TYPE", "float"}, {"D_TYPE", "float16_t"}, {"RMS_NORM_SET_ROWS_FUSION", "1"}}));
|
|
string_to_spv("rms_norm_partials_f32", "rms_norm_partials.comp", merge_maps(base_dict, {{"A_TYPE", "float"}, {"B_TYPE", "float"}, {"D_TYPE", "float"}}));
|
|
string_to_spv("rms_norm_mul_rope_f32_f32", "rms_norm.comp", merge_maps(base_dict, {{"A_TYPE", "float"}, {"B_TYPE", "float"}, {"D_TYPE", "float"}, {"ROPE_D_TYPE", "float"}, {"RMS_NORM_ROPE_FUSION", "1"}}));
|
|
string_to_spv("rms_norm_mul_rope_f32_f16", "rms_norm.comp", merge_maps(base_dict, {{"A_TYPE", "float"}, {"B_TYPE", "float"}, {"D_TYPE", "float"}, {"ROPE_D_TYPE", "float16_t"}, {"RMS_NORM_ROPE_FUSION", "1"}}));
|
|
string_to_spv("rms_norm_back_f32", "rms_norm_back.comp", merge_maps(base_dict, {{"A_TYPE", "float"}, {"B_TYPE", "float"}, {"D_TYPE", "float"}}));
|
|
string_to_spv("l2_norm_f32", "l2_norm.comp", merge_maps(base_dict, {{"A_TYPE", "float"}, {"D_TYPE", "float"}}));
|
|
|
|
string_to_spv("cpy_f32_f32", "copy.comp", {{"A_TYPE", "float"}, {"D_TYPE", "float"}});
|
|
string_to_spv("cpy_f32_f16", "copy.comp", {{"A_TYPE", "float"}, {"D_TYPE", "float16_t"}});
|
|
string_to_spv("cpy_f16_f16", "copy.comp", {{"A_TYPE", "float16_t"}, {"D_TYPE", "float16_t"}, {"OPTIMIZATION_ERROR_WORKAROUND", "1"}});
|
|
string_to_spv("cpy_f16_f32", "copy.comp", {{"A_TYPE", "float16_t"}, {"D_TYPE", "float"}, {"OPTIMIZATION_ERROR_WORKAROUND", "1"}});
|
|
string_to_spv("cpy_f32_bf16","copy.comp", {{"A_TYPE", "float"}, {"D_TYPE", "uint16_t"}, {"DATA_D_BF16", "1"}});
|
|
string_to_spv("cpy_bf16_f32","copy.comp", {{"A_TYPE", "uint16_t"}, {"D_TYPE", "float"}, {"DATA_A_BF16", "1"}});
|
|
string_to_spv("contig_cpy_f32_f32", "contig_copy.comp", {{"A_TYPE", "float"}, {"D_TYPE", "float"}});
|
|
string_to_spv("contig_cpy_f32_i32", "contig_copy.comp", {{"A_TYPE", "float"}, {"D_TYPE", "int"}});
|
|
string_to_spv("contig_cpy_i32_f32", "contig_copy.comp", {{"A_TYPE", "int"}, {"D_TYPE", "float"}});
|
|
string_to_spv("contig_cpy_f32_f16", "contig_copy.comp", {{"A_TYPE", "float"}, {"D_TYPE", "float16_t"}});
|
|
string_to_spv("contig_cpy_f16_f16", "contig_copy.comp", {{"A_TYPE", "float16_t"}, {"D_TYPE", "float16_t"}, {"OPTIMIZATION_ERROR_WORKAROUND", "1"}});
|
|
string_to_spv("contig_cpy_f16_f32", "contig_copy.comp", {{"A_TYPE", "float16_t"}, {"D_TYPE", "float"}, {"OPTIMIZATION_ERROR_WORKAROUND", "1"}});
|
|
string_to_spv("contig_cpy_f32_bf16","contig_copy.comp",{{"A_TYPE", "float"}, {"D_TYPE", "uint16_t"}, {"DATA_D_BF16", "1"}});
|
|
string_to_spv("contig_cpy_bf16_f32","contig_copy.comp",{{"A_TYPE", "uint16_t"}, {"D_TYPE", "float"}, {"DATA_A_BF16", "1"}});
|
|
string_to_spv("cpy_f32_i32", "copy.comp", {{"A_TYPE", "float"}, {"D_TYPE", "int"}});
|
|
string_to_spv("cpy_i32_f32", "copy.comp", {{"A_TYPE", "int"}, {"D_TYPE", "float"}});
|
|
|
|
string_to_spv("cpy_transpose_16", "copy_transpose.comp", {{"A_TYPE", "uint16_t"}, {"D_TYPE", "uint16_t"}});
|
|
string_to_spv("cpy_transpose_32", "copy_transpose.comp", {{"A_TYPE", "uint"}, {"D_TYPE", "uint"}});
|
|
string_to_spv("cpy_transpose_02_16", "copy_transpose_02.comp", {{"A_TYPE", "uint16_t"}, {"D_TYPE", "uint16_t"}});
|
|
string_to_spv("cpy_transpose_02_32", "copy_transpose_02.comp", {{"A_TYPE", "uint"}, {"D_TYPE", "uint"}});
|
|
|
|
for (std::string t : {"q1_0", "q2_0", "q4_0", "q4_1", "q5_0", "q5_1", "q8_0", "iq4_nl"}) {
|
|
string_to_spv("cpy_f32_" + t, "copy_to_quant.comp", {{"DATA_A_" + to_uppercase(t), "1"}, {"S_TYPE", "float"}, {"D_TYPE", "float"}, {"FLOAT_TYPE", "float"}});
|
|
string_to_spv("cpy_" + t + "_f32", "copy_from_quant.comp", {{"DATA_A_" + to_uppercase(t), "1"}, {"D_TYPE", "float"}, {"FLOAT_TYPE", "float"}});
|
|
}
|
|
|
|
for (auto src : {std::pair{"f32", "float"}, std::pair{"f16", "float16_t"}}) {
|
|
for (std::string dst : {"f32", "f16", "bf16", "q1_0", "q2_0", "q4_0", "q4_1", "q5_0", "q5_1", "q8_0", "iq4_nl"}) {
|
|
string_to_spv("set_rows_" + std::string(src.first) + "_" + dst + "_i32", "copy_to_quant.comp", {{"SET_ROWS", "1"}, {"DATA_A_" + to_uppercase(dst), "1"}, {"B_TYPE", "uint"}, {"B_SIZE", "32"}, {"S_TYPE", src.second}, {"D_TYPE", "float"}, {"FLOAT_TYPE", "float"}});
|
|
string_to_spv("set_rows_" + std::string(src.first) + "_" + dst + "_i64", "copy_to_quant.comp", {{"SET_ROWS", "1"}, {"DATA_A_" + to_uppercase(dst), "1"}, {"B_TYPE", "uvec2"}, {"B_SIZE", "64"}, {"S_TYPE", src.second}, {"D_TYPE", "float"}, {"FLOAT_TYPE", "float"}});
|
|
}
|
|
}
|
|
|
|
auto get_type_str = [](bool f16) {
|
|
return f16 ? "float16_t" : "float";
|
|
};
|
|
auto get_suffix = [](bool src0_f16, bool src1_f16, bool dst_f16) {
|
|
std::string s;
|
|
s += std::string(src0_f16 ? "_f16" : "_f32");
|
|
s += std::string(src1_f16 ? "_f16" : "_f32");
|
|
s += std::string(dst_f16 ? "_f16" : "_f32");
|
|
return s;
|
|
};
|
|
for (std::string op : {"add", "sub", "mul", "div", "add_rms", }) {
|
|
for (auto src0_f16 : {false, true}) {
|
|
for (auto src1_f16 : {false, true}) {
|
|
for (auto dst_f16 : {false, true}) {
|
|
auto source = op == "add_rms" ? std::string("add") : op;
|
|
auto name = op + get_suffix(src0_f16, src1_f16, dst_f16);
|
|
auto add_rms = op == "add_rms" ? "1" : "0";
|
|
string_to_spv(name.c_str(), source + ".comp", {{"A_TYPE", get_type_str(src0_f16)}, {"B_TYPE", get_type_str(src1_f16)}, {"D_TYPE", get_type_str(dst_f16)}, {"FLOAT_TYPE", "float"}, {"ADD_RMS" , add_rms}});
|
|
}
|
|
}
|
|
}
|
|
}
|
|
|
|
string_to_spv("sub_f32", "sub.comp", {{"A_TYPE", "float"}, {"B_TYPE", "float"}, {"D_TYPE", "float"}, {"FLOAT_TYPE", "float"}});
|
|
|
|
string_to_spv("acc_f32", "acc.comp", {{"A_TYPE", "float"}, {"B_TYPE", "float"}, {"D_TYPE", "float"}, {"FLOAT_TYPE", "float"}});
|
|
|
|
string_to_spv("split_k_reduce", "mul_mat_split_k_reduce.comp", {});
|
|
string_to_spv("fa_split_k_reduce", "flash_attn_split_k_reduce.comp", {});
|
|
|
|
string_to_spv("fa_mask_opt", "flash_attn_mask_opt.comp", {});
|
|
|
|
string_to_spv("quantize_q8_1", "quantize_q8_1.comp", {});
|
|
string_to_spv("quantize_q8_1_subgroup", "quantize_q8_1.comp", {{"USE_SUBGROUPS", "1"}});
|
|
|
|
string_to_spv("quantize_q8_1_x4", "quantize_q8_1.comp", {{"QBLOCK_X4", "1"}});
|
|
string_to_spv("quantize_q8_1_x4_subgroup", "quantize_q8_1.comp", {{"QBLOCK_X4", "1"}, {"USE_SUBGROUPS", "1"}});
|
|
|
|
string_to_spv("mul_f32", "mul.comp", {{"A_TYPE", "float"}, {"B_TYPE", "float"}, {"D_TYPE", "float"}, {"FLOAT_TYPE", "float"}});
|
|
|
|
string_to_spv("div_f32", "div.comp", {{"A_TYPE", "float"}, {"B_TYPE", "float"}, {"D_TYPE", "float"}, {"FLOAT_TYPE", "float"}});
|
|
|
|
string_to_spv("repeat_i32", "repeat.comp", {{"A_TYPE", "int32_t"}, {"D_TYPE", "int32_t"}});
|
|
string_to_spv("repeat_back_f32", "repeat_back.comp", {{"A_TYPE", "float"}, {"D_TYPE", "float"}});
|
|
string_to_spv("get_rows_back_f32", "get_rows_back.comp", {{"A_TYPE", "float"}, {"B_TYPE", "int"}, {"D_TYPE", "float"}});
|
|
|
|
string_to_spv("repeat_i16", "repeat.comp", {{"A_TYPE", "int16_t"}, {"D_TYPE", "int16_t"}});
|
|
|
|
string_to_spv("scale_f32", "scale.comp", {{"A_TYPE", "float"}, {"D_TYPE", "float"}, {"FLOAT_TYPE", "float"}});
|
|
|
|
string_to_spv("pad_f32", "pad.comp", {{"A_TYPE", "float"}, {"D_TYPE", "float"}});
|
|
string_to_spv("pad_reflect_1d_f32", "pad_reflect_1d.comp", {{"A_TYPE", "float"}, {"D_TYPE", "float"}});
|
|
|
|
string_to_spv("concat_i8", "concat.comp", {{"A_TYPE", "uint8_t"}, {"B_TYPE", "uint8_t"}, {"D_TYPE", "uint8_t"}});
|
|
string_to_spv("concat_i16", "concat.comp", {{"A_TYPE", "uint16_t"}, {"B_TYPE", "uint16_t"}, {"D_TYPE", "uint16_t"}});
|
|
string_to_spv("concat_i32", "concat.comp", {{"A_TYPE", "uint"}, {"B_TYPE", "uint"}, {"D_TYPE", "uint"}});
|
|
string_to_spv("concat_i64", "concat.comp", {{"A_TYPE", "uvec2"}, {"B_TYPE", "uvec2"}, {"D_TYPE", "uvec2"}});
|
|
|
|
string_to_spv("upscale_f32", "upscale.comp", {{"A_TYPE", "float"}, {"B_TYPE", "float"}, {"D_TYPE", "float"}});
|
|
|
|
string_to_spv("exp_f16", "unary.comp", {{"A_TYPE", "float16_t"}, {"D_TYPE", "float16_t"}, {"OP", "op_exp"}});
|
|
string_to_spv("exp_f32", "unary.comp", {{"A_TYPE", "float"}, {"D_TYPE", "float"}, {"OP", "op_exp"}});
|
|
string_to_spv("expm1_f16", "unary.comp", {{"A_TYPE", "float16_t"}, {"D_TYPE", "float16_t"}, {"OP", "op_expm1"}});
|
|
string_to_spv("expm1_f32", "unary.comp", {{"A_TYPE", "float"}, {"D_TYPE", "float"}, {"OP", "op_expm1"}});
|
|
|
|
string_to_spv("log_f16", "log.comp", {{"A_TYPE", "float16_t"}, {"D_TYPE", "float16_t"}});
|
|
string_to_spv("log_f32", "log.comp", {{"A_TYPE", "float"}, {"D_TYPE", "float"}});
|
|
string_to_spv("gelu_f16", "unary.comp", {{"A_TYPE", "float16_t"}, {"D_TYPE", "float16_t"}, {"OP", "op_gelu"}});
|
|
string_to_spv("gelu_f32", "unary.comp", {{"A_TYPE", "float"}, {"D_TYPE", "float"}, {"OP", "op_gelu"}});
|
|
string_to_spv("gelu_erf_f16", "unary.comp", {{"A_TYPE", "float16_t"}, {"D_TYPE", "float16_t"}, {"OP", "op_gelu_erf"}});
|
|
string_to_spv("gelu_erf_f32", "unary.comp", {{"A_TYPE", "float"}, {"D_TYPE", "float"}, {"OP", "op_gelu_erf"}});
|
|
string_to_spv("gelu_quick_f16", "unary.comp", {{"A_TYPE", "float16_t"}, {"D_TYPE", "float16_t"}, {"OP", "op_gelu_quick"}});
|
|
string_to_spv("gelu_quick_f32", "unary.comp", {{"A_TYPE", "float"}, {"D_TYPE", "float"}, {"OP", "op_gelu_quick"}});
|
|
string_to_spv("silu_f16", "unary.comp", {{"A_TYPE", "float16_t"}, {"D_TYPE", "float16_t"}, {"OP", "op_silu"}});
|
|
string_to_spv("silu_f32", "unary.comp", {{"A_TYPE", "float"}, {"D_TYPE", "float"}, {"OP", "op_silu"}});
|
|
string_to_spv("relu_f16", "unary.comp", {{"A_TYPE", "float16_t"}, {"D_TYPE", "float16_t"}, {"OP", "op_relu"}});
|
|
string_to_spv("relu_f32", "unary.comp", {{"A_TYPE", "float"}, {"D_TYPE", "float"}, {"OP", "op_relu"}});
|
|
string_to_spv("sqr_f16", "unary.comp", {{"A_TYPE", "float16_t"}, {"D_TYPE", "float16_t"}, {"OP", "op_sqr"}});
|
|
string_to_spv("sqr_f32", "unary.comp", {{"A_TYPE", "float"}, {"D_TYPE", "float"}, {"OP", "op_sqr"}});
|
|
string_to_spv("sqrt_f16", "unary.comp", {{"A_TYPE", "float16_t"}, {"D_TYPE", "float16_t"}, {"OP", "op_sqrt"}});
|
|
string_to_spv("sqrt_f32", "unary.comp", {{"A_TYPE", "float"}, {"D_TYPE", "float"}, {"OP", "op_sqrt"}});
|
|
string_to_spv("sin_f16", "unary.comp", {{"A_TYPE", "float16_t"}, {"D_TYPE", "float16_t"}, {"OP", "op_sin"}});
|
|
string_to_spv("sin_f32", "unary.comp", {{"A_TYPE", "float"}, {"D_TYPE", "float"}, {"OP", "op_sin"}});
|
|
string_to_spv("cos_f16", "unary.comp", {{"A_TYPE", "float16_t"}, {"D_TYPE", "float16_t"}, {"OP", "op_cos"}});
|
|
string_to_spv("cos_f32", "unary.comp", {{"A_TYPE", "float"}, {"D_TYPE", "float"}, {"OP", "op_cos"}});
|
|
string_to_spv("clamp_f16", "unary.comp", {{"A_TYPE", "float16_t"}, {"D_TYPE", "float16_t"}, {"OP", "op_clamp"}});
|
|
string_to_spv("clamp_f32", "unary.comp", {{"A_TYPE", "float"}, {"D_TYPE", "float"}, {"OP", "op_clamp"}});
|
|
string_to_spv("leaky_relu_f16", "unary.comp", {{"A_TYPE", "float16_t"}, {"D_TYPE", "float16_t"}, {"OP", "op_leaky_relu"}});
|
|
string_to_spv("leaky_relu_f32", "unary.comp", {{"A_TYPE", "float"}, {"D_TYPE", "float"}, {"OP", "op_leaky_relu"}});
|
|
string_to_spv("neg_f16", "unary.comp", {{"A_TYPE", "float16_t"}, {"D_TYPE", "float16_t"}, {"OP", "op_neg"}});
|
|
string_to_spv("neg_f32", "unary.comp", {{"A_TYPE", "float"}, {"D_TYPE", "float"}, {"OP", "op_neg"}});
|
|
string_to_spv("tanh_f16", "unary.comp", {{"A_TYPE", "float16_t"}, {"D_TYPE", "float16_t"}, {"OP", "op_tanh"}});
|
|
string_to_spv("tanh_f32", "unary.comp", {{"A_TYPE", "float"}, {"D_TYPE", "float"}, {"OP", "op_tanh"}});
|
|
string_to_spv("sigmoid_f16", "unary.comp", {{"A_TYPE", "float16_t"}, {"D_TYPE", "float16_t"}, {"OP", "op_sigmoid"}});
|
|
string_to_spv("sigmoid_f32", "unary.comp", {{"A_TYPE", "float"}, {"D_TYPE", "float"}, {"OP", "op_sigmoid"}});
|
|
string_to_spv("hardsigmoid_f16","unary.comp", {{"A_TYPE", "float16_t"}, {"D_TYPE", "float16_t"}, {"OP", "op_hardsigmoid"}});
|
|
string_to_spv("hardsigmoid_f32","unary.comp", {{"A_TYPE", "float"}, {"D_TYPE", "float"}, {"OP", "op_hardsigmoid"}});
|
|
string_to_spv("hardswish_f16", "unary.comp", {{"A_TYPE", "float16_t"}, {"D_TYPE", "float16_t"}, {"OP", "op_hardswish"}});
|
|
string_to_spv("hardswish_f32", "unary.comp", {{"A_TYPE", "float"}, {"D_TYPE", "float"}, {"OP", "op_hardswish"}});
|
|
string_to_spv("abs_f16", "unary.comp", {{"A_TYPE", "float16_t"}, {"D_TYPE", "float16_t"}, {"OP", "op_abs"}});
|
|
string_to_spv("abs_f32", "unary.comp", {{"A_TYPE", "float"}, {"D_TYPE", "float"}, {"OP", "op_abs"}});
|
|
string_to_spv("elu_f16", "unary.comp", {{"A_TYPE", "float16_t"}, {"D_TYPE", "float16_t"}, {"OP", "op_elu"}});
|
|
string_to_spv("elu_f32", "unary.comp", {{"A_TYPE", "float"}, {"D_TYPE", "float"}, {"OP", "op_elu"}});
|
|
string_to_spv("xielu_f16", "unary.comp", {{"A_TYPE", "float16_t"}, {"D_TYPE", "float16_t"}, {"OP", "op_xielu"}});
|
|
string_to_spv("xielu_f32", "unary.comp", {{"A_TYPE", "float"}, {"D_TYPE", "float"}, {"OP", "op_xielu"}});
|
|
string_to_spv("sgn_f16", "unary.comp", {{"A_TYPE", "float16_t"}, {"D_TYPE", "float16_t"}, {"OP", "op_sgn"}});
|
|
string_to_spv("sgn_f32", "unary.comp", {{"A_TYPE", "float"}, {"D_TYPE", "float"}, {"OP", "op_sgn"}});
|
|
|
|
string_to_spv("tri_f16", "tri.comp", {{"A_TYPE", "float16_t"}, {"D_TYPE", "float16_t"}});
|
|
string_to_spv("tri_f32", "tri.comp", {{"A_TYPE", "float"}, {"D_TYPE", "float"}});
|
|
string_to_spv("diag_f16", "diag.comp", {{"A_TYPE", "float16_t"}, {"D_TYPE", "float16_t"}});
|
|
string_to_spv("diag_f32", "diag.comp", {{"A_TYPE", "float"}, {"D_TYPE", "float"}});
|
|
|
|
string_to_spv("softplus_f16", "unary.comp", {{"A_TYPE", "float16_t"}, {"D_TYPE", "float16_t"}, {"OP", "op_softplus"}});
|
|
string_to_spv("softplus_f32", "unary.comp", {{"A_TYPE", "float"}, {"D_TYPE", "float"}, {"OP", "op_softplus"}});
|
|
|
|
string_to_spv("gelu_mul_f32", "unary.comp", {{"A_TYPE", "float"}, {"B_TYPE", "float"}, {"D_TYPE", "float"}, {"FLOAT_TYPE", "float"}, {"OP", "op_gelu"}, {"UNARY_MUL_FUSION", "1"}});
|
|
string_to_spv("gelu_mul_f16", "unary.comp", {{"A_TYPE", "float16_t"}, {"B_TYPE", "float16_t"}, {"D_TYPE", "float16_t"}, {"FLOAT_TYPE", "float"}, {"OP", "op_gelu"}, {"UNARY_MUL_FUSION", "1"}});
|
|
string_to_spv("sigmoid_mul_f32", "unary.comp", {{"A_TYPE", "float"}, {"B_TYPE", "float"}, {"D_TYPE", "float"}, {"FLOAT_TYPE", "float"}, {"OP", "op_sigmoid"}, {"UNARY_MUL_FUSION", "1"}});
|
|
string_to_spv("sigmoid_mul_f16", "unary.comp", {{"A_TYPE", "float16_t"}, {"B_TYPE", "float16_t"}, {"D_TYPE", "float16_t"}, {"FLOAT_TYPE", "float"}, {"OP", "op_sigmoid"}, {"UNARY_MUL_FUSION", "1"}});
|
|
string_to_spv("silu_mul_f32", "unary.comp", {{"A_TYPE", "float"}, {"B_TYPE", "float"}, {"D_TYPE", "float"}, {"FLOAT_TYPE", "float"}, {"OP", "op_silu"}, {"UNARY_MUL_FUSION", "1"}});
|
|
string_to_spv("silu_mul_f16", "unary.comp", {{"A_TYPE", "float16_t"}, {"B_TYPE", "float16_t"}, {"D_TYPE", "float16_t"}, {"FLOAT_TYPE", "float"}, {"OP", "op_silu"}, {"UNARY_MUL_FUSION", "1"}});
|
|
string_to_spv("softplus_mul_f32","unary.comp", {{"A_TYPE", "float"}, {"B_TYPE", "float"}, {"D_TYPE", "float"}, {"FLOAT_TYPE", "float"}, {"OP", "op_softplus"}, {"UNARY_MUL_FUSION", "1"}});
|
|
string_to_spv("softplus_mul_f16","unary.comp", {{"A_TYPE", "float16_t"}, {"B_TYPE", "float16_t"}, {"D_TYPE", "float16_t"}, {"FLOAT_TYPE", "float"}, {"OP", "op_softplus"}, {"UNARY_MUL_FUSION", "1"}});
|
|
|
|
string_to_spv("add1_f16_f16", "add1.comp", {{"A_TYPE", "float16_t"}, {"B_TYPE", "float16_t"}, {"D_TYPE", "float16_t"}, {"FLOAT_TYPE", "float"}});
|
|
string_to_spv("add1_f16_f32", "add1.comp", {{"A_TYPE", "float16_t"}, {"B_TYPE", "float"}, {"D_TYPE", "float16_t"}, {"FLOAT_TYPE", "float"}});
|
|
string_to_spv("add1_f32_f32", "add1.comp", {{"A_TYPE", "float"}, {"B_TYPE", "float"}, {"D_TYPE", "float"}, {"FLOAT_TYPE", "float"}});
|
|
string_to_spv("arange_f32", "arange.comp", {{"A_TYPE", "float"}, {"D_TYPE", "float"}, {"FLOAT_TYPE", "float"}});
|
|
string_to_spv("fill_f32", "fill.comp", {{"D_TYPE", "float"}, {"FLOAT_TYPE", "float"}});
|
|
string_to_spv("fill_f16", "fill.comp", {{"D_TYPE", "float16_t"}, {"FLOAT_TYPE", "float"}});
|
|
string_to_spv("step_f16", "unary.comp", {{"A_TYPE", "float16_t"}, {"D_TYPE", "float16_t"}, {"OP", "op_step"}});
|
|
string_to_spv("step_f32", "unary.comp", {{"A_TYPE", "float"}, {"D_TYPE", "float"}, {"OP", "op_step"}});
|
|
string_to_spv("round_f16", "unary.comp", {{"A_TYPE", "float16_t"}, {"D_TYPE", "float16_t"}, {"OP", "op_round"}});
|
|
string_to_spv("round_f32", "unary.comp", {{"A_TYPE", "float"}, {"D_TYPE", "float"}, {"OP", "op_round"}});
|
|
string_to_spv("ceil_f16", "unary.comp", {{"A_TYPE", "float16_t"}, {"D_TYPE", "float16_t"}, {"OP", "op_ceil"}});
|
|
string_to_spv("ceil_f32", "unary.comp", {{"A_TYPE", "float"}, {"D_TYPE", "float"}, {"OP", "op_ceil"}});
|
|
string_to_spv("floor_f16", "unary.comp", {{"A_TYPE", "float16_t"}, {"D_TYPE", "float16_t"}, {"OP", "op_floor"}});
|
|
string_to_spv("floor_f32", "unary.comp", {{"A_TYPE", "float"}, {"D_TYPE", "float"}, {"OP", "op_floor"}});
|
|
string_to_spv("trunc_f16", "unary.comp", {{"A_TYPE", "float16_t"}, {"D_TYPE", "float16_t"}, {"OP", "op_trunc"}});
|
|
string_to_spv("trunc_f32", "unary.comp", {{"A_TYPE", "float"}, {"D_TYPE", "float"}, {"OP", "op_trunc"}});
|
|
|
|
string_to_spv("geglu_f16", "geglu.comp", {{"A_TYPE", "float16_t"}, {"D_TYPE", "float16_t"}});
|
|
string_to_spv("geglu_f32", "geglu.comp", {{"A_TYPE", "float"}, {"D_TYPE", "float"}});
|
|
string_to_spv("reglu_f16", "reglu.comp", {{"A_TYPE", "float16_t"}, {"D_TYPE", "float16_t"}});
|
|
string_to_spv("reglu_f32", "reglu.comp", {{"A_TYPE", "float"}, {"D_TYPE", "float"}});
|
|
string_to_spv("swiglu_f16", "swiglu.comp", {{"A_TYPE", "float16_t"}, {"D_TYPE", "float16_t"}});
|
|
string_to_spv("swiglu_f32", "swiglu.comp", {{"A_TYPE", "float"}, {"D_TYPE", "float"}});
|
|
string_to_spv("swiglu_oai_f16", "swiglu_oai.comp", {{"A_TYPE", "float16_t"}, {"D_TYPE", "float16_t"}});
|
|
string_to_spv("swiglu_oai_f32", "swiglu_oai.comp", {{"A_TYPE", "float"}, {"D_TYPE", "float"}});
|
|
string_to_spv("swiglu_clamp_f16", "swiglu_clamp.comp", {{"A_TYPE", "float16_t"}, {"D_TYPE", "float16_t"}});
|
|
string_to_spv("swiglu_clamp_f32", "swiglu_clamp.comp", {{"A_TYPE", "float"}, {"D_TYPE", "float"}});
|
|
string_to_spv("geglu_erf_f16", "geglu_erf.comp", {{"A_TYPE", "float16_t"}, {"D_TYPE", "float16_t"}});
|
|
string_to_spv("geglu_erf_f32", "geglu_erf.comp", {{"A_TYPE", "float"}, {"D_TYPE", "float"}});
|
|
string_to_spv("geglu_quick_f16","geglu_quick.comp", {{"A_TYPE", "float16_t"}, {"D_TYPE", "float16_t"}});
|
|
string_to_spv("geglu_quick_f32","geglu_quick.comp", {{"A_TYPE", "float"}, {"D_TYPE", "float"}});
|
|
|
|
string_to_spv("silu_back_f32", "silu_back.comp", {{"A_TYPE", "float"}, {"B_TYPE", "float"}, {"D_TYPE", "float"}});
|
|
|
|
string_to_spv("diag_mask_inf_f32", "diag_mask_inf.comp", {{"A_TYPE", "float"}, {"D_TYPE", "float"}});
|
|
|
|
string_to_spv("soft_max_f32", "soft_max.comp", merge_maps(base_dict, {{"A_TYPE", "float"}, {"B_TYPE", "float"}, {"D_TYPE", "float"}}));
|
|
string_to_spv("soft_max_f32_f16", "soft_max.comp", merge_maps(base_dict, {{"A_TYPE", "float"}, {"B_TYPE", "float16_t"}, {"D_TYPE", "float"}}));
|
|
string_to_spv("soft_max_back_f32", "soft_max_back.comp", merge_maps(base_dict, {{"A_TYPE", "float"}, {"B_TYPE", "float"}, {"D_TYPE", "float"}}));
|
|
|
|
string_to_spv("soft_max_large1_f32", "soft_max_large1.comp", merge_maps(base_dict, {{"A_TYPE", "float"}, {"B_TYPE", "float"}, {"D_TYPE", "float"}}));
|
|
string_to_spv("soft_max_large2_f32", "soft_max_large2.comp", merge_maps(base_dict, {{"A_TYPE", "float"}, {"B_TYPE", "float"}, {"D_TYPE", "float"}}));
|
|
string_to_spv("soft_max_large3_f32", "soft_max_large3.comp", merge_maps(base_dict, {{"A_TYPE", "float"}, {"B_TYPE", "float"}, {"D_TYPE", "float"}}));
|
|
string_to_spv("soft_max_large1_f32_f16", "soft_max_large1.comp", merge_maps(base_dict, {{"A_TYPE", "float"}, {"B_TYPE", "float16_t"}, {"D_TYPE", "float"}}));
|
|
string_to_spv("soft_max_large2_f32_f16", "soft_max_large2.comp", merge_maps(base_dict, {{"A_TYPE", "float"}, {"B_TYPE", "float16_t"}, {"D_TYPE", "float"}}));
|
|
string_to_spv("soft_max_large3_f32_f16", "soft_max_large3.comp", merge_maps(base_dict, {{"A_TYPE", "float"}, {"B_TYPE", "float16_t"}, {"D_TYPE", "float"}}));
|
|
|
|
string_to_spv("rope_norm_f32", "rope_norm.comp", {{"A_TYPE", "float"}, {"ROPE_D_TYPE", "float"}});
|
|
string_to_spv("rope_norm_f16", "rope_norm.comp", {{"A_TYPE", "float16_t"}, {"ROPE_D_TYPE", "float16_t"}});
|
|
string_to_spv("rope_norm_f32_f16", "rope_norm.comp", {{"A_TYPE", "float"}, {"ROPE_D_TYPE", "float16_t"}});
|
|
|
|
string_to_spv("rope_neox_f32", "rope_neox.comp", {{"A_TYPE", "float"}, {"ROPE_D_TYPE", "float"}});
|
|
string_to_spv("rope_neox_f16", "rope_neox.comp", {{"A_TYPE", "float16_t"}, {"ROPE_D_TYPE", "float16_t"}});
|
|
string_to_spv("rope_neox_f32_f16", "rope_neox.comp", {{"A_TYPE", "float"}, {"ROPE_D_TYPE", "float16_t"}});
|
|
|
|
string_to_spv("rope_multi_f32", "rope_multi.comp", {{"A_TYPE", "float"}, {"ROPE_D_TYPE", "float"}});
|
|
string_to_spv("rope_multi_f16", "rope_multi.comp", {{"A_TYPE", "float16_t"}, {"ROPE_D_TYPE", "float16_t"}});
|
|
string_to_spv("rope_multi_f32_f16", "rope_multi.comp", {{"A_TYPE", "float"}, {"ROPE_D_TYPE", "float16_t"}});
|
|
|
|
string_to_spv("rope_vision_f32", "rope_vision.comp", {{"A_TYPE", "float"}, {"ROPE_D_TYPE", "float"}});
|
|
string_to_spv("rope_vision_f16", "rope_vision.comp", {{"A_TYPE", "float16_t"}, {"ROPE_D_TYPE", "float16_t"}});
|
|
|
|
string_to_spv("argsort_f32", "argsort.comp", {{"A_TYPE", "float"}});
|
|
string_to_spv("argsort_large_f32", "argsort_large.comp", {{"A_TYPE", "float"}});
|
|
|
|
string_to_spv("topk_argsort_f32", "topk_argsort.comp", {{"A_TYPE", "float"}});
|
|
string_to_spv("topk_nary_search_f32", "topk_nary_search.comp", {{"A_TYPE", "float"}});
|
|
string_to_spv("topk_radix_select_f32", "topk_radix_select.comp", {{"A_TYPE", "float"}});
|
|
|
|
string_to_spv("argmax_f32", "argmax.comp", merge_maps(base_dict, {{"A_TYPE", "float"}, {"D_TYPE", "int"}}));
|
|
string_to_spv("sum_rows_f32", "sum_rows.comp", merge_maps(base_dict, {{"A_TYPE", "float"}, {"D_TYPE", "float"}}));
|
|
string_to_spv("cross_entropy_loss_f32", "cross_entropy_loss.comp", merge_maps(base_dict, {{"A_TYPE", "float"}, {"B_TYPE", "float"}, {"D_TYPE", "float"}}));
|
|
string_to_spv("cross_entropy_loss_back_f32", "cross_entropy_loss_back.comp", merge_maps(base_dict, {{"A_TYPE", "float"}, {"B_TYPE", "float"}, {"D_TYPE", "float"}}));
|
|
string_to_spv("fwht_f32", "fwht.comp", {});
|
|
string_to_spv("fwht_shmem_f32", "fwht.comp", {{"FWHT_SHMEM", "1"}});
|
|
string_to_spv("count_equal_i32", "count_equal.comp", merge_maps(base_dict, {{"A_TYPE", "int"}, {"B_TYPE", "int"}, {"D_TYPE", "int"}}));
|
|
string_to_spv("dsv4_hc_comb_f32", "dsv4_hc_comb.comp", {});
|
|
string_to_spv("dsv4_hc_pre_f32", "dsv4_hc_pre.comp", {});
|
|
string_to_spv("dsv4_hc_post_f32", "dsv4_hc_post.comp", {});
|
|
string_to_spv("cumsum_f32", "cumsum.comp", merge_maps(base_dict, {{"A_TYPE", "float"}, {"D_TYPE", "float"}}));
|
|
string_to_spv("cumsum_multipass1_f32", "cumsum_multipass1.comp", merge_maps(base_dict, {{"A_TYPE", "float"}, {"D_TYPE", "float"}}));
|
|
string_to_spv("cumsum_multipass2_f32", "cumsum_multipass2.comp", merge_maps(base_dict, {{"A_TYPE", "float"}, {"D_TYPE", "float"}}));
|
|
|
|
string_to_spv("count_experts", "count_experts.comp", merge_maps(base_dict, {{"A_TYPE", "uint"}, {"D_TYPE", "uint"}}));
|
|
string_to_spv("count_experts_subgroup", "count_experts.comp", merge_maps(base_dict, {{"A_TYPE", "uint"}, {"D_TYPE", "uint"}, {"USE_SUBGROUPS", "1"}}));
|
|
|
|
for (std::string dim_str : {"", "_3d"}) {
|
|
for (bool bda : {false, true}) {
|
|
std::string bda_str = bda ? "_bda" : "";
|
|
std::string bda_def = bda ? "1" : "0";
|
|
string_to_spv("im2col" + dim_str + "_f32" + bda_str, "im2col" + dim_str + ".comp", merge_maps(base_dict, {{"A_TYPE", "float"}, {"D_TYPE", "float"}, {"D_SIZE", "4"}, {"BDA", bda_def}}));
|
|
string_to_spv("im2col" + dim_str + "_f32_f16" + bda_str, "im2col" + dim_str + ".comp", merge_maps(base_dict, {{"A_TYPE", "float"}, {"D_TYPE", "float16_t"}, {"D_SIZE", "2"}, {"BDA", bda_def}}));
|
|
}
|
|
}
|
|
|
|
string_to_spv("out_prod_f32", "out_prod.comp", {});
|
|
|
|
string_to_spv("timestep_embedding_f32", "timestep_embedding.comp", merge_maps(base_dict, {{"A_TYPE", "float"}, {"D_TYPE", "float"}}));
|
|
|
|
string_to_spv("conv_transpose_1d_f32", "conv_transpose_1d.comp", {{"A_TYPE", "float"}, {"B_TYPE", "float"}, {"D_TYPE", "float"}});
|
|
string_to_spv("col2im_1d_f32", "col2im_1d.comp", {{"DATA_A_F32", "1"}, {"A_TYPE", "float"}, {"D_TYPE", "float"}});
|
|
string_to_spv("col2im_1d_f16", "col2im_1d.comp", {{"DATA_A_F16", "1"}, {"A_TYPE", "float16_t"}, {"D_TYPE", "float16_t"}});
|
|
string_to_spv("col2im_1d_bf16", "col2im_1d.comp", {{"DATA_A_BF16", "1"}, {"A_TYPE", "uint16_t"}, {"D_TYPE", "uint16_t"}});
|
|
|
|
string_to_spv("snake_f32", "snake.comp", {{"DATA_A_F32", "1"}, {"A_TYPE", "float"}, {"D_TYPE", "float"}});
|
|
string_to_spv("snake_f16", "snake.comp", {{"DATA_A_F16", "1"}, {"A_TYPE", "float16_t"}, {"D_TYPE", "float16_t"}});
|
|
string_to_spv("snake_bf16", "snake.comp", {{"DATA_A_BF16", "1"}, {"DATA_D_BF16", "1"}, {"A_TYPE", "uint16_t"}, {"D_TYPE", "uint16_t"}});
|
|
|
|
string_to_spv("pool1d_f32", "pool1d.comp", merge_maps(base_dict, {{"A_TYPE", "float"}, {"D_TYPE", "float"}}));
|
|
string_to_spv("pool2d_f32", "pool2d.comp", merge_maps(base_dict, {{"A_TYPE", "float"}, {"D_TYPE", "float"}}));
|
|
|
|
string_to_spv("rwkv_wkv6_f32", "wkv6.comp", merge_maps(base_dict, {{"A_TYPE", "float"}}));
|
|
|
|
string_to_spv("gated_linear_attn_f32", "gla.comp", merge_maps(base_dict, {{"A_TYPE", "float"}}));
|
|
|
|
// Compile IQ4_NL support in so its shared LUT is available when K uses it.
|
|
// K quant type is selected at runtime via the FaTypeK spec constant.
|
|
std::map<std::string, std::string> li_dict = {{"FLOAT_TYPE", "float"}, {"FLOAT_TYPEV4", "vec4"}, {"DATA_A_IQ4_NL", "1"}};
|
|
string_to_spv("lightning_indexer_f32", "lightning_indexer.comp", li_dict);
|
|
string_to_spv("lightning_indexer_subgroup_f32", "lightning_indexer.comp", merge_maps(li_dict, {{"USE_SUBGROUP_ADD", "1"}}));
|
|
|
|
string_to_spv("rwkv_wkv7_f32", "wkv7.comp", merge_maps(base_dict, {{"A_TYPE", "float"}}));
|
|
|
|
string_to_spv("gated_delta_net_f32", "gated_delta_net.comp", merge_maps(base_dict, {{"FLOAT_TYPE", "float"}, {"USE_SUBGROUP_ADD", "1"}, {"USE_SUBGROUP_CLUSTERED", "1"}}));
|
|
string_to_spv("gated_delta_net_f32_nocluster", "gated_delta_net.comp", merge_maps(base_dict, {{"FLOAT_TYPE", "float"}, {"USE_SUBGROUP_ADD", "1"}, {"USE_SUBGROUP_CLUSTERED", "0"}}));
|
|
string_to_spv("gated_delta_net_f32_shmem", "gated_delta_net.comp", merge_maps(base_dict, {{"FLOAT_TYPE", "float"}, {"USE_SUBGROUP_ADD", "0"}, {"USE_SUBGROUP_CLUSTERED", "0"}}));
|
|
|
|
string_to_spv("opt_step_adamw_f32", "opt_step_adamw.comp", merge_maps(base_dict, {{"A_TYPE", "float"}}));
|
|
string_to_spv("opt_step_sgd_f32", "opt_step_sgd.comp", merge_maps(base_dict, {{"A_TYPE", "float"}}));
|
|
|
|
string_to_spv("solve_tri_f32", "solve_tri.comp", merge_maps(base_dict, {{"A_TYPE", "float"}, {"B_TYPE", "float"}, {"D_TYPE", "float"}}));
|
|
|
|
for (auto transpose : {false, true}) {
|
|
for (auto unroll : {false, true}) {
|
|
for (auto a_f16 : {false, true}) {
|
|
std::map<std::string, std::string> defines = {
|
|
{"A_TYPE", a_f16 ? "float16_t" : "float"}, {"B_TYPE", "float"}, {"D_TYPE", "float"},
|
|
{"USE_COLLECTIVES", "1"}, {"UNROLL", unroll ? "[[unroll]]" : ""},
|
|
};
|
|
if (transpose) defines["TRANSPOSE"] = "1";
|
|
std::string name = std::string(transpose ? "conv_transpose_2d": "conv2d")
|
|
+ (a_f16 ? "_f16" : "") + "_f32";
|
|
string_to_spv(name + (unroll ? "_unroll" : ""), "conv2d_mm.comp", defines);
|
|
#if defined(GGML_VULKAN_COOPMAT2_GLSLC_SUPPORT)
|
|
if (unroll) {
|
|
auto cm2_defines = defines;
|
|
cm2_defines["COOPMAT2"] = "1";
|
|
string_to_spv(name, "conv2d_mm.comp", cm2_defines, true, false, true);
|
|
}
|
|
#endif
|
|
#if defined(GGML_VULKAN_COOPMAT_GLSLC_SUPPORT)
|
|
if (unroll) {
|
|
auto cm1_defines = defines;
|
|
cm1_defines["COOPMAT"] = "1";
|
|
string_to_spv(name, "conv2d_mm.comp", cm1_defines, true, true, false);
|
|
}
|
|
#endif
|
|
}
|
|
}
|
|
}
|
|
|
|
for (auto unroll : {false, true}) {
|
|
for (auto a_f16 : {false, true}) {
|
|
std::map<std::string, std::string> defines = {
|
|
{"A_TYPE", a_f16 ? "float16_t" : "float"}, {"B_TYPE", "float"}, {"D_TYPE", "float"},
|
|
{"UNROLL", unroll ? "[[unroll]]" : ""},
|
|
};
|
|
std::string name = std::string("conv3d") + (a_f16 ? "_f16" : "") + "_f32";
|
|
string_to_spv(name + (unroll ? "_unroll" : ""), "conv3d_mm.comp", defines);
|
|
#if defined(GGML_VULKAN_COOPMAT2_GLSLC_SUPPORT)
|
|
if (unroll) {
|
|
auto cm2_defines = defines;
|
|
cm2_defines["COOPMAT2"] = "1";
|
|
string_to_spv(name, "conv3d_mm.comp", cm2_defines, true, false, true);
|
|
}
|
|
#endif
|
|
#if defined(GGML_VULKAN_COOPMAT_GLSLC_SUPPORT)
|
|
if (unroll) {
|
|
auto cm1_defines = defines;
|
|
cm1_defines["COOPMAT"] = "1";
|
|
string_to_spv(name, "conv3d_mm.comp", cm1_defines, true, true, false);
|
|
}
|
|
#endif
|
|
}
|
|
}
|
|
|
|
string_to_spv("conv2d_dw_whcn_f32", "conv2d_dw.comp", merge_maps(base_dict, {{"A_TYPE", "float"}, {"B_TYPE", "float"}, {"D_TYPE", "float"}, {"WHCN", "1"}}));
|
|
string_to_spv("conv2d_dw_cwhn_f32", "conv2d_dw.comp", merge_maps(base_dict, {{"A_TYPE", "float"}, {"B_TYPE", "float"}, {"D_TYPE", "float"}, {"CWHN", "1"}}));
|
|
string_to_spv("conv2d_dw_whcn_f16_f32", "conv2d_dw.comp", merge_maps(base_dict, {{"A_TYPE", "float16_t"}, {"B_TYPE", "float"}, {"D_TYPE", "float"}, {"WHCN", "1"}}));
|
|
string_to_spv("conv2d_dw_cwhn_f16_f32", "conv2d_dw.comp", merge_maps(base_dict, {{"A_TYPE", "float16_t"}, {"B_TYPE", "float"}, {"D_TYPE", "float"}, {"CWHN", "1"}}));
|
|
|
|
string_to_spv("roll_f32", "roll.comp", merge_maps(base_dict, {{"A_TYPE", "float"}, {"D_TYPE", "float"}}));
|
|
|
|
string_to_spv("add_id_f32", "add_id.comp", merge_maps(base_dict, {{"A_TYPE", "float"}, {"B_TYPE", "float"}, {"D_TYPE", "float"}}));
|
|
|
|
string_to_spv("multi_add_f32", "multi_add.comp", {{"A_TYPE", "float"}, {"B_TYPE", "float"}, {"D_TYPE", "float"}, {"FLOAT_TYPE", "float"}, {"ADD_RMS" , "0"}});
|
|
string_to_spv("multi_add_rms_f32", "multi_add.comp", {{"A_TYPE", "float"}, {"B_TYPE", "float"}, {"D_TYPE", "float"}, {"FLOAT_TYPE", "float"}, {"ADD_RMS" , "1"}});
|
|
|
|
string_to_spv("ssm_scan_f32", "ssm_scan.comp", {{"A_TYPE", "float"}});
|
|
string_to_spv("ssm_scan_subgroup_f32", "ssm_scan.comp", {{"A_TYPE", "float"}, {"USE_SUBGROUP_ADD", "1"}});
|
|
|
|
string_to_spv("ssm_conv_f32", "ssm_conv.comp", {{"A_TYPE", "float"}});
|
|
|
|
string_to_spv("topk_moe_f32", "topk_moe.comp", {});
|
|
|
|
for (auto &c : compiles) {
|
|
c.wait();
|
|
}
|
|
}
|
|
|
|
void write_output_files() {
|
|
std::stringstream hdr = make_generic_stringstream();
|
|
std::stringstream src = make_generic_stringstream();
|
|
|
|
hdr << "#include <cstdint>\n\n";
|
|
src << "#include \"" << basename(target_hpp) << "\"\n\n";
|
|
|
|
std::sort(shader_fnames.begin(), shader_fnames.end());
|
|
for (const auto& pair : shader_fnames) {
|
|
const std::string& name = pair.first;
|
|
#ifdef _WIN32
|
|
std::string path = pair.second;
|
|
std::replace(path.begin(), path.end(), '/', '\\' );
|
|
#else
|
|
const std::string& path = pair.second;
|
|
#endif
|
|
|
|
hdr << "extern const uint64_t " << name << "_len;\n";
|
|
hdr << "extern const unsigned char " << name << "_data[];\n\n";
|
|
|
|
if (input_filepath != "") {
|
|
std::string data = read_binary_file(path);
|
|
if (data.empty()) {
|
|
continue;
|
|
}
|
|
|
|
src << "const uint64_t " << name << "_len = " << data.size() << ";\n";
|
|
src << "const unsigned char " << name << "_data[" << data.size() << "] = {\n" << std::hex;
|
|
auto bytes = reinterpret_cast<const uint8_t*>(data.data());
|
|
for (size_t i = 0; i < data.size(); ++i) {
|
|
src << "0x" << static_cast<int>(bytes[i]) << ",";
|
|
if ((i + 1) % 12 == 0) src << "\n";
|
|
}
|
|
src << std::dec << "\n};\n\n";
|
|
}
|
|
}
|
|
|
|
std::string suffixes[2] = {"_f32", "_f16"};
|
|
for (std::string op : {"add", "sub", "mul", "div", "add_rms"}) {
|
|
hdr << "extern const void * " << op << "_data[2][2][2];\n";
|
|
hdr << "extern const uint64_t " << op << "_len[2][2][2];\n";
|
|
|
|
std::string op_file = op == "add_rms" ? "add.comp" : std::string(op) + ".comp";
|
|
if (basename(input_filepath) != op_file) {
|
|
continue;
|
|
}
|
|
std::stringstream data = make_generic_stringstream();
|
|
std::stringstream len = make_generic_stringstream();
|
|
data << "const void * " << op << "_data[2][2][2] = ";
|
|
len << "const uint64_t " << op << "_len[2][2][2] = ";
|
|
for (uint32_t t0 = 0; t0 < 2; ++t0) {
|
|
if (t0 == 0) {
|
|
data << "{";
|
|
len << "{";
|
|
}
|
|
for (uint32_t t1 = 0; t1 < 2; ++t1) {
|
|
if (t1 == 0) {
|
|
data << "{";
|
|
len << "{";
|
|
}
|
|
for (uint32_t t2 = 0; t2 < 2; ++t2) {
|
|
if (t2 == 0) {
|
|
data << "{";
|
|
len << "{";
|
|
}
|
|
data << op << suffixes[t0] << suffixes[t1] << suffixes[t2];
|
|
len << op << suffixes[t0] << suffixes[t1] << suffixes[t2];
|
|
data << "_data,";
|
|
len << "_len,";
|
|
if (t2 == 1) {
|
|
data << "}, ";
|
|
len << "}, ";
|
|
}
|
|
}
|
|
if (t1 == 1) {
|
|
data << "}, ";
|
|
len << "}, ";
|
|
}
|
|
}
|
|
if (t0 == 1) {
|
|
data << "};\n";
|
|
len << "};\n";
|
|
}
|
|
}
|
|
src << data.str();
|
|
src << len.str();
|
|
}
|
|
|
|
std::vector<std::string> btypes = {"f16", "f32"};
|
|
|
|
#if defined(GGML_VULKAN_INTEGER_DOT_GLSLC_SUPPORT)
|
|
btypes.push_back("q8_1");
|
|
#endif
|
|
|
|
for (const std::string& btype : btypes) {
|
|
for (const auto& tname : type_names) {
|
|
if (btype == "q8_1" && !is_legacy_quant(tname) && tname != "mxfp4" && !is_k_quant(tname) && tname != "iq1_s" && tname != "iq1_m") {
|
|
continue;
|
|
}
|
|
hdr << "extern const void * arr_dmmv_" << tname << "_" << btype << "_f32_data[3];\n";
|
|
hdr << "extern const uint64_t arr_dmmv_" << tname << "_" << btype << "_f32_len[3];\n";
|
|
if (basename(input_filepath) == "mul_mat_vec.comp") {
|
|
src << "const void * arr_dmmv_" << tname << "_" << btype << "_f32_data[3] = {mul_mat_vec_" << tname << "_" << btype << "_f32_data, mul_mat_vec_" << tname << "_" << btype << "_f32_subgroup_data, mul_mat_vec_" << tname << "_" << btype << "_f32_subgroup_no_shmem_data};\n";
|
|
src << "const uint64_t arr_dmmv_" << tname << "_" << btype << "_f32_len[3] = {mul_mat_vec_" << tname << "_" << btype << "_f32_len, mul_mat_vec_" << tname << "_" << btype << "_f32_subgroup_len, mul_mat_vec_" << tname << "_" << btype << "_f32_subgroup_no_shmem_len};\n";
|
|
}
|
|
|
|
if (btype == "f16") {
|
|
continue;
|
|
}
|
|
hdr << "extern const void * arr_dmmv_id_" << tname << "_" << btype << "_f32_data[3];\n";
|
|
hdr << "extern const uint64_t arr_dmmv_id_" << tname << "_" << btype << "_f32_len[3];\n";
|
|
if (basename(input_filepath) == "mul_mat_vec.comp") {
|
|
src << "const void * arr_dmmv_id_" << tname << "_" << btype << "_f32_data[3] = {mul_mat_vec_id_" << tname << "_" << btype << "_f32_data, mul_mat_vec_id_" << tname << "_" << btype << "_f32_subgroup_data, mul_mat_vec_id_" << tname << "_" << btype << "_f32_subgroup_no_shmem_data};\n";
|
|
src << "const uint64_t arr_dmmv_id_" << tname << "_" << btype << "_f32_len[3] = {mul_mat_vec_id_" << tname << "_" << btype << "_f32_len, mul_mat_vec_id_" << tname << "_" << btype << "_f32_subgroup_len, mul_mat_vec_id_" << tname << "_" << btype << "_f32_subgroup_no_shmem_len};\n";
|
|
}
|
|
}
|
|
}
|
|
|
|
#if defined(GGML_VULKAN_FLOAT_E2M1_GLSLC_SUPPORT) && defined(GGML_VULKAN_FLOAT_E4M3_GLSLC_SUPPORT)
|
|
for (const std::string& btype : {"f16", "f32"}) {
|
|
for (const std::string& tname : {"mxfp4", "nvfp4"}) {
|
|
hdr << "extern const void * arr_dmmv_" << tname << "_" << btype << "_f32_ocp_data[3];\n";
|
|
hdr << "extern const uint64_t arr_dmmv_" << tname << "_" << btype << "_f32_ocp_len[3];\n";
|
|
if (basename(input_filepath) == "mul_mat_vec.comp") {
|
|
src << "const void * arr_dmmv_" << tname << "_" << btype << "_f32_ocp_data[3] = {mul_mat_vec_" << tname << "_" << btype << "_f32_ocp_data, mul_mat_vec_" << tname << "_" << btype << "_f32_ocp_subgroup_data, mul_mat_vec_" << tname << "_" << btype << "_f32_ocp_subgroup_no_shmem_data};\n";
|
|
src << "const uint64_t arr_dmmv_" << tname << "_" << btype << "_f32_ocp_len[3] = {mul_mat_vec_" << tname << "_" << btype << "_f32_ocp_len, mul_mat_vec_" << tname << "_" << btype << "_f32_ocp_subgroup_len, mul_mat_vec_" << tname << "_" << btype << "_f32_ocp_subgroup_no_shmem_len};\n";
|
|
}
|
|
}
|
|
}
|
|
for (const std::string& tname : {"mxfp4", "nvfp4"}) {
|
|
hdr << "extern const void * arr_dmmv_id_" << tname << "_f32_f32_ocp_data[3];\n";
|
|
hdr << "extern const uint64_t arr_dmmv_id_" << tname << "_f32_f32_ocp_len[3];\n";
|
|
if (basename(input_filepath) == "mul_mat_vec.comp") {
|
|
src << "const void * arr_dmmv_id_" << tname << "_f32_f32_ocp_data[3] = {mul_mat_vec_id_" << tname << "_f32_f32_ocp_data, mul_mat_vec_id_" << tname << "_f32_f32_ocp_subgroup_data, mul_mat_vec_id_" << tname << "_f32_f32_ocp_subgroup_no_shmem_data};\n";
|
|
src << "const uint64_t arr_dmmv_id_" << tname << "_f32_f32_ocp_len[3] = {mul_mat_vec_id_" << tname << "_f32_f32_ocp_len, mul_mat_vec_id_" << tname << "_f32_f32_ocp_subgroup_len, mul_mat_vec_id_" << tname << "_f32_f32_ocp_subgroup_no_shmem_len};\n";
|
|
}
|
|
}
|
|
#endif
|
|
|
|
if (input_filepath == "") {
|
|
write_file_if_changed(target_hpp, hdr.str());
|
|
}
|
|
if (target_cpp != "") {
|
|
write_binary_file(target_cpp, src.str());
|
|
}
|
|
}
|
|
|
|
} // namespace
|
|
|
|
int main(int argc, char** argv) {
|
|
std::map<std::string, std::string> args;
|
|
for (int i = 1; i < argc; ++i) {
|
|
std::string arg = argv[i];
|
|
if (arg.rfind("--", 0) == 0) {
|
|
if (i + 1 < argc && argv[i + 1][0] != '-') {
|
|
args[arg] = argv[i + 1];
|
|
++i;
|
|
} else {
|
|
args[arg] = "";
|
|
}
|
|
}
|
|
}
|
|
|
|
if (args.find("--glslc") != args.end()) {
|
|
GLSLC = args["--glslc"]; // Path to glslc
|
|
}
|
|
if (args.find("--source") != args.end()) {
|
|
input_filepath = args["--source"]; // The shader source file to compile
|
|
}
|
|
if (args.find("--output-dir") != args.end()) {
|
|
output_dir = args["--output-dir"]; // Directory for containing SPIR-V output
|
|
}
|
|
if (args.find("--target-hpp") != args.end()) {
|
|
target_hpp = args["--target-hpp"]; // Path to generated header file
|
|
}
|
|
if (args.find("--target-cpp") != args.end()) {
|
|
target_cpp = args["--target-cpp"]; // Path to generated cpp file
|
|
}
|
|
|
|
if (!directory_exists(output_dir)) {
|
|
if (!create_directory(output_dir)) {
|
|
std::cerr << "Error creating output directory: " << output_dir << "\n";
|
|
return EXIT_FAILURE;
|
|
}
|
|
}
|
|
|
|
process_shaders();
|
|
|
|
if (compile_failed) {
|
|
std::cerr << "vulkan-shaders-gen: one or more shaders failed to compile" << std::endl;
|
|
return EXIT_FAILURE;
|
|
}
|
|
|
|
write_output_files();
|
|
|
|
return EXIT_SUCCESS;
|
|
}
|