在我自己的 Mac 上快速高效地运行 LLM 仅 2 MB

curl -sSf https://raw.githubusercontent.com/WasmEdge/WasmEdge/master/utils/install.sh | bash -s -- --plugin wasi_nn-ggml

curl -LO https://github.com/second-state/llama-utils/raw/main/chat/llama-chat.wasm

curl -LO https://huggingface.co/wasmedge/llama2/resolve/main/llama-2-7b-chat-q5_k_m.gguf

wasmedge --dir .:. --nn-preload default:GGML:AUTO:llama-2-7b-chat-q5_k_m.gguf llama-chat.wasm

Options: -m, --model-alias <ALIAS> Model alias [default: default] -c, --ctx-size <CTX_SIZE> Size of the prompt context [default: 4096] -n, --n-predict <N_PRDICT> Number of tokens to predict [default: 1024] -g, --n-gpu-layers <N_GPU_LAYERS> Number of layers to run on the GPU [default: 100] -b, --batch-size <BATCH_SIZE> Batch size for prompt processing [default: 4096] -r, --reverse-prompt <REVERSE_PROMPT> Halt generation at PROMPT, return control. -s, --system-prompt <SYSTEM_PROMPT> System prompt message string [default: "[Default system message
 for the prompt template]"] -p, --prompt-template <TEMPLATE> Prompt template. [default: llama-2-chat] [possible values: llama-2-chat, codellama-instruct, mistral-instruct-v0.1, mistrallite, openchat, belle-llama-2-chat, vicuna-chat, chatml] --log-prompts Print prompt strings to stdout --log-stat Print statistics to stdout --log-all Print all log information to stdout --stream-stdout Print the output to stdout in the streaming way -h, --help Print help

wasmedge --dir .:. --nn-preload default:GGML:AUTO:llama-2-7b-chat-q5_k_m.gguf \ llama-chat.wasm -c 2048 -n 512 --log-stat --stream-stdout
[USER]:Who is the "father of the atomic bomb"?（谁是“原子弹之父”?）
---------------- [LOG: STATISTICS] -----------------
llama_new_context_with_model: n_ctx = 2048llama_new_context_with_model: freq_base = 10000.0llama_new_context_with_model: freq_scale = 1llama_new_context_with_model: kv self size = 1024.00 MBllama_new_context_with_model: compute buffer total size = 630.14 MBllama_new_context_with_model: max tensor size = 102.54 MB[2023-11-10 17:52:12.768] [info] [WASI-NN] GGML backend: llama_system_info: AVX = 0 | AVX2 = 0 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 0 | NEON = 1 | ARM_FMA = 1 | F16C = 0 | FP16_VA = 1 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 0 | SSSE3 = 0 | VSX = 0 | The "father of the atomic bomb" is a term commonly associated with physicist J. Robert Oppenheimer. Oppenheimer was the director of the Manhattan Project, the secret research and development project that produced the atomic bomb during World War II. He is widely recognized as the leading figure in the development of the atomic bomb and is often referred to as the "father of the atomic bomb."llama_print_timings: load time = 15643.70 msllama_print_timings: sample time = 2.60 ms / 83 runs ( 0.03 ms per token, 31886.29 tokens per second)llama_print_timings: prompt eval time = 7836.72 ms / 54 tokens ( 145.12 ms per token, 6.89 tokens per second)llama_print_timings: eval time = 3198.24 ms / 82 runs ( 39.00 ms per token, 25.64 tokens per second)llama_print_timings: total time = 18852.93 ms
----------------------------------------------------

wasmedge --dir .:. --nn-preload default:GGML:AUTO:llama-2-7b-chat-q5_k_m.gguf \ llama-chat.wasm -c 2048 -n 512 --log-stat
[USER]:Who is the "father of the atomic bomb"? （谁是“原子弹之父”?）
---------------- [LOG: STATISTICS] -----------------llm_load_tensors: using CUDA for GPU accelerationllm_load_tensors: mem required = 86.04 MBllm_load_tensors: offloading 32 repeating layers to GPUllm_load_tensors: offloading non-repeating layers to GPUllm_load_tensors: offloaded 35/35 layers to GPUllm_load_tensors: VRAM used: 4474.93 MB..................................................................................................llama_new_context_with_model: n_ctx = 2048llama_new_context_with_model: freq_base = 10000.0llama_new_context_with_model: freq_scale = 1llama_kv_cache_init: offloading v cache to GPUllama_kv_cache_init: offloading k cache to GPUllama_kv_cache_init: VRAM kv self = 1024.00 MBllama_new_context_with_model: kv self size = 1024.00 MBllama_new_context_with_model: compute buffer total size = 630.14 MBllama_new_context_with_model: VRAM scratch buffer: 624.02 MBllama_new_context_with_model: total VRAM used: 6122.95 MB (model: 4474.93 MB, context: 1648.02 MB)[2023-11-11 00:02:22.402] [info] [WASI-NN] GGML backend: llama_system_info: AVX = 1 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 |
llama_print_timings: load time = 2601.44 msllama_print_timings: sample time = 2.63 ms / 84 runs ( 0.03 ms per token, 31987.81 tokens per second)llama_print_timings: prompt eval time = 203.90 ms / 54 tokens ( 3.78 ms per token, 264.84 tokens per second)llama_print_timings: eval time = 1641.84 ms / 83 runs ( 19.78 ms per token, 50.55 tokens per second)llama_print_timings: total time = 4254.95 ms
----------------------------------------------------
[ASSISTANT]:The "father of the atomic bomb" is a term commonly associated with physicist J. Robert Oppenheimer. Oppenheimer was the director of the Manhattan Project, the secret research and development project that produced the first atomic bomb during World War II. He is widely recognized as the leading figure in the development of the atomic bomb and is often referred to as the "father of the atomic bomb."（“原子弹之父”这个词通常与物理学家 J. Robert Oppenheimer 联系在一起。奥本海默是曼哈顿计划的主管,曼哈顿计划是二战期间研发出第一颗原子弹的秘密研发项目。他被广泛认为是原子弹研发的领导人物,常被称为“原子弹之父”。）

fn main() { let args: Vec<String> = env::args().collect(); let model_name: &str = &args[1];
 let graph = wasi_nn::GraphBuilder::new(wasi_nn::GraphEncoding::Ggml, wasi_nn::ExecutionTarget::AUTO) .build_from_cache(model_name) .unwrap(); let mut context = graph.init_execution_context().unwrap();
 let system_prompt = String::from("<<SYS>>You are a helpful, respectful and honest assistant. Always answer as short as possible, while being safe. <</SYS>>"); let mut saved_prompt = String::new();
 loop { println!("Question:"); let input = read_input(); if saved_prompt == "" { saved_prompt = format!("[INST] {} {} [/INST]", system_prompt, input.trim()); } else { saved_prompt = format!("{} [INST] {} [/INST]", saved_prompt, input.trim()); }
 // 将用户问题字符串预处理成张量格式后,设置为模型输入张量,以供模型进行下游推理计算。 let tensor_data = saved_prompt.as_bytes().to_vec(); context .set_input(0, wasi_nn::TensorType::U8, &[1], &tensor_data) .unwrap();
 // 执行推理计算 context.compute().unwrap();
 // 获取输出 let mut output_buffer = vec![0u8; 1000]; let output_size = context.get_output(0, &mut output_buffer).unwrap(); let output = String::from_utf8_lossy(&output_buffer[..output_size]).to_string(); println!("Answer:\n{}", output.trim());
 saved_prompt = format!("{} {} ", saved_prompt, output.trim()); }}

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | shrustup target add wasm32-wasi

# 克隆源项目git clone https://github.com/second-state/llama-utilscd llama-utils/chat/
# 构建cargo build --target wasm32-wasi --release
# 结果 wasm 文件cp target/wasm32-wasi/release/llama-chat.wasm .