vitisai llm new

[xieofxie]

Tested

• AMD Ryzen AI 9 HX 370 w/ Radeon 890M
  ○ Microsoft.WinML.AMD.NPU.EP.2_0.0.62.0_x64__8wekyb3d8bbwe
  ○ onnxruntime-genai-winml==0.13.2
• onnxruntime-windowsml==1.24.5.202604171637

# Load the chat template from chat_template.jinja in the model folder
with open(os.path.join(model_folder, "chat_template.jinja"), "r", encoding="utf-8") as f:
    template_str = f.read()

# Apply the chat template via onnxruntime_genai (messages must be a JSON string)
messages = json.dumps([{"role": "user", "content": text}])
prompt = tokenizer.apply_chat_template(
    messages,
    template_str=template_str,
    add_generation_prompt=True,
)

Result

Qwen: no meaningful output

llama - 60% npu

Overall, Isaac Newton was a brilliant scientist and mathematician who made significant contributions to our understanding of the natural world. 
Total tokens generated: 329
Average time per token: 0.0295 seconds
Tokens per second: 33.90

DS - 60% npu repeative unstable

Isaac Newton was a British mathematician and physicist who was an key figure in the development of the scientific method. He was born in 1644 and died in 176, and his contributions to the understanding of physics, particularly the laws of motion and the law of gravity, have been fundamental to the development of the science we know today. He also made significant contributions to the development of calculus, which is essential for many fields, including physics and engineering. Newton's work on the laws of motion and the law of gravity has been essential in understanding the behavior of objects in motion and the structure of the universe. His contributions have had a significant impact on the modern world, and his work is still studied and used in the fields he is known for.
Total tokens generated: 467
Average time per token: 0.0357 seconds
Tokens per second: 28.03

phi - slow as 25% npu usage

Total tokens generated: 256
Average time per token: 0.1797 seconds
Tokens per second: 5.57

llama 8b - 65% npu usage

Newton's legacy continues to be felt today, and his work remains a cornerstone of modern science and mathematics. 
Total tokens generated: 591
Average time per token: 0.1406 seconds
Tokens per second: 7.11