Re: EXAONE on MLA100 PCIe Card

General Q&A
1

Hello Team👋

Forwarding a technical question on MLA100 PCIe Card:

“Would love to learn more about the deployment details — especially the EXAONE model size, quantization approach, memory footprint, supported context length, and real-world inference performance on a single MLA100 card. Looking forward to more technical updates”

Could you share some information on this? Thank you so much in advance!

1 Like
2

Hello!
Thank you for reaching out with your questions. Here is a technical summary regarding the deployment and performance of EXAONE models on the MLA100 PCIe Card.

1. Model Size & Availability

  • Current Distribution: We currently provide EXAONE models with sizes up to 7.8B via our Hugging Face Hub.
  • Scalability: The MLA100 architecture is designed to scale; as memory allows, users can deploy larger models, such as the 32B variant.

2. Memory & Context Length

Feature Default Specification Extended Option
Memory Footprint 16 GB Up to 32 GB
Context Length 4,096 tokens Expandable based on available memory

3. Quantization Approach

The MLA100 offers a flexible quantization framework to optimize efficiency:

  • Standard: 8-bit quantization is provided as the default.
  • Granular Control: We support 4-bit quantization and allow users to configure quantization bits for each transformer module independently.
  • Mixed Precision: Users can compile models with hybrid configurations, such as W4V8 (4-bit Weights / 8-bit Vectors), to maximize performance while maintaining precision.

4. Real-World Inference Performance

Based on our benchmarks for the EXAONE-3.5-2.4B-Instruct model:

  • Throughput: Approximately 21 TPS (Tokens Per Second) for a single user using W4V8 quantization.
  • Accuracy: Under these settings, the model retains approximately 96% of the MMLU accuracy compared to the original floating-point precision model.
  • Further Optimization: Performance can be further enhanced through fine-tuned quantization parameters and advanced techniques such as speculative decoding.

We are continuously updating our software stack to provide even better performance. Please let us know if you need further technical documentation or additional benchmark data!

1 Like
3

P.S. Please feel free to ask any questions in Korean if that is more comfortable for you.