Baidu put Wenxin 5.5 Flash Base on Hugging Face and made it open source.
This is a Base model, not a conversational version. This means that without alignment and instruction fine-tuning, the effect of using it for direct chatting will not be good. It is used by developers to train themselves on industry models.
Specifications: MoE architecture, total parameters 280 billion, each token activated 14 billion, activation ratio is about 5%. Pre-training consumes 33.5 trillion tokens. The longest context in the pre-training stage is 128K, and can reach 1 million tokens in the fine-tuning and inference stages.
128K training, 1M usage, the difference is nearly 8 times. Whether this kind of extrapolation is stable in long texts is difficult to measure directly with the Base model. We will have to wait for the community to fine-tune several versions to know.
Baidu says it has the same size SOTA in terms of knowledge, reasoning, mathematics, coding, multi-language, and long context. With this caliber, we should pay attention to who the "same size" comparison object is and how to draw the size. The announcement did not elaborate on it.
The optimization ideas are clearly written and focus on efficiency: the attention mechanism, expert routing, and training goals have all been changed, with the aim of extracting more capabilities from the same computing power.
For domestic companies doing privatization deployment, this type of open-weighted medium-sized MoE is actually more practical than another closed-source flagship. Wenxin Big Model In the past, people had the impression that the closed source route would go to the end. In the past two years, the pace of open source has been significantly faster.
Whether it can be used depends mainly on two things: whether the license has many restrictions, and whether the inference framework can keep up with the adaptation. These two points will basically be revealed next week.