NVIDIA Tensorrt LLM - Search Videos

You now convert any LLM into a faster one without retraining from scratch.NVIDIA just did this to their 30B model. Here's the trick:1. Duplicate the model into two copies2. Freeze one copy, it just reads the prompt and remembers context3. Train the other copy to write chunks of text at once instead of one word at a time4. Run them togetherThe frozen copy barely costs anything (it's already trained). The new copy only needed ~8% of the original training data to learn the new trick.Result: 2.4x fa

You now convert any LLM into a faster one without retraining from scratch.NVIDIA just did this to their 30B model. Here's the trick:1. Duplicate the model into two copies2. Freeze one copy, it just reads the prompt and remembers context3. Train the other copy to write chunks of text at once instead of one word at a time4. Run them togetherThe frozen copy barely costs anything (it's already trained). The new copy only needed ~8% of the original training data to learn the new trick.Result: 2.4x fa

103.4K views1 day ago

x.comLior Alexander

See more