Notable LLMs that are Apache/MIT licensed Sometimes you want LLMs that are unencumbered by non-commercial licenses. Below is a list of some notable LLMs that have friendly license agreements. * Mistral family * Mistral 7B, Mixtral 8*7B, Mixtral 8*22B * Mistral Nemo 12GB with quantization aware training for good FP8 performance * Qwen2-0.5B, Qwen2-1.5B, Qwen2-7B, Qwen2-57B-A14B
Reducing idle power consumption for Nvidia P100 and P40 GPUs One overlooked aspect of GPU usage is the power they consume when idle. Idle power draw refers to the amount of electricity a GPU consumes when it's not performing intensive tasks. This can significantly impact both energy consumption and electricity costs over time. Without any tricks, a P40
Sometimes when you don't have 340 GB of VRAM You just have to resort to running on your computer with 12 sticks of 32GB RAM!
How many GPUs do you want to cram into your box? Yes. Custom case, or special server cards to fit 4 GPUs into a case? No need, we'll just squash them in there. Congrats to stonedoubt for this tetris-like feat and great thermal density! Behold my dumb sh*t 😂😂😂 by u/stonedoubt in LocalLLaMA
4xV100 SXM Build This build is several years old so the prices quoted are much higher than today. I've been very interested in V100 SXM builds as more of these come onto the market and prices fall. I've not yet pulled the trigger on such a build as prices
Power limiting RTX 3090 GPU to increase power efficiency I plotted this chart and thought I'd share it in case it was useful to others. It is the tok/s output at different power limits with a RTX 3090 during single-inferencing. While maximum efficiency is achieved around 211W, this reduces output by around 20% Running between 260W-280W
