Sometimes you want LLMs that are unencumbered by non-commercial licenses. Below is a list of some notable LLMs that have friendly license agreements. * Mistral family * Mistral 7B, Mixtral 8*7B, Mixtral 8*22B * Mistral Nemo 12GB with quantization aware training for good FP8 performance * Qwen2-0.5B, Qwen2-1.5B, Qwen2-7B, Qwen2-57B-A14B

One overlooked aspect of GPU usage is the power they consume when idle. Idle power draw refers to the amount of electricity a GPU consumes when it's not performing intensive tasks. This can significantly impact both energy consumption and electricity costs over time. Without any tricks, a P40

You just have to resort to running on your computer with 12 sticks of 32GB RAM!

Custom case, or special server cards to fit 4 GPUs into a case? No need, we'll just squash them in there. Congrats to stonedoubt for this tetris-like feat and great thermal density! Behold my dumb sh*t 😂😂😂 by u/stonedoubt in LocalLLaMA

This build is several years old so the prices quoted are much higher than today. I've been very interested in V100 SXM builds as more of these come onto the market and prices fall. I've not yet pulled the trigger on such a build as prices

I plotted this chart and thought I'd share it in case it was useful to others. It is the tok/s output at different power limits with a RTX 3090 during single-inferencing. While maximum efficiency is achieved around 211W, this reduces output by around 20% Running between 260W-280W