As everyone is aware, the world is still going nuts attempting to develop more, newer and much better AI tools. Mainly by tossing unreasonable quantities of cash at the issue. A number of those billions go towards developing cheap or free services that run at a substantial loss. The tech giants that run them all are wishing to bring in as lots of users as possible, so that they can catch the marketplace, and end up being the dominant or just celebration that can use them. It is the timeless Silicon Valley playbook. Once supremacy is reached, anticipate the enshittification to start.

A likely method to earn back all that money for developing these LLMs will be by tweaking their outputs to the taste of whoever pays the a lot of. An example of what that such tweaking looks like is the rejection of DeepSeek's R1 to discuss what occurred at Tiananmen Square in 1989. That a person is certainly politically motivated, however ad-funded services will not exactly be fun either. In the future, I completely anticipate to be able to have a frank and truthful discussion about the Tiananmen occasions with an American AI agent, however the only one I can afford will have presumed the personality of Father Christmas who, while holding a can of Coca-Cola, will intersperse the stating of the awful events with a joyful "Ho ho ho ... Didn't you understand? The vacations are coming!"

Or perhaps that is too improbable. Right now, dispite all that cash, the most popular service for code completion still has difficulty dealing with a number of basic words, in spite of them existing in every dictionary. There must be a bug in the "totally free speech", or something.

But there is hope. Among the techniques of an approaching gamer to shock the market, is to damage the incumbents by releasing their model totally free, under a permissive license. This is what DeepSeek just made with their DeepSeek-R1. Google did it earlier with the Gemma models, as did Meta with Llama. We can download these models ourselves and run them on our own hardware. Better yet, people can take these designs and scrub the biases from them. And we can download those scrubbed designs and run those on our own hardware. And then we can finally have some genuinely helpful LLMs.

That hardware can be a difficulty, though. There are two choices to pick from if you want to run an LLM in your area. You can get a big, effective video card from Nvidia, or you can buy an Apple. Either is pricey. The main specification that indicates how well an LLM will perform is the amount of memory available. VRAM when it comes to GPU's, regular RAM in the case of Apples. Bigger is better here. More RAM implies bigger designs, which will considerably improve the quality of the output. Personally, I 'd say one requires a minimum of over 24GB to be able to run anything beneficial. That will fit a 32 billion specification design with a little headroom to spare. Building, or buying, a workstation that is geared up to handle that can quickly cost countless euros.

So what to do, if you do not have that quantity of cash to spare? You buy pre-owned! This is a viable choice, but as constantly, there is no such thing as a free lunch. Memory might be the main issue, however do not undervalue the importance of memory bandwidth and other specs. Older devices will have lower efficiency on those aspects. But let's not stress excessive about that now. I have an interest in building something that a minimum of can run the LLMs in a functional method. Sure, the current Nvidia card might do it faster, however the point is to be able to do it at all. Powerful online designs can be nice, but one must at the minimum have the alternative to switch to a regional one, if the situation requires it.

Below is my effort to develop such a capable AI computer system without investing excessive. I wound up with a workstation with 48GB of VRAM that cost me around 1700 euros. I might have done it for less. For instance, it was not strictly necessary to buy a brand name new dummy GPU (see below), or I might have found someone that would 3D print the cooling fan shroud for me, rather of delivering a ready-made one from a faraway nation. I'll confess, I got a bit restless at the end when I discovered out I had to buy yet another part to make this work. For me, this was an acceptable tradeoff.

Hardware

This is the complete cost breakdown:

And this is what it appeared like when it initially booted with all the parts set up:

I'll give some context on the parts below, and after that, I'll run a couple of quick tests to get some numbers on the efficiency.

HP Z440 Workstation

The Z440 was a simple choice because I already owned it. This was the starting point. About 2 years earlier, I wanted a computer that could act as a host for my virtual machines. The Z440 has a Xeon processor with 12 cores, and this one sports 128GB of RAM. Many threads and a lot of memory, that must work for hosting VMs. I bought it previously owned and then swapped the 512GB hard disk for a 6TB one to save those virtual machines. 6TB is not required for running LLMs, and therefore I did not include it in the breakdown. But if you prepare to collect numerous designs, 512GB may not suffice.

I have actually pertained to like this workstation. It feels all extremely solid, and I haven't had any problems with it. At least, until I began this task. It turns out that HP does not like competitors, and I encountered some troubles when switching parts.

2 x NVIDIA Tesla P40

This is the magic component. GPUs are costly. But, as with the HP Z440, often one can find older equipment, that utilized to be leading of the line and is still really capable, pre-owned, for fairly little money. These Teslas were indicated to run in server farms, for things like 3D making and other graphic processing. They come equipped with 24GB of VRAM. Nice. They suit a PCI-Express 3.0 x16 slot. The Z440 has 2 of those, so we purchase two. Now we have 48GB of VRAM. Double good.

The catch is the part about that they were indicated for servers. They will work great in the PCIe slots of a typical workstation, however in servers the cooling is handled in a different way. Beefy GPUs consume a great deal of power and can run very hot. That is the factor consumer GPUs constantly come equipped with huge fans. The cards need to take care of their own cooling. The Teslas, however, have no fans whatsoever. They get simply as hot, however anticipate the server to provide a constant circulation of air to cool them. The enclosure of the card is rather shaped like a pipeline, and you have 2 options: blow in air from one side or blow it in from the opposite. How is that for versatility? You absolutely should blow some air into it, though, or you will harm it as quickly as you put it to work.

The service is easy: just install a fan on one end of the pipeline. And certainly, it seems a whole home industry has grown of people that sell 3D-printed shrouds that hold a standard 60mm fan in just the right location. The issue is, the cards themselves are currently rather large, and it is difficult to find a configuration that fits 2 cards and 2 fan installs in the computer case. The seller who sold me my two Teslas was kind sufficient to consist of two fans with shrouds, but there was no other way I might fit all of those into the case. So what do we do? We buy more parts.

NZXT C850 Gold

This is where things got frustrating. The HP Z440 had a 700 Watt PSU, townshipmarket.co.za which might have been enough. But I wasn't sure, and I required to purchase a brand-new PSU anyway because it did not have the best connectors to power the Teslas. Using this useful site, I deduced that 850 Watt would be enough, and I bought the NZXT C850. It is a modular PSU, implying that you only need to plug in the cables that you really require. It included a neat bag to keep the extra cable televisions. One day, I may give it a great cleaning and use it as a toiletry bag.

Unfortunately, HP does not like things that are not HP, so they made it difficult to switch the PSU. It does not fit physically, and they likewise changed the main board and CPU ports. All PSU's I have ever seen in my life are rectangular boxes. The HP PSU likewise is a rectangular box, but with a cutout, making certain that none of the typical PSUs will fit. For no technical factor at all. This is simply to tinker you.

The mounting was eventually resolved by using two random holes in the grill that I in some way managed to align with the screw holes on the NZXT. It sort of hangs stable now, and I feel lucky that this worked. I have seen Youtube videos where individuals resorted to double-sided tape.

The port required ... another purchase.

Not cool HP.

Gainward GT 1030

There is another issue with using server GPUs in this customer workstation. The Teslas are meant to crunch numbers, not to play video games with. Consequently, they do not have any ports to link a display to. The BIOS of the HP Z440 does not like this. It refuses to boot if there is no chance to output a video signal. This computer system will run headless, but we have no other option. We have to get a third video card, that we don't to intent to utilize ever, simply to keep the BIOS delighted.

This can be the most scrappy card that you can discover, naturally, however there is a requirement: we must make it fit on the main board. The Teslas are large and fill the 2 PCIe 3.0 x16 slots. The only slots left that can physically hold a card are one PCIe x4 slot and one PCIe x8 slot. See this website for some background on what those names imply. One can not purchase any x8 card, however, because often even when a GPU is marketed as x8, the real adapter on it might be just as broad as an x16. Electronically it is an x8, physically it is an x16. That will not work on this main board, we actually require the small adapter.

Nvidia Tesla Cooling Fan Kit

As said, the obstacle is to find a fan shroud that fits in the case. After some browsing, I discovered this package on Ebay a bought two of them. They came delivered complete with a 40mm fan, and everything fits completely.

Be warned that they make an awful lot of sound. You don't wish to keep a computer with these fans under your desk.

To keep an eye on the temperature level, I whipped up this fast script and put it in a cron task. It regularly reads out the temperature on the GPUs and sends out that to my Homeassistant server:

In Homeassistant I added a graph to the control panel that displays the values over time:

As one can see, the fans were loud, but not especially effective. 90 degrees is far too hot. I browsed the web for an affordable upper limit however might not find anything specific. The documentation on the Nvidia website discusses a temperature of 47 degrees Celsius. But, what they mean by that is the temperature of the ambient air surrounding the GPU, not the measured value on the chip. You know, the number that in fact is reported. Thanks, Nvidia. That was valuable.

After some more searching and reading the viewpoints of my fellow internet residents, my guess is that things will be fine, provided that we keep it in the lower 70s. But do not estimate me on that.

My first attempt to correct the situation was by setting an optimum to the power usage of the GPUs. According to this Reddit thread, one can decrease the power consumption of the cards by 45% at the cost of only 15% of the performance. I tried it and ... did not observe any distinction at all. I wasn't sure about the drop in performance, having just a number of minutes of experience with this setup at that point, however the temperature attributes were certainly the same.

And after that a light bulb flashed on in my head. You see, right before the GPU fans, there is a fan in the HP Z440 case. In the image above, it remains in the ideal corner, inside the black box. This is a fan that draws air into the case, and I figured this would operate in tandem with the GPU fans that blow air into the Teslas. But this case fan was not spinning at all, because the remainder of the computer system did not require any cooling. Looking into the BIOS, I found a setting for the minimum idle speed of the case fans. It varied from 0 to 6 stars and was currently set to 0. Putting it at a higher setting did wonders for the temperature. It also made more noise.

confess that the third video card was practical when adjusting the BIOS setting.

MODDIY Main Power Adaptor Cable and Akasa Multifan Adaptor

Fortunately, often things just work. These two items were plug and play. The MODDIY adaptor cable linked the PSU to the main board and CPU power sockets.

I utilized the Akasa to power the GPU fans from a 4-pin Molex. It has the good feature that it can power 2 fans with 12V and two with 5V. The latter certainly decreases the speed and thus the cooling power of the fan. But it also lowers sound. Fiddling a bit with this and the case fan setting, I found an acceptable tradeoff between noise and temperature. For now a minimum of. Maybe I will need to review this in the summer season.

Some numbers

Inference speed. I gathered these numbers by running ollama with the-- verbose flag and asking it five times to write a story and averaging the outcome:

Performancewise, ollama is set up with:

All models have the default quantization that ollama will pull for you if you don't specify anything.

Another essential finding: Terry is without a doubt the most popular name for a tortoise, followed by Turbo and Toby. Harry is a preferred for hares. All LLMs are loving alliteration.

Power consumption

Over the days I watched on the power usage of the workstation:

Note that these numbers were taken with the 140W power cap active.

As one can see, there is another tradeoff to be made. Keeping the design on the card enhances latency, however takes in more power. My current setup is to have two models packed, one for coding, the other for generic text processing, and keep them on the GPU for up to an hour after last usage.

After all that, am I pleased that I started this task? Yes, I believe I am.

I spent a bit more cash than prepared, but I got what I wanted: a way of locally running medium-sized models, completely under my own control.

It was a good option to start with the workstation I already owned, and see how far I might feature that. If I had started with a brand-new device from scratch, it certainly would have cost me more. It would have taken me a lot longer too, as there would have been many more alternatives to select from. I would also have been really lured to follow the hype and purchase the current and greatest of everything. New and glossy toys are enjoyable. But if I buy something brand-new, I want it to last for several years. Confidently anticipating where AI will go in 5 years time is impossible today, so having a more affordable device, that will last a minimum of some while, feels acceptable to me.

I want you great luck on your own AI journey. I'll report back if I find something new or intriguing.