I've built a server that's both powerful and quite energy efficient. It has modern hardware running on less than 10 Watts!
In this chapter I've documented my reasoning around the build and each component. I also explain the tweaks I've done for my specific setup.
This post was originally part of my developer notes.
Hardware
Let's start with the hardware. What did I get and why?
In order to tell you why, I need to tell you what requirements I had.
Requirements
- Since it will be running Home Assistant, and basically be the brain of my home, it needs to be reliable.
- It also needs to run 24/7 without costing a fortune in electricity, so low power usage.
- I really wanted it to be able to run a few virtualised workloads for when I want to play around with stuff, so decent performance.
- My 19" rack is connected to the kitchen, so it needs to be quiet.
Research
I did a lot of research on what hardware to get and looked at many different options.
One of the main resources I used was The Big Efficient Server Topic (Part 3) on a Dutch tech website called Tweakers.net. It really is a great resource for anyone looking to build an energy efficient server.
Components
These are the components I ended up with. I've tried to describe my reasoning for each component, but if you have any questions please don't hesitate to drop a comment.
Main components
# | Component | Product | Reasoning |
---|---|---|---|
1 | Enclosure | Inter-Tech IPC 2U-20248 | 2U, small enough to fit in my very small server rack |
1 | Motherboard | Fujitsu D3642-B | With a cost around €200 it's not exactly a budget board, but it's a modern and energy efficient mobo that's well worth the price. |
1 | CPU | Intel Core i7-9700K Tray (SRG15) | Supports virtualisation technologies like VT-x. 9th gen because it's more efficient than 10th gen, and especially 11~13th gen. I7 over I3 because it's more powerful, even with the TDP being limited |
1 | CPU Cooling | Noctua NH-L9i Brown | I love this brand. Absolutely silent. |
1 | Memory | 64GB Crucial CT2K32G4DFD832A | Memory size has a relatively small effect on energy usage. 64GB for running virtual workloads without worries. |
1 | Drive bay | DeLOCK 47221 drive bay panel | I didn't want to go for a SASS backplane because it might need upgrading when disk sizes become larger. Also challenging to find SASS platforms that don't require much power. This is rather simple: 6 sata cables directly to the mobo. |
2 | Solid state drives | Samsung 870 Evo (MZ-77E4T0B/EU) 4TB | To be used as the Array in Unraid, which requires 1 Parity disk (which you can not use). Unraid is basically utilising a modern version of software RAID. |
1 | Power Supply | Mini-box PicoPSU-90 + 84W (Fortron) Adapter Power Kit | Power supplies become very inefficient when utilised below 20%. This PSU is tailored towards idle load, which is expected to be lower than 10W |
Accessories
# | Component | Product | Reasoning |
---|---|---|---|
1 | Rack rails | Inter-Tech 18" telescopic rails 350mm - 450mm | For fitting it in the server rack. I had to customise these to fit in my very small 60cm rack |
2 | Sata cables | 3 x 7-pin SATA 3 / 6Gbit/s 30cm | Standard cables that look nice |
1 | Sata to Molex | zdyCGTime SATA 15 Pin to Dual 4 Pin Molex One to Two Power Cable | PicoPSU only has 1x SATA power and 1x Molex power. This cable splits the SATA power connector into two Molex connectors, both of which go into the drive bay. It powers all six potential SSDs |
1 | USB flash drive (OS) | Verbatim Metal Executive USB 3.0 flash drive 64 GB, USB flash drive | The operating system will run from this USB flash drive, simply because it uses less power than when using another disk |
The build
Here's what the build looks like.
Efficiency tweaks
Measuring
To measure the power usage I used a calibrated Nous A1T.
I calibrated the plug using a multi meter and a 40W light bulb (resistive load).
First boot
First thing I did was boot it up with no additional settings. Not surprisingly, it crashed while booting into Unraid. Loading an OS is CPU heavy task and the energy usage spiked above the maximum the PSU could handle.
So the first tweak was to
- set the CPU TDP limit to 35W.
After this everything worked very smoothly. 🎉
Time for stats!
Stress testing
I ran a stress test with mprime
to see what the maximum power usage was.
Source: https://www.mersenne.org/download/
# Download
wget https://www.mersenne.org/download/software/v30/30.8/p95v308b17.linux64.tar.gz
# Extract
mkdir mprime && tar -xf ./p95v308b17.linux64.tar.gz -C mprime
# Manual
./mprime/mprime --help
# Run
./mprime/mprime -t
For the visual people among us, here's what that looks like:
Initial stats
Now that we know both our idle and stress power usage, we can start tweaking.
But before that, time for the initial stats!
Idle low | Idle avg | Idle high | Stress low | Stress avg | Stress high | |
---|---|---|---|---|---|---|
11W | 13W | 16W | 54W | 56W | 62W |
What's interesting is that even without any tweaks, the idle power usage of the server is already very low.
This has everything to do with the specific hardware that was chosen, especially the motherboard with its focus on energy efficiency and the power supply unit (PSU) not being so large that it becomes inefficient at low loads.
The server will be running 24/7, so the average idle power usage is the most important metric here.
The artificial load of the stress test is not representative of real world usage. Therefore, I will
not be using the high
values but at maximum the avg
values when reconfiguring max TDP for the
CPU.
BIOS tweaks
BIOS tweaks: Turn off unused devices
Firstly, the most obvious tweaks: turning off all the things we don't need.
- Audio: Disabled
- PCI Express Slot 1: Disabled
- PCI Express Slot 2: Disabled
- PCI Express Slot 3: Disabled
- PCI Express Slot 4: Disabled
- Serial port: Disabled
Great results so far!
Idle low | Idle avg | Idle high |
---|---|---|
8W | 10W | 13W |
BIOS tweaks: Review all relevant settings
CPU
- Enhanced Intel SpeedStep Technology: Enabled (was Enabled)
- Turbo Boost: Disabled (was Enabled)
- C States: Enabled (was Enabled)
- Package C State Limit: Auto (was Auto)
And since there's some headroom left, I also increase the maximum power limit.
- CPU TDP Limit: 60W (was 35W)
USB
- USB 3.1 Gen 2 speed: Disabled (was Enabled)
- USB Port Control: Enable rear and internal ports (was Enable all ports)
NIC
- Wake on LAN: Disabled (was Enabled)
PCI
- Above 4G Decoding: Disabled (was Enabled)
Idle low | Idle avg | Idle high |
---|---|---|
7W | 9.5W | 12W |
BIOS tweaks: Disable graphics
The default settings for my motherboard are as follows:
- Primary display: Auto
- Internal graphics: Auto
When using Internal graphics: Auto
, the BIOS detects automatically if the internal graphics
controller can be disabled.
This means that the next test is to pull out the monitor cable and see how it affects power usage. Unfortunately pulling out the cable didn't make any difference.
I then manually disabled the internal graphics controller, which did seem to make a small difference, but still too small to measure.
At this point the difference also becomes hard to measure, as the power usage can jump around a bit and the update interval of the Nous A1T is about 1 second.
I wasn't able to detect a significant difference in power usage with this tweak.
Software tweaks
Software tweaks can help out where hardware settings are not sufficient or not available. One of the main things to focus on is the CPU, or more specifically, the C-states.
C-states
C-states, also known as CPU states, are power-saving states for processors (CPUs) in modern computer systems. These states are designed to reduce the power consumption of the CPU when it is idle or not performing any computational tasks.
The different C-states (C0 through C10) represent varying levels of power-saving modes:
- C0 is the operating state.
- C10 is the deepest idle state.
- The higher the C-state, the more power is saved.
We'll use Powertop to try and improve the C-state usage.
Powertop
Powertop is a tool provided by Intel to enable various power saving modes in userspace, kernel and on hardware level.
Powertop comes as one of the recommended tools to tweak power usage on Linux. It's a tool that gives you quite a bit of insight into what's going on with your system. It also allows you to tweak some settings.
I've followed this guide to install it on Unraid. I'll describe some of the steps here.
Install
Download the file in /boot/extra, then reboot to automatically install it.
mkdir /boot/extra
cd /boot/extra
wget https://github.com/mgutt/unraid-packages/raw/main/6.11.0/powertop-2.15-x86_64-1.txz
Analyse
Run powertop without --auto-tune
(for now) to get an overview of what's going on.
powertop
On the Idle stats
tab it looks like much of the time is spent in the C2 state.
Looking at the Tunables
tab: It seems like there are many things to tune still.
Tuning
Now that we have an overview of what's going on, let's try to tune some of the settings.
powertop --auto-tune
# or alternatively
powertop --quiet --auto-tune
This command will be undone after every reboot, unless you add it to the go
file. Keep reading 🙂
After running this command, the Tunables
tab looks like this:
The Idle stats
tab looks a lot better now too: Instead of the CPU spending most of the time in the
C2 state it's now spending most of the time in the C8 state.
Keep tuning after reboot
First of, let's test that the array still comes up after tuning.
In Unraid stop the array, then run the tuning command again.
powertop --quiet --auto-tune
Now start the array again. If the array comes up, you should be good to go.
To keep the tuning after reboot, add the tuning command to the go
file like so:
echo "powertop --quiet --auto-tune" >> /boot/config/go
Finally, reboot and check if everything still works as expected.
reboot
Results
Et voilà! After applying tuning, the power usage is down to 7W on average. Awesome! 🎉
Idle low | Idle avg | Idle high |
---|---|---|
6W | 7W | 10W |
Conclusion
And there we have it!
A powerful server that uses only 7W on average when idle. I'm really happy with the results.
Credits
I could not have done it without the awesome contributions people made to the Big Efficient Server Topic (Dutch).
Special thanks to mrmrmr in particular. 🙏🏻