Skip to main content

Building an energy efficient server

· 11 min read
Webber

I've built a server that's both powerful and quite energy efficient. It has modern hardware running on less than 10 Watts!

In this chapter I've documented my reasoning around the build and each component. I also explain the tweaks I've done for my specific setup.

info

This post was originally part of my developer notes.

Hardware

Let's start with the hardware. What did I get and why?

In order to tell you why, I need to tell you what requirements I had.

Requirements

  • Since it will be running Home Assistant, and basically be the brain of my home, it needs to be reliable.
  • It also needs to run 24/7 without costing a fortune in electricity, so low power usage.
  • I really wanted it to be able to run a few virtualised workloads for when I want to play around with stuff, so decent performance.
  • My 19" rack is connected to the kitchen, so it needs to be quiet.

Research

I did a lot of research on what hardware to get and looked at many different options.

info

One of the main resources I used was The Big Efficient Server Topic (Part 3) on a Dutch tech website called Tweakers.net. It really is a great resource for anyone looking to build an energy efficient server.

Components

These are the components I ended up with. I've tried to describe my reasoning for each component, but if you have any questions please don't hesitate to drop a comment.

Main components

#ComponentProductReasoning
1EnclosureInter-Tech IPC 2U-202482U, small enough to fit in my very small server rack
1MotherboardFujitsu D3642-BWith a cost around €200 it's not exactly a budget board, but it's a modern and energy efficient mobo that's well worth the price.
1CPUIntel Core i7-9700K Tray (SRG15)Supports virtualisation technologies like VT-x. 9th gen because it's more efficient than 10th gen, and especially 11~13th gen. I7 over I3 because it's more powerful, even with the TDP being limited
1CPU CoolingNoctua NH-L9i BrownI love this brand. Absolutely silent.
1Memory64GB Crucial CT2K32G4DFD832AMemory size has a relatively small effect on energy usage. 64GB for running virtual workloads without worries.
1Drive bayDeLOCK 47221 drive bay panelI didn't want to go for a SASS backplane because it might need upgrading when disk sizes become larger. Also challenging to find SASS platforms that don't require much power. This is rather simple: 6 sata cables directly to the mobo.
2Solid state drivesSamsung 870 Evo (MZ-77E4T0B/EU) 4TBTo be used as the Array in Unraid, which requires 1 Parity disk (which you can not use). Unraid is basically utilising a modern version of software RAID.
1Power SupplyMini-box PicoPSU-90 + 84W (Fortron) Adapter Power KitPower supplies become very inefficient when utilised below 20%. This PSU is tailored towards idle load, which is expected to be lower than 10W

Accessories

#ComponentProductReasoning
1Rack railsInter-Tech 18" telescopic rails 350mm - 450mmFor fitting it in the server rack. I had to customise these to fit in my very small 60cm rack
2Sata cables3 x 7-pin SATA 3 / 6Gbit/s 30cmStandard cables that look nice
1Sata to MolexzdyCGTime SATA 15 Pin to Dual 4 Pin Molex One to Two Power CablePicoPSU only has 1x SATA power and 1x Molex power. This cable splits the SATA power connector into two Molex connectors, both of which go into the drive bay. It powers all six potential SSDs
1USB flash drive (OS)Verbatim Metal Executive USB 3.0 flash drive 64 GB, USB flash driveThe operating system will run from this USB flash drive, simply because it uses less power than when using another disk

The build

Here's what the build looks like.

Efficiency tweaks

Measuring

To measure the power usage I used a calibrated Nous A1T.

I calibrated the plug using a multi meter and a 40W light bulb (resistive load).

First boot

First thing I did was boot it up with no additional settings. Not surprisingly, it crashed while booting into Unraid. Loading an OS is CPU heavy task and the energy usage spiked above the maximum the PSU could handle.

So the first tweak was to

  • set the CPU TDP limit to 35W.

After this everything worked very smoothly. 🎉

Time for stats!

Stress testing

I ran a stress test with mprime to see what the maximum power usage was.

Source: https://www.mersenne.org/download/

# Download
wget https://www.mersenne.org/download/software/v30/30.8/p95v308b17.linux64.tar.gz

# Extract
mkdir mprime && tar -xf ./p95v308b17.linux64.tar.gz -C mprime

# Manual
./mprime/mprime --help

# Run
./mprime/mprime -t

For the visual people among us, here's what that looks like:

Initial stats

Now that we know both our idle and stress power usage, we can start tweaking.

But before that, time for the initial stats!

Idle lowIdle avgIdle highStress lowStress avgStress high
11W13W16W54W56W62W

What's interesting is that even without any tweaks, the idle power usage of the server is already very low.

This has everything to do with the specific hardware that was chosen, especially the motherboard with its focus on energy efficiency and the power supply unit (PSU) not being so large that it becomes inefficient at low loads.

info

The server will be running 24/7, so the average idle power usage is the most important metric here.

note

The artificial load of the stress test is not representative of real world usage. Therefore, I will not be using the high values but at maximum the avg values when reconfiguring max TDP for the CPU.

BIOS tweaks

BIOS tweaks: Turn off unused devices

Firstly, the most obvious tweaks: turning off all the things we don't need.

  • Audio: Disabled
  • PCI Express Slot 1: Disabled
  • PCI Express Slot 2: Disabled
  • PCI Express Slot 3: Disabled
  • PCI Express Slot 4: Disabled
  • Serial port: Disabled

Great results so far!

Idle lowIdle avgIdle high
8W10W13W

BIOS tweaks: Review all relevant settings

CPU

  • Enhanced Intel SpeedStep Technology: Enabled (was Enabled)
  • Turbo Boost: Disabled (was Enabled)
  • C States: Enabled (was Enabled)
  • Package C State Limit: Auto (was Auto)

And since there's some headroom left, I also increase the maximum power limit.

  • CPU TDP Limit: 60W (was 35W)

USB

  • USB 3.1 Gen 2 speed: Disabled (was Enabled)
  • USB Port Control: Enable rear and internal ports (was Enable all ports)

NIC

  • Wake on LAN: Disabled (was Enabled)

PCI

  • Above 4G Decoding: Disabled (was Enabled)
Idle lowIdle avgIdle high
7W9.5W12W

BIOS tweaks: Disable graphics

The default settings for my motherboard are as follows:

  • Primary display: Auto
  • Internal graphics: Auto
info

When using Internal graphics: Auto, the BIOS detects automatically if the internal graphics controller can be disabled.

This means that the next test is to pull out the monitor cable and see how it affects power usage. Unfortunately pulling out the cable didn't make any difference.

I then manually disabled the internal graphics controller, which did seem to make a small difference, but still too small to measure.

info

At this point the difference also becomes hard to measure, as the power usage can jump around a bit and the update interval of the Nous A1T is about 1 second.

I wasn't able to detect a significant difference in power usage with this tweak.

Software tweaks

Software tweaks can help out where hardware settings are not sufficient or not available. One of the main things to focus on is the CPU, or more specifically, the C-states.

C-states

info

C-states, also known as CPU states, are power-saving states for processors (CPUs) in modern computer systems. These states are designed to reduce the power consumption of the CPU when it is idle or not performing any computational tasks.

The different C-states (C0 through C10) represent varying levels of power-saving modes:

  • C0 is the operating state.
  • C10 is the deepest idle state.
  • The higher the C-state, the more power is saved.

We'll use Powertop to try and improve the C-state usage.

Powertop

info

Powertop is a tool provided by Intel to enable various power saving modes in userspace, kernel and on hardware level.

Powertop comes as one of the recommended tools to tweak power usage on Linux. It's a tool that gives you quite a bit of insight into what's going on with your system. It also allows you to tweak some settings.

I've followed this guide to install it on Unraid. I'll describe some of the steps here.

Install

Download the file in /boot/extra, then reboot to automatically install it.

mkdir /boot/extra
cd /boot/extra
wget https://github.com/mgutt/unraid-packages/raw/main/6.11.0/powertop-2.15-x86_64-1.txz

Analyse

Run powertop without --auto-tune (for now) to get an overview of what's going on.

powertop

On the Idle stats tab it looks like much of the time is spent in the C2 state.

Looking at the Tunables tab: It seems like there are many things to tune still.

Tuning

Now that we have an overview of what's going on, let's try to tune some of the settings.

powertop --auto-tune

# or alternatively
powertop --quiet --auto-tune
note

This command will be undone after every reboot, unless you add it to the go file. Keep reading 🙂

After running this command, the Tunables tab looks like this:

The Idle stats tab looks a lot better now too: Instead of the CPU spending most of the time in the C2 state it's now spending most of the time in the C8 state.

Keep tuning after reboot

First of, let's test that the array still comes up after tuning.

In Unraid stop the array, then run the tuning command again.

powertop --quiet --auto-tune

Now start the array again. If the array comes up, you should be good to go.

To keep the tuning after reboot, add the tuning command to the go file like so:

echo "powertop --quiet --auto-tune" >> /boot/config/go

Finally, reboot and check if everything still works as expected.

reboot

Results

Et voilà! After applying tuning, the power usage is down to 7W on average. Awesome! 🎉

Idle lowIdle avgIdle high
6W7W10W

Conclusion

And there we have it!

A powerful server that uses only 7W on average when idle. I'm really happy with the results.

Credits

I could not have done it without the awesome contributions people made to the Big Efficient Server Topic (Dutch).

Special thanks to mrmrmr in particular. 🙏🏻

What are your thoughts?