homelab

I wanted to see what my mostly stock Drupal server was capable of.  For this first test I am starting with a simple homepage load. I used the ab tool (Apache Bench).  It couldn't be simpler!  But it only downloads the HTML, not the rest of the assets.  Here is one page load:

user@hostname:~/$ ab -n 1 -c 1 https://aaron.axvigs.com/
This is ApacheBench, Version 2.3 <$Revision: 1923142 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking aaron.axvigs.com (be patient).....done

Server Software:        nginx/1.22.1
Server Hostname:        aaron.axvigs.com
Server Port:            443
SSL/TLS Protocol:       TLSv1.3,TLS_AES_256_GCM_SHA384,256,256
Server Temp Key:        X25519 253 bits
TLS Server Name:        aaron.axvigs.com

Document Path:          /
Document Length:        44878 bytes

Concurrency Level:      1
Time taken for tests:   0.092 seconds
Complete requests:      1
Failed requests:        0
Total transferred:      45304 bytes
HTML transferred:       44878 bytes
Requests per second:    10.87 [#/sec] (mean)
Time per request:       92.023 [ms] (mean)
Time per request:       92.023 [ms] (mean, across all concurrent requests)
Transfer rate:          480.77 [Kbytes/sec] received

Connection Times (ms)
             min  mean[+/-sd] median   max
Connect:        3    3   0.0      3       3
Processing:    89   89   0.0     89      89
Waiting:       86   86   0.0     86      86
Total:         92   92   0.0     92      92
 

45KB of data transfered.  This doesn't really match what Firefox shows in F12 (18KB transfered, 62KB size), so I am confused.  Anyways, now let's run it 5000 times.

user@hostname:~/$ ab -n 5000 -c 50 https://aaron.axvigs.com/
This is ApacheBench, Version 2.3 <$Revision: 1923142 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking aaron.axvigs.com (be patient)
Completed 500 requests
Completed 1000 requests
<snip>
Completed 5000 requests
Finished 5000 requests

Server Software:        nginx/1.22.1
Server Hostname:        aaron.axvigs.com
Server Port:            443
SSL/TLS Protocol:       TLSv1.3,TLS_AES_256_GCM_SHA384,256,256
Server Temp Key:        X25519 253 bits
TLS Server Name:        aaron.axvigs.com

Document Path:          /
Document Length:        44878 bytes

Concurrency Level:      50
Time taken for tests:   189.289 seconds
Complete requests:      5000
Failed requests:        3
  (Connect: 0, Receive: 0, Length: 3, Exceptions: 0)
Non-2xx responses:      3
Total transferred:      226385060 bytes
HTML transferred:       224255867 bytes
Requests per second:    26.41 [#/sec] (mean)
Time per request:       1892.886 [ms] (mean)
Time per request:       37.858 [ms] (mean, across all concurrent requests)
Transfer rate:          1167.95 [Kbytes/sec] received

Connection Times (ms)
             min  mean[+/-sd] median   max
Connect:        2  270 1141.2      3   35655
Processing:    85 1491 2484.0   1450   60003
Waiting:       83 1488 2484.0   1448   60003
Total:         88 1761 2747.3   1473   63081

Percentage of the requests served within a certain time (ms)
 50%   1473
 66%   1519
 75%   1563
 80%   1614
 90%   2281
 95%   3852
 98%   7524
 99%  15358
100%  63081 (longest request)
 

Three VMs are involved in servicing the requests to my desktop PC.

• Application server
  • nginx/fast-cgi/php-fpm
  • Physical host was Dell Optiplex SFF with Intel i7-8700 (6 cores 3.2GHz).
  • 2 CPUs, runs at 50% during the test.  Possible single-threaded issue here.
  • 1GiB RAM, 80% used.  No change when testing starts.
• Database server
  • Physical host was QOTOM Everything 1U server with Intel Atom C3758R (8 cores 2.4GHz).
  • 3 CPUs, runs at 80-100% during the testing and 5-10% normally.
  • 16GiB RAM.  Disk activity doesn't change so presumably the relevant parts of the database are entirely in RAM.
  • Network traffic jumps up to 25MB/s (200Mbps) according to Proxmox.
• Reverse proxy server
  • nginx
  • Physical host was Supermicro E300 with Intel Atom C3558 (4 cores @2.2GHz).
  • 2 CPUs, runs at 15 percent during testing and 2% normally.
  • 512MiB RAM, no change in usage for testing.
  • 1.5MB/s of network bandwidth.

Yeah the physical hosts are nothing amazing.  They aren't heavily loaded by other VMs at least.  5-10% average CPU usage.

It appears to be limited by the database server.  This normally sits at 5-10% CPU usage, but runs at 100% during the test.  28 somethings per second seems alright.

Testing with some software that actually loads all the page resources is on my to-do list.

I recently purchased a used SuperMicro SuperServer E300-9A-4C because I had come upon a bunch of 64GB ECC RDIMM sticks from the e-waste pile and supposedly it could handle four of them.  This makes for a power efficient way to get 256GB of RAM in a homelab node, although it is a bit weak on CPU at 4 cores.  I would have preferred a SYS-5019A-FTN4 with 8 cores but couldn't find used ones for sale.

The used server from eBay of course did not come with a power supply so I had to order one and wait a while.  Finally that arrived so I stuffed in 4x Samsung M393A8G40AB2CWE sticks and booted up the Proxmox installer.  After a very long wait for memory training or something, it would get to the first screen where I could choose graphical or terminal install.  And then after choosing either one I would get the "Loading initial ramdisk" message and then it would freeze at a black screen.

Eventually I found some errors in the IPMI's Server Health Log.

Uncorrectable ECC @ DIMMB2 - Assertion

Sometimes it would be DIMMB1 too.  I swapped out both B1 and B2 and still had the same issues.  Then I ran the memtest86 program via the Proxmox installer's Advanced Options and sure enough it failed after less than a minute, right at the 130GB mark.

I swapped out for 4x 32GB SK Hynix HMA84GR7MFR4N-TF sticks and the memtest ran for a while.  Next I tried the Proxmox installer and that ran as expected too.

So 128GB of RAM in the system is good enough for me that I don't want to sink hours more into figuring out why the larger ones aren't working.  This has to be a very rare configuration for such a small server with low CPU power, so it wouldn't surprise me if there are hardware limitations.  Or I am simply not looking carefully at module speeds, timings, etc.