The latest...

Today my Midco connection was converted from cable to fiber.  It was a couple hour outage and went pretty smooth.

First the aerial run was swapped to fiber.  It is much thinner, as the tech noted that I had beefy RG-11 stuff installed before.  That terminates in a box on the outside of the house.  Then a small fiber patch cord was run in through ledger using the conduit that I previously put in for my garage network run.

The tech was very good to work with.  He said he normally does commercial installs, and was on this job because I have a business connection.  The cable routing of the patch cord in the basement was outstanding and meticulous.  Without me even asking, he followed my other cables through 4 different section of finger-duct all around the corner under the stairs.  Glorious!

My OPNsense router recognized the new connection and the Dynamic DNS service automatically updated my outside DNS entries to point to the new IP address.  This was also a win!

The new service is 300x300mbps for $85.  Previously I had 200x20mbps for $115.  Business class, because port 25 started getting blocked a few years back on the normal plans.

The day was overshadowed by the PHP vulnerabilities.  Early in the morning I deemed them worth shutting things down so I had disabled my 80 and 443 NAT rules.  Then later when I had time I disabled all the entries on my reverse proxy server, re-enabled the NAT rules, and then enabled reverse proxy entries one by one as I verified they were not affected or patched them.  Oh and also I had some trouble the day before when a couple VMs curiously spiked in CPU and became unresponsive.  So reliability has not been great in the last 24 hours!

Our third child Annette was born on August 7th and is now one month old and healthy.  For the previous children I took the three paid weeks of leave offered by my employer, but for this one I opted to try out the 12 weeks leave available through the FMLA program.  If I hadn't, hiring help seemed likely.  Now I am through with 4 weeks of it and wanted to do a look back.

Having three kids three and under of course keeps one pretty busy, even with two parents at home.  Getting an hour or two in a day for household projects or hobbies is challenging.  So getting anything like that done seems like the most notable thing about the weeks and is mostly what I write about below, but 90% of the waking hours are feeding and entertaining children.

In the hospital I read The Hitchhiker's Guide to the Galaxy--a relatively short book.

Week 1

We had Anna's mom helping us for the first week which was super nice.  It seems like the first week was all about recovering from the hospital experience.  What they do there is as pleasant as reasonable, but you don't leave feeling well-rested!

Avril was going through a rough 3 weeks of bedtime.  After 7:45pm putdown she would cry on and on no matter how many times we checked on her or "reset", often finally going down amicably at 9:30-10:00.  This put a serious dent in our late evening relaxing!  I don't recall exactly why this started or ended.

It seemed like we had some sort of doctor, chiro, or whatever follow-up appointment every day.

Week 2

Anna found her dream dining table in a rummage sale listing so we picked it up Sunday morning.  Duncan Phyfe-style table with missing leaf, and 6 chairs.  The table itself needed lots of TLC so I spent a few hours on both Sunday and Monday gluing various pieces of it back together.  It turned out pretty good!  The chairs are solid but once the kids are done making messes they will be due for a reupholstering.

A few days later Louise jumped up on the table to get some food and scratched it quite a lot with her claws.

Again almost every day some sort of appointment.  One of them was newborn photos which turned out lovely.

Week 3

I looked out the front window one morning and saw a Blackstone grill and some junky furniture on the curb.  The house across the street is a rental and sometimes the tenants just put stuff out in a "free pile" (as we call it from our boating days, when there would be a free pile in many of the cruiser-friendly marinas).  So I went over and rolled it across the street to its new home.

It needed some work, so same drill as the table gluing--a few hours sunk into it each of a few days.  It is funny how restoring a grill--normally a Sunday afternoon project--can consume all free-time for days when you have very little of it.  Anyways I tested it out with a propane tank, used an angle grinder flap-disc to take it down to bare metal, re-seasoned it, converted it to natural gas, and bought a hard cover for it.

Sometime in here I finished the second book in the HHGTTG series.

At the end of the week I attended a meeting of the Mandan Historical Society.  For the past few months I had been investigating why their website was offline and communicating with them to see if there was interest in reviving it.  After some back-and-forth, the final answer at this meeting was yes.  Since then I have been putting in some time on this occasionally.  The website had a lot of great information on it so I think is worth saving.

Week 4

On Monday we had the van in to get the lane-detection camera calibrated, since they had forgotten to do that a few weeks prior when replacing the cracked windshield.  Then that evening we got a new rock chip, and only about 1/4 mile away on the same road where the previous windshield had gotten cracked!  This one didn't spread so we "only" had to take it in on Wednesday to get the chip sealed.  While that was getting done I took Alice and Avril in the stroller for donuts a few blocks away.  Hopefully we can be done with all that window business for a while now.

Our ISP (Midco) was out for a site survey.  After years of "fiber is coming" mailers and commercials in the area, it seems like we are just days or weeks away from it finally happening.

Overall

I am happy with the choice to take this much time off.  I do miss certain aspects of work and especially the exciting projects that are probably going on with a merger that closed shortly before I left.  But the work here at home is important and also rewarding.

There doesn't appear to be any information online about Arista serial numbers containing the date of manufacture.  Extreme, Cisco, and APC are examples of ones I have seen well-discussed online.

After analyzing a fleet of ~100 Arista switches I am reasonably certain that the date is in the serial number in a manner quite similar to these other manufacturers.  Let's look at an example from ebay: WTW23100042

On some older Palo Alto firewalls such as a PA-3250, the command show counter interface ethernet1/13 will show over 100 "Detailed physical port counters read from MAC".  I have firsthand experience that some of these counters will increment for cable issues while nothing shows up in the summary RX or TX error values that are more commonly monitored.

show counter interface ethernet1/13

Interface: ethernet1/13
--------------------------------------------------------------------------------

--------------------------------------------------------------------------------
Physical port counters read from MAC:
--------------------------------------------------------------------------------
rx-broadcast                  4095113
<snip>              
--------------------------------------------------------------------------------

--------------------------------------------------------------------------------
Detailed physical port counters read from MAC:
--------------------------------------------------------------------------------                
snmpBcmCustomReceive0                    0                                
<snip>                  
snmpBcmReceivedPkts1024to1518Octets           4132098419                                
snmpBcmReceivedPkts128to255Octets           10411057465                                
snmpBcmReceivedPkts1519to2047Octets           0                                
snmpBcmReceivedPkts2048to4095Octets           0                                
snmpBcmReceivedPkts256to511Octets           5239213697                                
snmpBcmReceivedPkts4095to9216Octets           0                                
snmpBcmReceivedPkts512to1023Octets           264711813                                
snmpBcmReceivedPkts64Octets              1667989221                                
snmpBcmReceivedPkts65to127Octets           799363660                                
snmpBcmReceivedPkts9217to16383Octets           0                                
<snip>                              
snmpBcmRxFecCorrectable                  268435456                                
snmpBcmRxFecUncorrectable                3050                                
snmpBcmRxLlfcPrimary                     0                                
snmpBcmRxLlfcSecondary                   0                                
snmpBcmRxPFCControlFrame                 0      
<snip>
snmpDot3StatsFCSErrors                   304106033                                
snmpDot3StatsFrameTooLongs               0                                
snmpDot3StatsInRangeLengthError           0                                
snmpDot3StatsInternalMacReceiveErrors           26440618                                
snmpDot3StatsInternalMacTransmitErrors           0                                
snmpDot3StatsLateCollisions              0                                
snmpDot3StatsMultipleCollisionFrames           0                                
snmpDot3StatsSQETTestErrors              0                                
snmpDot3StatsSingleCollisionFrames           0                                
snmpDot3StatsSymbolErrors                25382788    
snmpEtherRxOversizePkts                  0                                
snmpEtherStatsBroadcastPkts              2767864081                                
snmpEtherStatsCRCAlignErrors             304106033                                

The same values can generally be seen in "show system state filter-pretty sys.s1.p13.detail".  Values from a different firewall, still a PA-3250:

show system state filter-pretty sys.s1.p13.detail

sys.s1.p13.detail: { 
  snmpBcmCustomReceive0: 0x0, 
<snip>
  snmpBcmReceivedPkts1024to1518Octets: 0xf64ad173, 
  snmpBcmReceivedPkts128to255Octets: 0x26c8c2728, 
  snmpBcmReceivedPkts1519to2047Octets: 0x0, 
  snmpBcmReceivedPkts2048to4095Octets: 0x0, 
  snmpBcmReceivedPkts256to511Octets: 0x138480e9b, 
  snmpBcmReceivedPkts4095to9216Octets: 0x0, 
  snmpBcmReceivedPkts512to1023Octets: 0xfc72e85, 
  snmpBcmReceivedPkts64Octets: 0x636b8b91, 
  snmpBcmReceivedPkts65to127Octets: 0x2fa5524c, 
  snmpBcmReceivedPkts9217to16383Octets: 0x0, 
<snip>
  snmpBcmRxFecCorrectable: 0x10000000, 
  snmpBcmRxFecUncorrectable: 0xbea, 
  snmpBcmRxLlfcPrimary: 0x0, 
  snmpBcmRxLlfcSecondary: 0x0, 
  snmpBcmRxPFCControlFrame: 0x0, 
<snip>
  snmpDot3StatsFCSErrors: 0x0, 
  snmpDot3StatsFrameTooLongs: 0x0, 
  snmpDot3StatsInRangeLengthError: 0x0, 
  snmpDot3StatsInternalMacReceiveErrors: 0x0, 
  snmpDot3StatsInternalMacTransmitErrors: 0x0, 
  snmpDot3StatsLateCollisions: 0x0, 
  snmpDot3StatsMultipleCollisionFrames: 0x0, 
  snmpDot3StatsSQETTestErrors: 0x0, 
  snmpDot3StatsSingleCollisionFrames: 0x0, 
  snmpDot3StatsSymbolErrors: 0x0, 
  snmpEtherRxOversizePkts: 0x0, 
  snmpEtherStatsBroadcastPkts: 0x4185d8, 
  snmpEtherStatsCRCAlignErrors: 0x0, 
<snip>
}

This has been reduced in the output of both commands on the newer firewalls such as the PA-1420.  Everything that has a value of zero seems to be hidden.  There is a KB article that acknowledges this hiding.

So the guidance is to monitor to see if anything has been revealed.  It is difficult to do that if you don't know what you are looking for!  I will document a few that I have found.

From an actual firewall I manage:

show counter interface ethernet1/19

--------------------------------------------------------------------------------
Detailed physical port counters read from MAC:
--------------------------------------------------------------------------------                    
rx packets 64 bytes                      1798167                                
rx packets 65 to 127 bytes               43692423                                
rx packets 128 to 255 bytes              3654577895                                
rx packets 256 to 511 bytes              365404112                                
rx packets 512 to 1023 bytes             185088270                                
rx packets 1024+ bytes                   1167646688                                
packets with bad crc                     5                                
mac receive error                        2                                
bad_octets_rcv                           1320                                    
--------------------------------------------------------------------------------

show system state filter-pretty sys.s1.p19.detail

sys.s1.p19.detail: { 
 bad_crc: 0x5, 
 bad_octets_rcv: 0x528, 
 mac_rcv_error: 0x2, 
 pkts1024tomax_octets: 0x45b0fbe0, 
 pkts128to255_octets: 0xda152b2a, 
 pkts256to511_octets: 0x15cdc3b2, 
 pkts512to1023_octets: 0xb0b043d, 
 pkts64_octets: 0x1b74e8, 
 pkts65to127_octets: 0x29b21f5, 
}

From this LIVEcommunity post:

flow control transmitted 10954
receive fifo overrun 29127
packets with bad crc 1
bad_octets_rcv 1522

Another LIVEcommunity post:

sys.s1.p5.detail: { 'bad_crc': 0x157352, 'fragments_pkts': 0x1247, 'mac_rcv_error': 0x228b09, 

From a KB article:

sys.s1.p2.detail: { 'oversize_pkts': 0x1208cc42,

So we could compile a table like this:

I have not yet found a source of all possible values that one might want to watch for.

This article claims that the counter (Hardware interface counter > receive incoming errors) shows the sum of mac_rcv_error and rcv_fifo_overrun.

Microsoft Access was a great (in some ways) tool that allowed people to create CRUD apps.  Very useful for business, and some computer aficionados even dabbled for their personal use.

The use of Access faded over time for various reasons.  Later versions tried to bridge the gap to web apps.

Now we have LLM tools that allow people with similar computer skills to create web apps of similar functionality/complexity.  Let's call it CRUD.  This goes beyond it being a CRUD app--it is just CRUD.

I imagine CRUD will be about as useful as Access was.  Why spend $40/employee/month for some light-duty ERP/CRM type thing that you have to meld the business too? The dashing hero entrepreneur leader of the business can crank out something custom in three weekends!  This is naive thinking in some ways but it happens.  I have to respect the "keep it small/local" implication of it.  Not every business needs to send 10% of profits and 100% of customer data off to a big city software shop that will eventually have a multi-million customer data breach.

Anyways--let's compare Access and CRUDs across a few different?

Learning

Access was easy--form building, database schema, and reporting tools all built in to one program.

A LLM-generated CRUD is likely to be a web app with a lot of different technologies in play.  HTML, CSS, JS, certs, ORMs, SQL, web servers, etc.  LLMs really help with the speed of building, but one must still understand how it all works together.  LLMs are also pretty good at teaching.

Security

Security was an afterthought with typical Access use.  The file itself could be password protected.  No user roles.

A CRUD is unlikely to have robust access control.  There will be temptations to roll one's own by doing fancy things like MD5 hash of passwords.  But maybe it would use a pre-built authentication framework like is available in Django.  There are good options.

Performance

It was common to store the Access file on a network share.  Then any latency would cause performance issues.

In the world of CRUD, one is almost certainly going to build a web app and it will handle latency just fine.

Data integrity

Access would often crash due to network performance issues, and occasionally this would result in corruption. Hopefully you had a backup.  I think multi-user access sometimes caused disasters too.

CRUD is likely to be built using a database, and probably an ORM.  This type of boilerplate code is what LLMs excel at.  SQLite or PostgreSQL/MySQL are going to be the backend in most cases.  Data is not likely to get corrupted due to computer or network issues.  BUT bad code could really mess up a database.  Backups are still your friend.

Bus factor

If the Access wizard disappears one day, it is not likely that a small business has someone who can truly move the Access system along.

I think the same is true for a CRUD.  Someone has to have a basic understanding of devops to properly maintain and deploy the app.

Scaling

There are rumors of Access databases that have been migrated into Microsoft SQL Server and web front ends created.  And I think later versions of Access facilitated that.  It is probably easier to start over and import data.

Odds are good that a CRUD is already a web app with a proper database back end and can be taken over by a moderately experienced devops person once the business grows to a certain size.

Extensibility

I don't think Access had much to facilitate interaction with outside systems.  In a CRUD you are going to have many option available.

Summary

I think CRUD will fill a similar role as Access used to.  But it will not reach the popularity of Access because of the proliferation of SaaS competitors (themselves built on LLMs).  Gains and risks to the business are about the same.  It is a powerful tool and easy to wield improperly.

It was nice to get my shiny new theme mostly figured out last week.  The blog is more mine than it has ever been.

The thing that I wanted next was a way to browse posts from the archive, by year and month.  I recall trying this a few years back, finding it difficult, and giving up.  I tried again for a few hours over a few days recently, and it is just baffling.  Some people claim it can be done with a View, and it seems possible, but I have tried almost everything and it is not even close.

Examples from the Internet (people want this!):

• https://drupal.stackexchange.com/questions/215419/blog-monthly-archive-block-list-of-posts-grouped-by-year-then-month
• https://drupal.stackexchange.com/questions/238838/views-archive-page-showing-year-and-months-of-article-content-type
• https://drupal.stackexchange.com/questions/205581/how-to-show-year-months-in-archive-block
• https://drupal.stackexchange.com/questions/275848/view-of-blog-posts-grouped-by-year
•  

The goal is this, with years being able to expand/contract:

• 2025 (15)
  • 12 (3)
  • 11 (5)
  • 10 (3)
  • 09 (4)
• 2024 (6)
  • 09 (2)
  • 08 (1)
  • 07 (1)
  • 05 (3)
• 2023 (8)
  • 12 (4)
  • 11 (1)
  • 10 (3)

I believe this is an easy thing to do in Wordpress, as I have seen it occasionally around the Web.  Possibly stock.  It is impossible to find if you are looking for an example though!  There is an obsolete Drupal module WP Blog which claims to have duplicated this view.  In Drupal, the stock Archive module is a thing, but is kind of limited.  It looks like this:

• October 2023 (3)
• November 2023 (1)
• December 2023 (8)
• May 2024 (3)
• July 2024 (1)
• etc. with pagination menu at the bottom

Limited indeed.  Perhaps the ordering can be reversed to have most recent at the top.  And one can (duplicate it and) customize it to some great extent.  The years part of the good example above can be done.  It took me a couple hours just to get that.  I wrote that up with specifics for future-folk as the answer to someone else's question.  This was exactly what they wanted.  But I have always dreamed of having the months too.

So after hours more trying to get those months to show up right, it seemed impossible to do with available Views functionality.  I investigated using the Views field view module but that seemed like a path to frustration.  The years as one view, and under each year its own view of the months.  Probably possible, but a terrible journey.

There are rumors that this can be done with Views accordion.  Also a path to hell IMHO.  There is no way that would turn out in just the right simple way.  I tried, briefly.

Finally I just decided to try a custom module.  LLMs should at least give me a fighting chance.

I had some terrible adventures with not being able to get the prototype plugin to load on my production blog.  There was confusion between Annotation and Attribute techniques for decorating the block class.  Eventually I realized it wasn't ideal to have root:root ownership on all the files for the whole blog.  After I changed that to www-data:www-data then mysteriously there were schema updates that Drupal wanted to apply to the database.  Wow I guess my site was really messed up by those permissions?  At some point in here I did things the right way and set up a development environment on my desktop using ddev.  That's a pretty slick stack--great experience.

So I got the module mostly developed using that dev instance.  LLMs saved countless hours of time.  It is such a relief to not have to continuously fight specific syntax details.  I basically don't have to relearn PHP, knowing generic programming concepts gets me 98% of the way.  Hopefully I know enough about Drupal and security that the module doesn't open me up to some epic hacking.

Anyways I had the module working on my dev site, but then I STILL wasted literal hours trying to get this custom module to deploy on my production blog.  The module would install but the block would not show up in the Place Block screen and the views would not show up in...Views.  Finally I realized that I still had an early prototype of the module in web/modules/custom/archive_tree, and I was installing the nearly finished version using composer (from my git repo) which was putting it in web/modules/contrib/archive_tree.  Dangit!

Anyways, the module does exactly what I wanted it to do.

https://git.axvig.com/aaron.axvig/drupal-module-archive-tree

I wanted to see what my mostly stock Drupal server was capable of.  For this first test I am starting with a simple homepage load. I used the ab tool (Apache Bench).  It couldn't be simpler!  But it only downloads the HTML, not the rest of the assets.  Here is one page load:

user@hostname:~/$ ab -n 1 -c 1 https://aaron.axvigs.com/
This is ApacheBench, Version 2.3 <$Revision: 1923142 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking aaron.axvigs.com (be patient).....done

Server Software:        nginx/1.22.1
Server Hostname:        aaron.axvigs.com
Server Port:            443
SSL/TLS Protocol:       TLSv1.3,TLS_AES_256_GCM_SHA384,256,256
Server Temp Key:        X25519 253 bits
TLS Server Name:        aaron.axvigs.com

Document Path:          /
Document Length:        44878 bytes

Concurrency Level:      1
Time taken for tests:   0.092 seconds
Complete requests:      1
Failed requests:        0
Total transferred:      45304 bytes
HTML transferred:       44878 bytes
Requests per second:    10.87 [#/sec] (mean)
Time per request:       92.023 [ms] (mean)
Time per request:       92.023 [ms] (mean, across all concurrent requests)
Transfer rate:          480.77 [Kbytes/sec] received

Connection Times (ms)
             min  mean[+/-sd] median   max
Connect:        3    3   0.0      3       3
Processing:    89   89   0.0     89      89
Waiting:       86   86   0.0     86      86
Total:         92   92   0.0     92      92
 

45KB of data transfered.  This doesn't really match what Firefox shows in F12 (18KB transfered, 62KB size), so I am confused.  Anyways, now let's run it 5000 times.

user@hostname:~/$ ab -n 5000 -c 50 https://aaron.axvigs.com/
This is ApacheBench, Version 2.3 <$Revision: 1923142 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking aaron.axvigs.com (be patient)
Completed 500 requests
Completed 1000 requests
<snip>
Completed 5000 requests
Finished 5000 requests

Server Software:        nginx/1.22.1
Server Hostname:        aaron.axvigs.com
Server Port:            443
SSL/TLS Protocol:       TLSv1.3,TLS_AES_256_GCM_SHA384,256,256
Server Temp Key:        X25519 253 bits
TLS Server Name:        aaron.axvigs.com

Document Path:          /
Document Length:        44878 bytes

Concurrency Level:      50
Time taken for tests:   189.289 seconds
Complete requests:      5000
Failed requests:        3
  (Connect: 0, Receive: 0, Length: 3, Exceptions: 0)
Non-2xx responses:      3
Total transferred:      226385060 bytes
HTML transferred:       224255867 bytes
Requests per second:    26.41 [#/sec] (mean)
Time per request:       1892.886 [ms] (mean)
Time per request:       37.858 [ms] (mean, across all concurrent requests)
Transfer rate:          1167.95 [Kbytes/sec] received

Connection Times (ms)
             min  mean[+/-sd] median   max
Connect:        2  270 1141.2      3   35655
Processing:    85 1491 2484.0   1450   60003
Waiting:       83 1488 2484.0   1448   60003
Total:         88 1761 2747.3   1473   63081

Percentage of the requests served within a certain time (ms)
 50%   1473
 66%   1519
 75%   1563
 80%   1614
 90%   2281
 95%   3852
 98%   7524
 99%  15358
100%  63081 (longest request)
 

Three VMs are involved in servicing the requests to my desktop PC.

• Application server
  • nginx/fast-cgi/php-fpm
  • Physical host was Dell Optiplex SFF with Intel i7-8700 (6 cores 3.2GHz).
  • 2 CPUs, runs at 50% during the test.  Possible single-threaded issue here.
  • 1GiB RAM, 80% used.  No change when testing starts.
• Database server
  • Physical host was QOTOM Everything 1U server with Intel Atom C3758R (8 cores 2.4GHz).
  • 3 CPUs, runs at 80-100% during the testing and 5-10% normally.
  • 16GiB RAM.  Disk activity doesn't change so presumably the relevant parts of the database are entirely in RAM.
  • Network traffic jumps up to 25MB/s (200Mbps) according to Proxmox.
• Reverse proxy server
  • nginx
  • Physical host was Supermicro E300 with Intel Atom C3558 (4 cores @2.2GHz).
  • 2 CPUs, runs at 15 percent during testing and 2% normally.
  • 512MiB RAM, no change in usage for testing.
  • 1.5MB/s of network bandwidth.

Yeah the physical hosts are nothing amazing.  They aren't heavily loaded by other VMs at least.  5-10% average CPU usage.

It appears to be limited by the database server.  This normally sits at 5-10% CPU usage, but runs at 100% during the test.  28 somethings per second seems alright.

Testing with some software that actually loads all the page resources is on my to-do list.