Archive by year

On some older Palo Alto firewalls such as a PA-3250, the command show counter interface ethernet1/13 will show over 100 "Detailed physical port counters read from MAC".  I have firsthand experience that some of these counters will increment for cable issues while nothing shows up in the summary RX or TX error values that are more commonly monitored.

show counter interface ethernet1/13

Interface: ethernet1/13
--------------------------------------------------------------------------------

--------------------------------------------------------------------------------
Physical port counters read from MAC:
--------------------------------------------------------------------------------
rx-broadcast                  4095113
<snip>              
--------------------------------------------------------------------------------

--------------------------------------------------------------------------------
Detailed physical port counters read from MAC:
--------------------------------------------------------------------------------                
snmpBcmCustomReceive0                    0                                
<snip>                  
snmpBcmReceivedPkts1024to1518Octets           4132098419                                
snmpBcmReceivedPkts128to255Octets           10411057465                                
snmpBcmReceivedPkts1519to2047Octets           0                                
snmpBcmReceivedPkts2048to4095Octets           0                                
snmpBcmReceivedPkts256to511Octets           5239213697                                
snmpBcmReceivedPkts4095to9216Octets           0                                
snmpBcmReceivedPkts512to1023Octets           264711813                                
snmpBcmReceivedPkts64Octets              1667989221                                
snmpBcmReceivedPkts65to127Octets           799363660                                
snmpBcmReceivedPkts9217to16383Octets           0                                
<snip>                              
snmpBcmRxFecCorrectable                  268435456                                
snmpBcmRxFecUncorrectable                3050                                
snmpBcmRxLlfcPrimary                     0                                
snmpBcmRxLlfcSecondary                   0                                
snmpBcmRxPFCControlFrame                 0      
<snip>
snmpDot3StatsFCSErrors                   304106033                                
snmpDot3StatsFrameTooLongs               0                                
snmpDot3StatsInRangeLengthError           0                                
snmpDot3StatsInternalMacReceiveErrors           26440618                                
snmpDot3StatsInternalMacTransmitErrors           0                                
snmpDot3StatsLateCollisions              0                                
snmpDot3StatsMultipleCollisionFrames           0                                
snmpDot3StatsSQETTestErrors              0                                
snmpDot3StatsSingleCollisionFrames           0                                
snmpDot3StatsSymbolErrors                25382788    
snmpEtherRxOversizePkts                  0                                
snmpEtherStatsBroadcastPkts              2767864081                                
snmpEtherStatsCRCAlignErrors             304106033                                

The same values can generally be seen in "show system state filter-pretty sys.s1.p13.detail".  Values from a different firewall, still a PA-3250:

show system state filter-pretty sys.s1.p13.detail

sys.s1.p13.detail: { 
  snmpBcmCustomReceive0: 0x0, 
<snip>
  snmpBcmReceivedPkts1024to1518Octets: 0xf64ad173, 
  snmpBcmReceivedPkts128to255Octets: 0x26c8c2728, 
  snmpBcmReceivedPkts1519to2047Octets: 0x0, 
  snmpBcmReceivedPkts2048to4095Octets: 0x0, 
  snmpBcmReceivedPkts256to511Octets: 0x138480e9b, 
  snmpBcmReceivedPkts4095to9216Octets: 0x0, 
  snmpBcmReceivedPkts512to1023Octets: 0xfc72e85, 
  snmpBcmReceivedPkts64Octets: 0x636b8b91, 
  snmpBcmReceivedPkts65to127Octets: 0x2fa5524c, 
  snmpBcmReceivedPkts9217to16383Octets: 0x0, 
<snip>
  snmpBcmRxFecCorrectable: 0x10000000, 
  snmpBcmRxFecUncorrectable: 0xbea, 
  snmpBcmRxLlfcPrimary: 0x0, 
  snmpBcmRxLlfcSecondary: 0x0, 
  snmpBcmRxPFCControlFrame: 0x0, 
<snip>
  snmpDot3StatsFCSErrors: 0x0, 
  snmpDot3StatsFrameTooLongs: 0x0, 
  snmpDot3StatsInRangeLengthError: 0x0, 
  snmpDot3StatsInternalMacReceiveErrors: 0x0, 
  snmpDot3StatsInternalMacTransmitErrors: 0x0, 
  snmpDot3StatsLateCollisions: 0x0, 
  snmpDot3StatsMultipleCollisionFrames: 0x0, 
  snmpDot3StatsSQETTestErrors: 0x0, 
  snmpDot3StatsSingleCollisionFrames: 0x0, 
  snmpDot3StatsSymbolErrors: 0x0, 
  snmpEtherRxOversizePkts: 0x0, 
  snmpEtherStatsBroadcastPkts: 0x4185d8, 
  snmpEtherStatsCRCAlignErrors: 0x0, 
<snip>
}

This has been reduced in the output of both commands on the newer firewalls such as the PA-1420.  Everything that has a value of zero seems to be hidden.  There is a KB article that acknowledges this hiding.

So the guidance is to monitor to see if anything has been revealed.  It is difficult to do that if you don't know what you are looking for!  I will document a few that I have found.

From an actual firewall I manage:

show counter interface ethernet1/19

--------------------------------------------------------------------------------
Detailed physical port counters read from MAC:
--------------------------------------------------------------------------------                    
rx packets 64 bytes                      1798167                                
rx packets 65 to 127 bytes               43692423                                
rx packets 128 to 255 bytes              3654577895                                
rx packets 256 to 511 bytes              365404112                                
rx packets 512 to 1023 bytes             185088270                                
rx packets 1024+ bytes                   1167646688                                
packets with bad crc                     5                                
mac receive error                        2                                
bad_octets_rcv                           1320                                    
--------------------------------------------------------------------------------

show system state filter-pretty sys.s1.p19.detail

sys.s1.p19.detail: { 
 bad_crc: 0x5, 
 bad_octets_rcv: 0x528, 
 mac_rcv_error: 0x2, 
 pkts1024tomax_octets: 0x45b0fbe0, 
 pkts128to255_octets: 0xda152b2a, 
 pkts256to511_octets: 0x15cdc3b2, 
 pkts512to1023_octets: 0xb0b043d, 
 pkts64_octets: 0x1b74e8, 
 pkts65to127_octets: 0x29b21f5, 
}

From this LIVEcommunity post:

flow control transmitted 10954
receive fifo overrun 29127
packets with bad crc 1
bad_octets_rcv 1522

Another LIVEcommunity post:

sys.s1.p5.detail: { 'bad_crc': 0x157352, 'fragments_pkts': 0x1247, 'mac_rcv_error': 0x228b09, 

From a KB article:

sys.s1.p2.detail: { 'oversize_pkts': 0x1208cc42,

So we could compile a table like this:

I have not yet found a source of all possible values that one might want to watch for.

This article claims that the counter (Hardware interface counter > receive incoming errors) shows the sum of mac_rcv_error and rcv_fifo_overrun.

Microsoft Access was a great (in some ways) tool that allowed people to create CRUD apps.  Very useful for business, and some computer aficionados even dabbled for their personal use.

The use of Access faded over time for various reasons.  Later versions tried to bridge the gap to web apps.

Now we have LLM tools that allow people with similar computer skills to create web apps of similar functionality/complexity.  Let's call it CRUD.  This goes beyond it being a CRUD app--it is just CRUD.

I imagine CRUD will be about as useful as Access was.  Why spend $40/employee/month for some light-duty ERP/CRM type thing that you have to meld the business too? The dashing hero entrepreneur leader of the business can crank out something custom in three weekends!  This is naive thinking in some ways but it happens.  I have to respect the "keep it small/local" implication of it.  Not every business needs to send 10% of profits and 100% of customer data off to a big city software shop that will eventually have a multi-million customer data breach.

Anyways--let's compare Access and CRUDs across a few different?

Learning

Access was easy--form building, database schema, and reporting tools all built in to one program.

A LLM-generated CRUD is likely to be a web app with a lot of different technologies in play.  HTML, CSS, JS, certs, ORMs, SQL, web servers, etc.  LLMs really help with the speed of building, but one must still understand how it all works together.  LLMs are also pretty good at teaching.

Security

Security was an afterthought with typical Access use.  The file itself could be password protected.  No user roles.

A CRUD is unlikely to have robust access control.  There will be temptations to roll one's own by doing fancy things like MD5 hash of passwords.  But maybe it would use a pre-built authentication framework like is available in Django.  There are good options.

Performance

It was common to store the Access file on a network share.  Then any latency would cause performance issues.

In the world of CRUD, one is almost certainly going to build a web app and it will handle latency just fine.

Data integrity

Access would often crash due to network performance issues, and occasionally this would result in corruption. Hopefully you had a backup.  I think multi-user access sometimes caused disasters too.

CRUD is likely to be built using a database, and probably an ORM.  This type of boilerplate code is what LLMs excel at.  SQLite or PostgreSQL/MySQL are going to be the backend in most cases.  Data is not likely to get corrupted due to computer or network issues.  BUT bad code could really mess up a database.  Backups are still your friend.

Bus factor

If the Access wizard disappears one day, it is not likely that a small business has someone who can truly move the Access system along.

I think the same is true for a CRUD.  Someone has to have a basic understanding of devops to properly maintain and deploy the app.

Scaling

There are rumors of Access databases that have been migrated into Microsoft SQL Server and web front ends created.  And I think later versions of Access facilitated that.  It is probably easier to start over and import data.

Odds are good that a CRUD is already a web app with a proper database back end and can be taken over by a moderately experienced devops person once the business grows to a certain size.

Extensibility

I don't think Access had much to facilitate interaction with outside systems.  In a CRUD you are going to have many option available.

Summary

I think CRUD will fill a similar role as Access used to.  But it will not reach the popularity of Access because of the proliferation of SaaS competitors (themselves built on LLMs).  Gains and risks to the business are about the same.  It is a powerful tool and easy to wield improperly.

It was nice to get my shiny new theme mostly figured out last week.  The blog is more mine than it has ever been.

The thing that I wanted next was a way to browse posts from the archive, by year and month.  I recall trying this a few years back, finding it difficult, and giving up.  I tried again for a few hours over a few days recently, and it is just baffling.  Some people claim it can be done with a View, and it seems possible, but I have tried almost everything and it is not even close.

Examples from the Internet (people want this!):

• https://drupal.stackexchange.com/questions/215419/blog-monthly-archive-block-list-of-posts-grouped-by-year-then-month
• https://drupal.stackexchange.com/questions/238838/views-archive-page-showing-year-and-months-of-article-content-type
• https://drupal.stackexchange.com/questions/205581/how-to-show-year-months-in-archive-block
• https://drupal.stackexchange.com/questions/275848/view-of-blog-posts-grouped-by-year
•  

The goal is this, with years being able to expand/contract:

• 2025 (15)
  • 12 (3)
  • 11 (5)
  • 10 (3)
  • 09 (4)
• 2024 (6)
  • 09 (2)
  • 08 (1)
  • 07 (1)
  • 05 (3)
• 2023 (8)
  • 12 (4)
  • 11 (1)
  • 10 (3)

I believe this is an easy thing to do in Wordpress, as I have seen it occasionally around the Web.  Possibly stock.  It is impossible to find if you are looking for an example though!  There is an obsolete Drupal module WP Blog which claims to have duplicated this view.  In Drupal, the stock Archive module is a thing, but is kind of limited.  It looks like this:

• October 2023 (3)
• November 2023 (1)
• December 2023 (8)
• May 2024 (3)
• July 2024 (1)
• etc. with pagination menu at the bottom

Limited indeed.  Perhaps the ordering can be reversed to have most recent at the top.  And one can (duplicate it and) customize it to some great extent.  The years part of the good example above can be done.  It took me a couple hours just to get that.  I wrote that up with specifics for future-folk as the answer to someone else's question.  This was exactly what they wanted.  But I have always dreamed of having the months too.

So after hours more trying to get those months to show up right, it seemed impossible to do with available Views functionality.  I investigated using the Views field view module but that seemed like a path to frustration.  The years as one view, and under each year its own view of the months.  Probably possible, but a terrible journey.

There are rumors that this can be done with Views accordion.  Also a path to hell IMHO.  There is no way that would turn out in just the right simple way.  I tried, briefly.

Finally I just decided to try a custom module.  LLMs should at least give me a fighting chance.

I had some terrible adventures with not being able to get the prototype plugin to load on my production blog.  There was confusion between Annotation and Attribute techniques for decorating the block class.  Eventually I realized it wasn't ideal to have root:root ownership on all the files for the whole blog.  After I changed that to www-data:www-data then mysteriously there were schema updates that Drupal wanted to apply to the database.  Wow I guess my site was really messed up by those permissions?  At some point in here I did things the right way and set up a development environment on my desktop using ddev.  That's a pretty slick stack--great experience.

So I got the module mostly developed using that dev instance.  LLMs saved countless hours of time.  It is such a relief to not have to continuously fight specific syntax details.  I basically don't have to relearn PHP, knowing generic programming concepts gets me 98% of the way.  Hopefully I know enough about Drupal and security that the module doesn't open me up to some epic hacking.

Anyways I had the module working on my dev site, but then I STILL wasted literal hours trying to get this custom module to deploy on my production blog.  The module would install but the block would not show up in the Place Block screen and the views would not show up in...Views.  Finally I realized that I still had an early prototype of the module in web/modules/custom/archive_tree, and I was installing the nearly finished version using composer (from my git repo) which was putting it in web/modules/contrib/archive_tree.  Dangit!

Anyways, the module does exactly what I wanted it to do.

https://git.axvig.com/aaron.axvig/drupal-module-archive-tree

I wanted to see what my mostly stock Drupal server was capable of.  For this first test I am starting with a simple homepage load. I used the ab tool (Apache Bench).  It couldn't be simpler!  But it only downloads the HTML, not the rest of the assets.  Here is one page load:

user@hostname:~/$ ab -n 1 -c 1 https://aaron.axvigs.com/
This is ApacheBench, Version 2.3 <$Revision: 1923142 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking aaron.axvigs.com (be patient).....done

Server Software:        nginx/1.22.1
Server Hostname:        aaron.axvigs.com
Server Port:            443
SSL/TLS Protocol:       TLSv1.3,TLS_AES_256_GCM_SHA384,256,256
Server Temp Key:        X25519 253 bits
TLS Server Name:        aaron.axvigs.com

Document Path:          /
Document Length:        44878 bytes

Concurrency Level:      1
Time taken for tests:   0.092 seconds
Complete requests:      1
Failed requests:        0
Total transferred:      45304 bytes
HTML transferred:       44878 bytes
Requests per second:    10.87 [#/sec] (mean)
Time per request:       92.023 [ms] (mean)
Time per request:       92.023 [ms] (mean, across all concurrent requests)
Transfer rate:          480.77 [Kbytes/sec] received

Connection Times (ms)
             min  mean[+/-sd] median   max
Connect:        3    3   0.0      3       3
Processing:    89   89   0.0     89      89
Waiting:       86   86   0.0     86      86
Total:         92   92   0.0     92      92
 

45KB of data transfered.  This doesn't really match what Firefox shows in F12 (18KB transfered, 62KB size), so I am confused.  Anyways, now let's run it 5000 times.

user@hostname:~/$ ab -n 5000 -c 50 https://aaron.axvigs.com/
This is ApacheBench, Version 2.3 <$Revision: 1923142 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking aaron.axvigs.com (be patient)
Completed 500 requests
Completed 1000 requests
<snip>
Completed 5000 requests
Finished 5000 requests

Server Software:        nginx/1.22.1
Server Hostname:        aaron.axvigs.com
Server Port:            443
SSL/TLS Protocol:       TLSv1.3,TLS_AES_256_GCM_SHA384,256,256
Server Temp Key:        X25519 253 bits
TLS Server Name:        aaron.axvigs.com

Document Path:          /
Document Length:        44878 bytes

Concurrency Level:      50
Time taken for tests:   189.289 seconds
Complete requests:      5000
Failed requests:        3
  (Connect: 0, Receive: 0, Length: 3, Exceptions: 0)
Non-2xx responses:      3
Total transferred:      226385060 bytes
HTML transferred:       224255867 bytes
Requests per second:    26.41 [#/sec] (mean)
Time per request:       1892.886 [ms] (mean)
Time per request:       37.858 [ms] (mean, across all concurrent requests)
Transfer rate:          1167.95 [Kbytes/sec] received

Connection Times (ms)
             min  mean[+/-sd] median   max
Connect:        2  270 1141.2      3   35655
Processing:    85 1491 2484.0   1450   60003
Waiting:       83 1488 2484.0   1448   60003
Total:         88 1761 2747.3   1473   63081

Percentage of the requests served within a certain time (ms)
 50%   1473
 66%   1519
 75%   1563
 80%   1614
 90%   2281
 95%   3852
 98%   7524
 99%  15358
100%  63081 (longest request)
 

Three VMs are involved in servicing the requests to my desktop PC.

• Application server
  • nginx/fast-cgi/php-fpm
  • Physical host was Dell Optiplex SFF with Intel i7-8700 (6 cores 3.2GHz).
  • 2 CPUs, runs at 50% during the test.  Possible single-threaded issue here.
  • 1GiB RAM, 80% used.  No change when testing starts.
• Database server
  • Physical host was QOTOM Everything 1U server with Intel Atom C3758R (8 cores 2.4GHz).
  • 3 CPUs, runs at 80-100% during the testing and 5-10% normally.
  • 16GiB RAM.  Disk activity doesn't change so presumably the relevant parts of the database are entirely in RAM.
  • Network traffic jumps up to 25MB/s (200Mbps) according to Proxmox.
• Reverse proxy server
  • nginx
  • Physical host was Supermicro E300 with Intel Atom C3558 (4 cores @2.2GHz).
  • 2 CPUs, runs at 15 percent during testing and 2% normally.
  • 512MiB RAM, no change in usage for testing.
  • 1.5MB/s of network bandwidth.

Yeah the physical hosts are nothing amazing.  They aren't heavily loaded by other VMs at least.  5-10% average CPU usage.

It appears to be limited by the database server.  This normally sits at 5-10% CPU usage, but runs at 100% during the test.  28 somethings per second seems alright.

Testing with some software that actually loads all the page resources is on my to-do list.