pfSense Router/Firewall Install

The RAM I mentioned in the last post arrived. Surprisingly, it didn't sit around the house for months. Instead, I installed it within 30 minutes of receipt and had the network back up and running in about 20 minutes after restarting a few switches and APs.

This doubles the machine's memory from four sticks of 512MB 533MHz (2GB) to two sticks of 2GB 800MHz (4GB) for about $16. There was a slight CPU spike (100% load for about ten seconds) upon pfSense starting up, but it quickly stabilized and has been running smoothly for hours. I'm hoping this is just a startup artifact, but if time permits I'll see what I can find in the logs.

8GB seems overkill for this project, so while I initially intended to fill the now open RAM slots, I'm going to see how things run and only add more RAM if necessary. 

Prior to shutting down for RAM installation, the system had been online and running smoothly for 8 days. This has got me thinking about the status of various critical components (mostly things with capacitors, like power supplies, MOBO). I suppose I'll cross that bridge when necessary, but I'd at least like to think about it so I'm not caught off guard. Maybe harvest cheap/free  PSUs in the meantime.

I'm going to review project logs, take some notes, and hopefully wrap this project up in the next post or two. Thanks for following along so far.

My goodness, it has been a year, and not in the melodramatic sense. I mean a literal year has passed since I last posted, though if any year were worthy of melodrama it would indeed be last year.

Nonsense aside, this build has been my home network's primary router for the past week. So far so good, though there have been a few hiccups.

I just ordered double the RAM and am hoping that will be sufficient to smooth out what seemed to be memory bottlenecks, though I should really go through the logs to see if I can glean more insight.

I decided to skip the BSD jails unless I can get at least 8GB of RAM working (16 seems like a possible max as it's a DDR2 machine on the cusp of DDR3, so 4GB memory units are both expensive and of dubious quality, thus 2GB units are far more practical). Instead I'll keep running the controller software on the old laptop and look into switching it to my Debian server.

Pfsense is recognizing the network card ports accurately as 1000baseT Full Duplex, but I should really test local speeds to verify functionality. To do that, I'll likely use iPerf.

More coming soon hopefully.

It's been awhile, and for good reason - but I won't get into that here.

Shortly after my last log, I received the HDD adapter mounts and installed a 250GB HDD salvaged from my first (and only) MacBook Pro also circa 2007 (system board died, R.I.P. - now tempted to resurrect this in the name of Louis Rossmann) I got that mounted, connected, etc. and it only required a little bit of finessing to get the cables to play nice. They're still a little tighter than I'd like but it'll have to do for now.

I also looked into the USB header connections and problem was clear: the header pin sleeves were loose. I firmly reseated the sleeves and the internal USB worked fine ever since.

At least, till tonight, when I removed it altogether and reinstalled pfSense to the HDD. As with the first install I used ZFS. I first copied my old configs by exporting the xml backups via the webGUI, loaded them to a fat32 USB with a partition table as outlined here.

Basically, you have two options: have the config available on USB at install time, or do it subsequently during any boot. Since I struggled (for reasons still unclear to me) to find the above linked documentation until after the install, I did the latter.

The install-time config requires the config to be located at /conf/config.xml while the post-install config requires either /conf/config.xml or /config.xml - thinking about it now, I guess that makes sense - but it tripped me up for one boot cycle until I reread the docs.

In any case, I got the config restore to work within a few minutes and as if that wasn't enough I upgraded to 2.4.5 (via the webGUI). All in all the above took about 20 minutes to get back up and running.

The stupidest thing about all of this is that almost everything described above was simply so that I could reclaim that 32GB USB (to use as a Live Multiboot Utility/OS tool).

I also picked up two 20" monitors ($30 shipped!) and a basic but NIB gaming keyboard (for $7!) for my home office, so now I can dedicate the old KVM setup to this box for local VGA login.

The last few things to try before I swap this thing out with my current router/AP:

1. Partition the HDD - I think there are already partitions for OS and SWAP, but it would be nice to have some isolation for logs and other files. I'll probably do this with a live USB GParted but I'm tempted to do it via SSH. 
2. Try to get remote VPN working - though testing this will likely be tricky due to the stay-in-place stuff. I'm thinking about doing this via mobile. May have to wait.
3. Setup and test SSH This was quick and easy.
4. See if I can get Jails working (with zfs) to run a Unifi controller, though I cringe at the MongoDB. My concern here is typically the Unifi controller should be on the same switch as the AP - but in my case it would be up one level, same as the router itself - but I've been thinking I might be able to assign an interface to that jail and connect it to the switch... I feel like I'm getting in over my head with this one. Right now I'm running said controller on the Windows 7 laptop that replaced my aforementioned dead MBP. I would love to be powering one less device if possible and the uptime overlap would make this an obvious choice. I've alternately considered installing FreeBSD and pfSense on top, plus jails/bhyve, but I'll try the above first.

Last week I got a local (for now) VPN set up between the user subnet and the admin ui subnet. I believe that I could simply change the firewall rules but I'd rather keep things as secure and simple as possible. That said, I really should test that just to make sure I'm correctly understanding the configuration. 

I also ordered a full height adapter bracket for the i350-T4, as I had to remove the included shorty for it to seat properly in the PCIe slot. This means that any movement of the attached ethernet cables may jar the card loose - not good. It should be here within a week or so. I returned the original PT card and received a refund. 

I also ordered a 3.5 to 2.5 hdd adapter 2-pack to have some decent storage, and I'm looking for an El Cheapo or free monitor to mount on my rack with my currently in disuse kvm (though not rack mount). I have a couple local and ebay leads almost within budget, though at this point the scope has expanded a bit. I'm leaning towards being lenient with this build by skimping on other projects. 

Finally, I've been considering the possibility that many of the issues that I had been attributing to the USB or Dupont to USB adapter may instead have been caused by the old NIC. I'll probably power everything down and try to boot again with the USB mounted internally.

Once the machine is physically sound, the next phase is swapping the pfSense machine with the current router/AP and putting it in bridge mode exclusively as an AP. I'd eventually like to flash it and my older DIR-655 with OpenWRT and DD-WRT but until all this quarantine business is over, I can't risk the wireless being down. Ideally that would allow for full VLAN options across both repurposed devices, but I haven't been able to confirm support as even the flashing instructions are sketchy at best. That might have to be its own hackaday project. Till next time. 

Since I already have a router in place, I needed a way to test pfSense's routing abilities without disturbing existing DHCP assignments or future DHCP client requests.

I'm fairly new at this - so anyone reading - please let me know if there's a better/easier way to do this. I'll describe my current setup and methodology below.

My Linksys router/AP allows admin to set a custom local IP address for management. By default, you get the standard 192.168.1.1/24 but it does allow up to a /16 CIDR subnet mask. Users, DHCP or manually assigned, are restricted to the 192.168 range, plus whatever the subnet mask allows.

The most straightforward way I can think of to isolate my pfSense machine but still have internet seems to be putting it both its WAN and LANs on their own subnets. I've enabled DHCP on the (so far) single LAN port. This avoids messy DHCP IP range assignment and will eventually give me a clean way to test all the interfaces on the card.

I'm then able to connect to the router/AP's subnet (and admin UI). If I connect to LAN via ethernet ( to usb adapter) that lets me access both the router/AP's admin UI and the pfSense admin UI.

All of the above is working well.

Next is to look into firewall stuff - I'm thinking that's blocking pfSense admin UI from wifi. It would be nice to administer both via wifi.

At the time of my last posting, I had been hitting this really weird kernel panic every time I'd connect the ethernet cable to the WAN interface. Sometimes it would happen immediately, sometimes within several seconds, sometimes not for 20 seconds. But it would happen, always upon WAN connection.

So, I tried:

1. Updating the BIOS
2. Disabling ACPI at pfSense boot - No option to do this via BIOS. Even for such an old heap, the system board's firmware is pretty garbage. It has Cool'n'Quiet. Always disable C'n'Q.
3. Disabling all unnecessary devices via BIOS (including onboard NIC and all floppy support)
4. Adding more memory - this finally worked, perhaps because of the BIOS update. But I think I also had a bad stick in my junk pile. Now at 2 meager GB. I'm pretty certain this thing can be maxed out around 12GB (users have reported 3x4GB sticks work, but adding a fourth doesn't)
5. Altering settings permanently via /boot/loader.conf.local:
   1. hint.acpi.0.disabled="1" (see #2 above)
   2. reviewed USB boot issues here - nothing really seemed to match.
   3. Tuning the network
      1. kern.ipc.nmbclusters="1000000" (this sounded exactly like what was happening)
      2. hw.em.fc_setting=0 (disabled flow control - for PT/em-based card)
6. Changing the IP address and releasing the DHCP reservation (was connecting WAN via a subnetwork through my existing router)

The new NIC arrived today, so I thought what the hell. Sure as shit, popped it in and it works straight out of the new old box. So far.

I still need to configure the LAN but I have web admin configuration access once again and the thing seems to be pretty stable.

20200222 - Saturday

The i350-T4 is ordered - shipping status unclear.

I was under the impression that the PT shouldn't need drivers - that FreeBSD should just run the card as expected; I'm starting to doubt that; despite pfSense recognizing the card, it seems to hang when receiving packets on WAN.

20200223 - Sunday

I backed up the 2.5" drive to my laptop

20200224 - Monday

I backed up the 2.5" drive to the old Dell 3.5" - but it's clear that something is wrong with the latter. Once I can back up both to a different drive I'll feel more comfortable wiping the 2.5" HDD for the potential pfSense install.

I also found some good info on troubleshooting possible boot/NIC issues. I posted those on the TODOs log.

Follower @weekleyj has pointed out some helpful info, specifically that it might be worth installing 2.3.x and upgrading from there.

https://www.netgate.com/blog/pfsense-2-4-0-release-now-available.html#important-information

1. [x] Try reseating both ends of the cable the Dupont connector and the USB itself - no improvement
2. [x] Try changing USB header positions - no improvement
3. [x] Update config and test - failed
4. [x] Collect TODOs
5. [x] Look into BIOS/Boot Issues as described here. - 20200227
   1. [x] Possibly update BIOS - 20200226 - no noticeable differences in performance or BIOS options
6. [x] Look into NIC related troubleshooting/tuning as described here.  - 20200227
   1. [x] Possibly install FreeBSD drivers for the PT - 20200226 - I looked into this more but the drivers are being mapped to the interfaces with the expected identifiers (em0-3)
7. [x] Optional: Figure out what happened to the USB boot drive - is it still useable? [tested 20200223; USB drive functions and boots; I believe the error I was seeing is related to the NIC; Updating TODOs below. ]
   If so,
   1. [ ] Test continuity in Dupont header to USB Female connector
   2. [ ] If there's enough clearance, try mounting DMM clips and observe while trying to force error
   3. [ ]  If the above fail, mount externally and return the connector (see below)
   4. [x] Look into the CSM settings as described here. This is essentially the same as the todo:
      1. pfSense's boot loader is still being wonky - preferring the DVD drive as the highest priority. I want to eliminate this as a sticking point because it adds minutes to the boot sequence - 20200227 - this seems to have mostly resolved itself by disabling several BIOS boot options.
8. [ ] Prep hard drives for 2.5" HDD install (I'm ready to give up on USB boot)
   1. [x] Move files from 2.5" to 3.5" HDD [completed 20200224; 3.5" drive is behaving erratically, I need a different backup solution.]
   2. [ ] backup both 2.5" and 3.5" HDDs
   3. [ ] partition 2.5" HDD to isolate logs/data from OS
9. [ ] Reinstall pfSense on 2.5" HDD? after more research, I'm not pursuing this idea.

I forgot the USB drive at home so debugging will have to wait. Last night I:

1. Tried resetting the config. Used auto-detect to configure WAN
2. As soon as LAN settings are saved, the system crashes
3. Reinstalled pfSense, reconfigured as above but only WAN.
4. Restarted, then the install wouldn't load. I got numerous errors including drive detatched and periph destroyed. Subsequent restarts just hang with the same errors.
   1. This might be
5. So I tried loading the drive on my OSX machine and it was unrecognizable - still need to test this (below)

I also:

1. Tried upgrading RAM - no success. First 2x2GB, then 2x512MB. Neither worked.
2. So I tried clearing/resetting CMOS to ensure the hardware was being recognized. No fix.

In addition to the previous TODOs (some of which I have completed), here are some more:

1. Test continuity in Dupont header to USB Female connector
2. Prep hard drives for 2.5" drive install (I think I'm done with using USB to boot.)
   1. Move files to 3.5" drive and partition 2.5" to isolate logs/data from OS
   2. Look into the CSM settings as described here.
3.  Collect TODOs into a (singular post to follow)

20200219 - Tuesday

From now on I'm going to refer to the NICs by their most obvious identifier for two reasons:

1. It's shorter
2. I'm probably returning the original, but would like to easily be able to refer to either.

I'll refer to the original as PT.

I'll refer to the ideal model as i350.

Snag 2: I mistook regular PCI slots for PCIe. Fortunately this isn't a huge issue. There are two PCIe slots - I'm assuming both are 1.0 or 1.1:

1. PCIe x16
2. PCIe x1

In either case they shouldn't be a bottleneck for Gigabit, though I sure wish they were 2.0+.

Plenty of bandwidth, I think? But things are still a bit unclear to me. Both are x4 (lane) cards, but there's a 20% loss associated with the above transfer rate, and I don't know if all four lanes are associated with that transfer rate to begin with. Assuming the card is only transferring relevant information and the loss is included, we get:

So, you can see why I'm not too worried. It should be noted that while PCIe buses are capable of full duplex communication, each lane is unidirectional between its endpoints. Similarly full duplex Gigabit ethernet is 1Gb/s in each direction, each conductor being unidirectional.

The x1 slot was open - but the x16 slot had a video card installed. Luckily there's another onboard VGA video card, so I removed the x16 card and installed the PT.

Boot goes fine - the card is recognized and pfSense let me configure it -- but then the system goes into this weird loop. I'm thinking it's a config file issue, but here is what I'm thinking for workflow:

1. I still need to go through the list I defined in a previous log - but a few more pressing things have come up
2. Update config and test
3. pfSense's boot loader is still being wonky - preferring the DVD drive as the highest priority. I want to eliminate this as a sticking point because it adds minutes to the boot sequence.
4. If all above goes well, then I can see how stable the USB Flash install is.
5. Return to previous log workflow.

N.B. This has got me thinking -- I do have another old , larger tower with a more substantial motherboard. Now that I have re-familiarized myself with hardware of this era, I should inspect that board to see if it wouldn't be a better fit. I might be able to use either/both machines.

Anyhow - moving on.