additional hop is at the L3 switch. vlan subnet designated on the L3 switch. next hop. this is also associated to your pve setup as well. primarily wondering why you don’t LACP from your pve host and why you’re choosing to allow your switch to be involved in any routing at all given you’re putting all this behind pfsense. you said it was a home lab of sorts. your granularity is at your ability to make vlans and route those vlans. now you have to manage your L3 from your switch AND your pfsense instance. what does your carrier provide you with for your broadband connection? is it a kind of Soho router with WiFi built into it? are you on fiber? is your switch your edge device?
I’m really glad you brought this up as I think this makes for an excellent discussion on L3 switching in general and perhaps more specifically on L3 switches vs. router-on-a-stick. I think you also offer an interesting alternative network design, especially with the LACP angle. I may split this off into a separate topic since I think it warrants its own discussion.
First, let me start off with a general intro on L3 Switches vs. Router-On-A-Stick for the uninitiated and then go into my specific homelab architecture and decisions.
L3 Switch vs. Router-On-A-Stick
In general, L3 switches are preferred for inter-VLAN routing when possible. I don’t want to get bogged too down in that since you’ll find a lot of discussion already on the internet about why L3 switches are preferred over router-on-a-stick, but here are some highlights/resources that I think explain it best:
L3 switch is generally a better approach for all but the smallest business.
The reason is with router on a stick all vlans share the bandwidth of the physical interface whereas with a L3 switch you get the full switching capacity.
Obviously L3 switches do not have the full feature support of routers.
Keep in mind, when we’re talking about pfSense, we’re also talking about a software router:
Just to add to what Jon has noted, there’s often a huge performance difference between a L3 switch and a software based router, especially for the price. For example, a high-performance 4451 can router up to 2 Gbps, while multi gig port L3 switches can do wire speed for all their gig ports.
When To Use an L3 Switch vs. Router/Firewall
A tip I learned from the network guys at work: always terminate your vlans on your L3 switch. That’s what it’s for. The router/firewall should only see packets leaving or entering the network. Let your L3 switch route inside your network. It’s more efficient and you can reboot your router VM without losing connectivity to your VM host.
The best design for performance is to do your intra-VLA routing on your switches and use the firewall for external routing/filtering.
There are, however, plenty of people who use firewalls for l3 functions - I have done two separate network migrations in the last three months where L3 functions were moved off a 6509 core into a set of PA firewalls. I can certainly understand migrating away from legacy power hog 6509s, but I find moving routing and igp from dedicated silicon to firewalls infuriating. I just don’t see why anyone would want to cripple the performance of their network like that while reducing the efficacy of their security by using firewalls as a router.
If your primary goal is simplicity, then you’re right, router-on-a-stick is probably the way to go. Combined with LACP bonding, you probably will even overcome the performance loss from using a router compared to an L3 switch, even with a software-based router. At least at “low” network traffic conditions (if you consider 2 Gbps of network traffic “low”), before you run into the bottlenecking issue.
My Homelab Network
Now for my specific reasons for choosing to handle interVLAN routing within an L3 switch as opposed to pfSense/router:
- This is a homelab, it’s whole purpose is to learn. I hadn’t implemented an L3 switch before, and I wanted to learn how, so this was always goal #1 from the beginning. I also stood up a router-on-a-stick set up and allowed my pfSense router/firewall to handle
To understand the rest, you’ll need to know a little bit about my homelab and the kind of applications I run.
I give some insight here in my guide to securing your VLANs with ACLs on the HP 1910 switch, but I think a picture is worth a thousand words:
My VLANs on this switch all sit in a DMZ whose firewall is supported by an interface on pfSense. The vast majority of my traffic is intraVLAN traffic, followed by interVLAN traffic. Traffic that needs to traverse the WAN/pfSense router is rare in comparison. As part of good design, the VLANs were designed so that most devices on an individual VLAN only have to talk to the devices in that VLAN (example: NGINX proxy server + upstream web/application servers).
However, sometimes it is necessary for traffic to traverse the VLAN. Sometimes a web server needs to retrieve a new trained model from the “Datacenter”. The Datacenter and Web Server VLANs need to back up to the file servers. Handling my routing on the L3 switch, allows me to keep routing as close to the clients as possible. i.e. It doesn’t need to go all the way to the pfSense router just to come all the way back.
The File Server VLAN exports various NFS shares. These data transactions are often small, 4k-random-read/write-style transactions. In this situation, it’s not so much rote bandwidth, as it latency that’s the concern. An L3 switch is always going to beat a software router here.
As for your other questions, I have a gigabit connection, unfortunately not full duplex, so that’s 1 Gbps down, 35-40 Mbps up. Since it’s a business connection, the ISP provided a modem (i.e. not one of those all-in-one switch/wifi/router units).