Backup ISP via Cellular

[Scottmichaelj]

So, I got myself a LB1120-100NAS as well - and just got it going with my FreedomPOP SIM. Works well.

 

The basics:

 

The LB1120-100NAS is in router mode. I have mine at the default IP (192.168.5.1).

The pfSense interface (I called mine WAN_FAIL) has a static IP (I set 192.168.5.2).

On pfSense, configure an 'outbound' NAT rule applied to the WAN_FAIL interface to NAT outbound traffic to the WAN_FAIL address.

   (Yes, this results in double-NAT - this appears necessary because you cannot add static routes to the LB1120-100NAS)

Add a firewall rule to allow traffic from 'LAN Net' to 'WAN_FAIL Net'

 

Then you need to create a gateway group (to setup your failover policy) and then create outbound firewall rules to assign traffic from designated hosts tot he failover gateway group.

 

Michael.

 

Thanks to Michael, I was also able to get the Netgear LB1120-100NAS working as a backup cellar data/internet on my pfsense. I did need to make a couple different gateways so that certain traffic would flow outside the VPN. Still early stages of tweaking things but overall happy. I have AT&T Wireless as my cell provider and now with their "unlimited plus" plan I was able to add this device for $20 a month. Before with the HAI C3 Communicator I was paying $10 per month for just cell (talk) backup for the alarm. Now for $10 more a month I have cell internet/data backup for the alarm and for the rest of my home, with some restrictions. I was not aware of that I can not use PLEX on LTE as backup, seems to block access. It also seems to be blocking the CAO Gadgets Wireless Tag Manager. I am sure there will be more but so far this is what I have found. I will post more later as I discover. If anyone figures out a way to get the Tag Manager working please let me know. Plex is not a big deal. The Netgear is giving me about 50d and 20u which is really good.

[MWareman]

MWareman, how did you purchase a freedompop SIM? Did you just get the SIM or did you purchase their service with a phone or cellular hotspot and then just take that sim and put it into the netgear gateway?

They sell a sim only for use in any unlocked device. It works on the AT&T network, but uses a different APN. 200MB free per month - just perfect for backup to ISY notifications and my Elk M1XEP to call home...

[Scottmichaelj]

I just wanted to give a special "Thank you" to @Michael/MWareman for his help with my pfsense setup. I had some decent knowledge of how port forwarding worked, DDNS, DHCP, VPNs, so not a NOOB but not very versed in Enterprise level stuff like advanced Firewall Rules, NATs, Gateways, and the like. Michael spent some time not just "doing things" for me just to get me going, but actually too time "explaining" how these things worked in conjunction of each other. While I am no means a guru I now have a decent understanding how this stuff works and now confident knowing how to actually use these advanced features. I have a lot more to learn but I wanted to post a special thank you for his help. Its so easy to complain about something but seems no one can give a compliment when its due.

 

TLDR; Thank you Michael/MWareman

[larryllix]

A friend of mine, vacationing in Florida, loaned me one of the bunch of Samsung Galaxy S3 mini's he picked up somewhere surplus.

In this cell phone he installed a FreedonPop SIM he bought from them at some $5 each special.

 

The plan has unlimited data, voice, and text for a free 30 day intro. No credit cards required, assuming you will sgn up later.

 

On the Samsung you just enable HotSpot and voila! You  have WiFi at a really good speed to connect your real mobile phone into. Worked great for a Canuck travelling in the States instead of a $40 / month add-on that would allow 100MByte and 100 MInutes.

 

Only negative was I had to carry a second cell phone in another pocket and charge it.

[MustangChris04]

Thanks MWareMan! I purchased the LB1121-100NAS yesterday and set it up today. It took my Verizon sim with no problems and automatically knew the APN information. I was unable to get it working in Bridged mode. My router just wouldn't receive the IP information from the Netgear, so I had to use it in Router mode and then setup DMZ to go to the router's IP. Were you successful at using it in Bridged mode? I'd prefer to not be natting twice, but since this is just a backup for my alarm and a few other things, its not a big deal. 

[MWareman]

No, I never got the t working in bridge mode. pfSense got an IP, but it was a 10. Address anyway so inbound port forwards are not going t work anyway...

 

I'm leaving mine in router mode, and double NATing.

[elvisimprsntr]

 

I think this is what I did to get dual WAN failover to work.  

 

LB1120:

• Put in bridge mode with default IP 192.168.5.1

pfSense:

• Interfaces -> OPT1 -> Enable
• System -> Routing -> GW Groups -> Add -> WAN1 -> Tier1 -> OPT1 -> Tier2
• System -> General -> Added 2 DNS servers each for WAN1 and OPT1 (also use DNS forwarder w/host overrides, DHCP static leases.)  
• FW -> Rules -> LAN -> Changed IPv4 rule to use GW Group created above (allows all LAN traffic, need to tweak to limit IPs over OPT1.)

Power cycle:

• Similar to a cable modem, I think you have to power cycle the LTE modem and router together.
• I have seen the LTE modem GW offline, but if I temporarily check System -> General -> DNS override and Disable DNS forwarder, I can force the LTE GW back online.  

Testing from couch:

• System -> Routing -> Gateways -> WAN -> Force State -> Mark GW as Down

Reference: 

• https://doc.pfsense.org/index.php/Multi-WAN#Interfaces

 

 

 

 

I was able to get the LB1120 to work in bridge mode and restrict which IPs are allowed on failover OPT1 without any problem. In addition to the settings above, I followed the recommendations in this post.  https://forum.pfsense.org/index.php?topic=118618.msg656606#msg656606.  Not sure if it is the same as @MWareman suggested in his previous post or not.  http://forum.universal-devices.com/topic/21113-backup-isp-via-cellular/?p=206141

 

FW Rules LAN

 

To test failover and IP blocking, I disabled the WAN interface via pfSense.  Simply marking GW as down, would still route traffic over the WAN connection, which seems odd.    I also tested by pulling the WAN cable.  

 

The only minor issue is pfSense will only reliabily send a WAN restored email notification. I know the WAN down notification works reliably when I simply mark WAN as down, which continues to route traffic over the WAN as noted above.  I tried adding both the LAN side and localhost 127.0.0.1 IP address of pfSense to my AllowLTE FW alias.   Not sure if it's a timing race condition or I need to add a different IP address to AllowLTE alias.  If anyone has any suggestions, let me know.  

 

Otherwise, it works like a boss!

 

P.S.  I haven't heard back from the dead beats at Linksys regarding a refund.  

[MustangChris04]

Elvis,

I've seen this before if using SMTP port 25. This may or may not be what is happening in your case. When WAN1 goes down, it is trying to use WAN2 to send the email but the ISP for WAN2 is blocking the SMTP port for your email provider so you may need to call them and make an exception, use their SMTP server to send the mail, or use a reputable SMTP such as Gmail on a secure port such as 465/589.

 

You could try and force a down so it is running on WAN2 and then send a test email.

[elvisimprsntr]

Elvis,

I've seen this before if using SMTP port 25. This may or may not be what is happening in your case. When WAN1 goes down, it is trying to use WAN2 to send the email but the ISP for WAN2 is blocking the SMTP port for your email provider so you may need to call them and make an exception, use their SMTP server to send the mail, or use a reputable SMTP such as Gmail on a secure port such as 465/589.

 

You could try and force a down so it is running on WAN2 and then send a test email.

 

 

 

pfSense is configured to use SMTP port 587, the same as all my all my other email clients which work without issue when already on LTE failover.   The symptoms are pfSense does not send the WAN down email notification reliably. Thus, I am beginning to think it is a timing issue during the failover process.  Primary WAN goes down and the transition to routing DNS requests is not complete when pfSense attempts to send the notification email.    The problem also might disappear if the pfSense local DNS cache stores the SMTP server IP address, thus subsequent email notifications occur without issue.  Only the very first email notification fails after pfSense reboot and/or DNS cache flush.  

[MustangChris04]

If you think it is a DNS routing issue, maybe you can try a direct IP and see if that temporarily works (may have to ignore certificates because it wouldn't match the SSL of your smtp domain, or use port 25 if available). Another option would be to use snmp monitoring.

 

On a side note, my LB1121 is officially dead after 18 hours. Back to Amazon it goes.

[elvisimprsntr]

If you think it is a DNS routing issue, maybe you can try a direct IP and see if that temporarily works (may have to ignore certificates because it wouldn't match the SSL of your smtp domain, or use port 25 if available). Another option would be to use snmp monitoring.

 

On a side note, my LB1121 is officially dead after 18 hours. Back to Amazon it goes.

 

 

 

It's a very minor issue, just an annoyance.  

 

That's too bad.  I have no complaints about the LB1120.  

[Scottmichaelj]

So I finally got around to moving my pfsense configuration into Virtual Box (Virtual Machine). While it was "easy" there were a few hiccups. I made sure to watch this video once from start to finish then I watched it again to follow step by step. Be warned the music is horrible!

 

 

Once I had the pfsense installed I did have to play a bit with the adapters and assigning the interfaces. I have a Intel I340-T4 quad port nic. The top is my WAN, second is LAN, third is LTE, and fourth is not used. I had to enable three adapters in the VB settings and put them in "bridge mode". Then when pfsense booted up I then saw all the network adapter ports, set them, rebooted and away I went. I made sure to backup the working config on the old box so once the reboot was done, logged into the admin gui page on the new VM, restored the backup and it automatically reinstalled my packages and all was working smoothly again.

 

A key thing I learned was to make sure the network card is showing in Windows. I needed to install the Intel network card drivers, which I didn't do.

 

I then ran speed tests from ISP and with VPN on the VB. Zero loss, if anything my VPN was faster!

 

ISP http://beta.speedtest.net/result/6204454196.png

 

VPN http://beta.speedtest.net/result/6204537841.png

 

Running this combo now saves me about 40W and the heat just from pfsense running on a separate box. Now I need to do some deep thinking to really plan things out a bit better for resiliency. Overall I really am enjoying the pfsense router and the cool things that can be done.

[elvisimprsntr]

Bad news...

 

The first time in over 10+ years I was asked to work a split shift to cover an important test and my Crapca$t internet went down.

 

Good news...

 

My pfSense appliance from protectli.com worked "like a boss". Automagically switched to LTE failover for essential security services.

 

Glad I was at home to observe it live.

[Scottmichaelj]

Bad news...

 

The first time in over 10+ years I was asked to work a split shift to cover an important test and my Crapca$t internet went down.

 

Good news...

 

My pfSense appliance from protectli.com worked "like a boss". Automagically switched to LTE failover for essential security services.

 

Glad I was at home to observe it live.

I am really happy with pfsense. I liked the protectli box but just wasn't a fit for me since I needed more CPU power for fulltime VPN.

 

Too bad your Linksys was a failure. Glad to hear everything works as expected with your new setup though!

[elvisimprsntr]

Running this combo now saves me about 40W and the heat just from pfsense running on a separate box. Now I need to do some deep thinking to really plan things out a bit better for resiliency. Overall I really am enjoying the pfsense router and the cool things that can be done.

I picked up a QNAP TS-453A 4 Bay NAS to replace my 10 year old Linux NAS and ZoneMinder NVR server. It has 4 GB Ethernet ports and supports virtualization.

 

I am extremely please with the protectli.com device, but I bet one could run pfSense on a QNAP VM.

[Scottmichaelj]

I picked up a QNAP TS-453A 4 Bay NAS to replace my 10 year old Linux NAS and ZoneMinder NVR server. It has 4 GB Ethernet ports and supports virtualization.

 

I am extremely please with the protectli.com device, but I bet one could run pfSense on the QNAP.

Yes you can run it as a VM on the QNAP. I thought about grabbing a QNAP as well (rack mount version) as running "docker" images on it might be nice. There are RaspPi docker images so you could have multiple "Pis" on the QNAP. Also a PLEX server if your into that.

[elvisimprsntr]

Yes you can run it as a VM on the QNAP. I thought about grabbing a QNAP as well (rack mount version) as running "docker" images on it might be nice. There are RaspPi docker images so you could have multiple "Pis" on the QNAP. Also a PLEX server if your into that.

At the moment I am using the QNAP as a 3 NAS disk RAID5 array and a single NVR surveillance drive for my ONVIF IP cameras.

 

I'm also running a Ubuntu server VM and imported my existing VirtualBox Window$ 7 VM to the QNAP for the ElkRP application, the one and only Window$ app I use at home. It was a bit tricky to migrate the Window$ 7 VM and not upset WPA.

[Scottmichaelj]

A week in and so far so good with pfsense running on a VM. I have a few things to tweak, like auto stating the pfsense VM on reboot, but all in all no complaints. Plus I am now saving around 50W by running pfsense on the VM rather than its own box. Even better with the new AT&T Unlimited plans, I actually saving money each month than before with just the cell dial backup, plus I now have internet backup for all the devices in my home!

[Scottmichaelj]

Quick update: I really wanted to have the VirtualBox instance of pfsense auto start when the computer does. So to do that I went to C:\Program Files\Oracle\VirtualBox\ and right clicked on VBoxManage.exe to "create a shortcut". I then cut the shortcut and pasted it into the Windows Startup folder. Once that was done I right clicked on the shortcut then when to "Properties" then "Target" and removed everything there and replaced it with the line below. My pfsense VBox machine name is pfsense, but you may have rename as required for whatever you called yours. Then just save and now when your computer reboots the pfsense will run in the background with no windows. You can always get back to it by running the VirtualBox software and clicking on the VM. 

 

Hopefully this helps someone! So far pfsense as a VM is working smoothly.

 

"C:\Program Files\Oracle\VirtualBox\VBoxManage.exe" startvm "pfsense" --type headless

[JayC]

Quick update: I really wanted to have the VirtualBox instance of pfsense auto start when the computer does. So to do that I went to C:\Program Files\Oracle\VirtualBox\ and right clicked on VBoxManage.exe to "create a shortcut". I then cut the shortcut and pasted it into the Windows Startup folder. Once that was done I right clicked on the shortcut then when to "Properties" then "Target" and removed everything there and replaced it with the line below. My pfsense VBox machine name is pfsense, but you may have rename as required for whatever you called yours. Then just save and now when your computer reboots the pfsense will run in the background with no windows. You can always get back to it by running the VirtualBox software and clicking on the VM. 

 

Hopefully this helps someone! So far pfsense as a VM is working smoothly.

 

"C:\Program Files\Oracle\VirtualBox\VBoxManage.exe" startvm "pfsense" --type headless

 

I may entertain setting this up, I use unRaid so I am considering coupling my router into this using a VM.  Do you see a benefit in grabbing a 4 port nic card over a 2 port nic?

[Scottmichaelj]

I may entertain setting this up, I use unRaid so I am considering coupling my router into this using a VM. Do you see a benefit in grabbing a 4 port nic card over a 2 port nic?

The reason for the four port was, one for WAN, one for LAN, one for Cell backup (WAN) and the last goes unused since the motherboard has one.

[elvisimprsntr]

The Netgear LTE modem w/failover built-in is in stock (LB2120)

 

 

http://www.provantage.com/service/searchsvcs?QUERY=Netgear+LTE+&SUBMIT.x=20&SUBMIT.y=13

 

https://www.amazon.com/gp/aw/d/B01MQRHQYT/ref=mp_s_a_1_1?ie=UTF8&qid=1495175166&sr=8-1&pi=AC_SX236_SY340_QL65&keywords=netgear+lb2120

[Scottmichaelj]

When I posted here about the Protectli vault sent an email to Protectli regarding my results. They are really nice guys and are working on a bunch of products. One which will be hitting Amazon soon but they are selling direct to anyone who wants one now. Retail is $499.00 USD plus shipping for barebones system (No RAM or SSD) SO I purchased one last week and finally was able to get it going today.
 
I had a spare 240GB SSD drive so I used that and purchased (2) Patriot Signature Line 4GB PC4-17000 DDR4 2133MHz CL15 SODIMM Module PSD44G213381S from Amazon here https://www.amazon.com/gp/product/B016TQ36SQ/ref=oh_aui_detailpage_o01_s00?ie=UTF8&psc=1 - I was also told the Crucial 8GB Single DDR4 2133 MT/s (PC4-17000) DR x8 SODIMM 260-Pin Laptop Memory - CT8G4SFD8213 has been tested and working.

So $520 for the Vault (Shipped), $65 for the RAM - brings this device up to $600 without the SSD. A bit expensive if you ask me. The promising thing was the Intel i3 with AES-NI on the chip to help with VPN encryption for 24/7 use on pfsense, but does it work?
 
The Protectli FW6B Datasheet Specifications:
 
Model FW6B
Processor Intel Kaby Lake 7100U (14nm, 2.4GHz, 3MB SmartCache) Found here: http://ark.intel.com/products/95442/Intel-Core-i3-7100U-Processor-3M-Cache-2_40-GHz-
Memory 2x SODIM DDR4 1866/2133 MHz,Max: 32 GB
Graphic Support Integrated Intel HD Graphics 610
Storage 1x mSATA, 1x SATA
Audio N/A
Network 6x Intel 82583V 1000M LAN, support for Wake On LAN
Front I /O 4x USB 3.0
1x HDMI (no sound output)
1x Power Switch
1x RJ45 COM (cable included)
Rear I/O 6x RJ45 1000M LAN
1x DC 12V 3.33A Input
2x Predrilled Wifi Holes
Built-in I/O 1x MINI PCIE for USB channel Wifi
1x MINI SATA
1x SATA
2x SODIMM
1x CPU Fan Header
Power supply DC-12V, 3.33A
Temp -10°C~54°C (14 – 130F)
Humidity 0~95% relative humidity, non-condensing
Dimensions 155*126.5*52.5 mm (6.1*4.98*2.07 in)
Weight 1.6kg (3.5lb)

 

Power Draw is 11Watts!
 
So without further delay after getting the memory today via Amazon I removed the pfsense SSD from the previous PC used above, put it in the Vault and booted it up. I had to reassign the network cards which took a couple minutes and I was up and running again.

The first thing I did was a speed test.
 
Vault on ISP 298D/30U
 

 
Vault with VPN 259/28U 
 



So is this device really worth the $600 or go back top the old PC since the speed difference is minimal? Will this box work for fiber users and get to 1GB speeds? Hard to know as I don't have access to more than 300Mbps down. Hands down it does perform and max out the ISP connection better than the previous Vault box.

[MWareman]

I have the same exact device now - and my fiber service is 500 down and 100 up.

 

The challenge I have is when I use my providers in-city speed test (1ms latency) - I get 500 down and 100 up. If I switch to another server, I get lower. Not because it's slower - but because latency increases (14-15ms).

 

The other thing I've noticed - when the VPN is connected to 'PIA' - I am hard-limited to 100Mbps. Cannot go any faster, no matter what.

With my personal VPN host (it's running on a tiny Linux instance hosted by Afterburst and located in Chicago) I get 394 down and 98 up - but the latency doubled. That's better than I've ever had to this host - but the host I'm conencting to is now the constraint (it's CPU got slammed for the test!). 

 

In the weeds of VPN - there will always be some overhead. Your ISP (generally) allows 1500 byte packets to be sent. When you encrypt those packets, there is an additional header added to support that (an embedded or 'inner' source and destination IP address, for example) - but the limit is still 1500 bytes. So, there are fewer bytes available for data in each packet. This results is a reduction in the data flow for a given number of bytes on the wire. 

 

In my case - the max packet size I can send thru the VPN is 1472...

C:\Users\user>ping -n 1 -l 1472 -f 8.8.8.8

Pinging 8.8.8.8 with 1472 bytes of data:
Reply from 8.8.8.8: bytes=64 (sent 1472) time=22ms TTL=45

Ping statistics for 8.8.8.8:
    Packets: Sent = 1, Received = 1, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 22ms, Maximum = 22ms, Average = 22ms

C:\Users\user>ping -n 1 -l 1473 -f 8.8.8.8

Pinging 8.8.8.8 with 1473 bytes of data:
Packet needs to be fragmented but DF set.

Ping statistics for 8.8.8.8:
    Packets: Sent = 1, Received = 0, Lost = 1 (100% loss),

This can sometimes be made up for with compression, but in modern times when non-VPN connections are already generally compressed the compression gain is marginal at best.

 

Finally - understand the the crowd nature of VPN necessarily means that your VPN host is sharing that hardware with hundreds of other users. There is contention there which will likely have some impact to a varying degree. As soon as I move away from the ISP provided speed test the results get variable. But the ISP provided speed test is useless for comparing to.

 

It's an expensive device, sure. But for it's compute capacity it's very low power and allows me to move the bottleneck very successfully. Given the CPU hardly misses a beat at 400Mbps, I don't have any trouble believing this could handle 1Gbps VPN - assuming the latency is low enough and the provider can handle it...  The real test would be back-to-back connecting two of them, setting up a VPN between them and seeing how much you can push thru..

 

Speed costs money - how fast do you want to go?  

[Scottmichaelj]

Speed costs money - how fast do you want to go?  

This is where I am at right now. I am not getting anymore throughput using the Vault vs my PC or VM. My PC CPU is an Intel Core2 Duo CPU, E8400 @ 3.00GHz, Wolfdale with 2GB DDR2 Ram. Per my speed tests I am able to get the similar results as the Vault (see above). Using a Virtual Box VM on a AMD FX-8350 Black Edition Vishera 8-Core 4.0 GHz (4.2 GHz Turbo)with 32GB of RAM, I again see same results if not slightly better than the Intel PC.

 

So the question now for me is, "Is $600 for the Vault worth it?" especially since my MAX ISP speeds right now are 300Gbps and my PC and VM both seem to be on par with the Vault speed wise. My inclination is no and return the Vault, and stick with the VM. If the Vault beat my current setup then no question I would keep it, but thats not the case.

 

EDIT: I retract my statements. I went back to the VirtualBox VM and PC was not able to match my speeds to what I was seeing previously. See my newer post below.