Bumbershoot Posted January 17 Posted January 17 4 hours ago, ndfan77 said: 1) EISY was running Alpine Linux with Docker (this combination is very small, very fast, and very stable). In a thread on this board long ago, the possibly of using Linux to host the ISY software was discussed, and I believe the GPL proved to be an issue for UDI, which had more than a little to do with the selection of FreeBSD. 1
Michel Kohanim Posted January 17 Posted January 17 @Bumbershoot, yes precisely. @ndfan77 and @akss, in order to run eisy on a VM/docker under Linux, we will have to rewrite millions of C/C++ code in Python, Java, node.js, script, etc. Or, we have to recompile under Linux. As a background, before we understood the gravity of GPL licenses, we spent over a year compiling everything under Linux. It was a nightmare since every little thing requires _devel packages to be installed and maintained. We then had to choose an OS or write our own OS. The choice was BusyBox - a lions den of GPL monsters - and Debian - too slow. FreeBSD is not immune to shortcomings: device drivers and docker support. Given the aforementioned, we just do not have the resources to make IoX Linux compatible so that you can run it inside a Linux docker. Sorry! With kind regards, Michel
akss Posted January 17 Posted January 17 @Michel Kohanim speaking for myself, I hope you don’t think I’m trying to tell you or the community how things “should” be done, I don’t know jack beyond what I’ve found useful personally, and my questions are only coming from a place of raw curiosity. I appreciate your patience in explaining why decisions were made, I find it all fascinating for its own sake and feel I can empathize a bit with the demands to make product decisions in a timely and committed manner. re: the Linux recompile, couldn’t the FreeBSD image as-is just be run as a VM on any x86? I mean, no recompile necessary, no conversion to Linux, etc? Isn’t that the big advantage of such deep abstraction? A container is probably never an option but a VM? Again, not saying this should be done or that UD “should” support this, just curious if it would work. But yeah, maybe there are other non-hardware ancillary product ideas for UD down that path. 1
Michel Kohanim Posted January 17 Posted January 17 @akss, On the contrary, not only I enjoy these discussions, but I also find them perfect opportunity to learn (outside of my own ecosystem). From my own experimentations, the only things that can run BSD in a VM are qemu (which is slow), virtualbox (which is even slower), or bhyve running on BSD. If you know of other solutions, I am all ears! With kind regards, Michel
akss Posted January 17 Posted January 17 (edited) When you say slow, I gather you're talking specifically about hardware access (how the pass-through is adding latency)? Or just noticeable impacts to execution times? It seems to be a thing on ProxMox: FreeBSD Guest Notes - Proxmox VE I think pfSense is FreeBSD and that seems to be a not uncommon client of proxmox in the home server space. But I've certainly not done it personally and can't attest to performance. EDIT: reading through a few posts on the proxmox forums, it looks like a headache. I think in the enterprise space, hyper-v runs FreeBSD and has some direct hardware driver work done for it to address performance, but that's a whole different thing, obviously.. unless one is running windows server at home, which.. I mean, people do. I used to. Edited January 17 by akss
Michel Kohanim Posted January 17 Posted January 17 @akss, I have tried VirtualBox and QEMU a long time ago on Windows. They were horrendous. I think Promox is made for server class devices. With kind regards, Michel
firstone Posted January 18 Posted January 18 21 hours ago, akss said: Not specific to me, just responding to post saying they had a gaggle of systems to load up on one host. If they can’t load them all up on the eisy hardware, what about running the eisy VM on a different host? I can imagine some issues there, but if I had one I’d have a hard time not trying it just to satisfy curiosity. What’s your position on doing so, acknowledging such a person would floating out into deep space alone without a tether (no support, no sympathy)? Known issues with boot process (like tied to the tpm)? Prefer people don’t do that (which is entirely fair)? Had it running on VMware - converted image from polisy. It starts up and I can open admin console but the settings looked out of wack, could not set time zone, etc. Wouldn't connect to Insteon PLM either.
brians Posted January 18 Posted January 18 9 hours ago, Michel Kohanim said: @akss, I have tried VirtualBox and QEMU a long time ago on Windows. They were horrendous. I think Promox is made for server class devices. With kind regards, Michel Proxmox can run on lots of different hardware. I run it on an older Intel nuc for Home Assistant and couple other VMs. Benefit is can backup and restore VM in minutes, can do snapshots before updates for quick rollback if any issues etc. It would be neat to run Iox in a virtual machine under Proxmox. Proxmox also works great with usb pass through. 2
akss Posted January 18 Posted January 18 Yep, I run proxmox on a Gigabyte Brix (like a Nuc) with a 6th-gen i7 and 16GB of RAM. Runs like a top, ultra reliable. The restoration story of a ProxMox VM is pretty good - gives me a lot of options to bring it up on any spare local compute resource if need be. It's starting to feel like a Snapcast server may be my next project, which I see runs on FreeBSD. May be interesting to try that. I'd typically just use Ubuntu server as my go-to.
giomania Posted January 18 Posted January 18 In addition to running HA as a VM with bhyve on EISY I think I'd also like to try running HA as a Docker container on EISY. (I already have Docker running on several ESX Alpine VMs, and on a bare metal Fedora box -- but I can see some advantages to having Docker running on the EISY, and I suspect HA would be easier to manage/upgrade/downgrade as a container than as a VM.) I've done a lot of enterprise network and system administration with Cisco, Red Hat, CentOS, Alpine Linux, Windows Server (MCSE), Active Directory (MCSE), vSphere/VMware/ESX, and Docker, along with quite a few other types of system/network administration. (And before all that, back "in the DOS days" I wrote a lot of systems level code in x86 protected mode assembly language, including a TCP/IP stack from scratch, routing and firewall code, and multi-threaded smtp/http/dns/etc servers.) FreeBSD is one of the *NIX's I haven't messed a lot with (but that never stopped me before). I know there's some Docker on FreeBSD documentation already out there, and I'll be fine on my own, but I suspect I may want to "reflash" a few times along the way. It'd be great if you could share (privately if necessary) the reflash steps. If Docker seems like a viable approach I'll be happy to share my documentation.Is it going to be difficult to upgrade HA when in a VM? Would that not be accomplished in the HA UI once installed?Thanks MarkSent from my iPhone using Tapatalk Pro
giomania Posted January 19 Posted January 19 (edited) I purchase and installed a new NVME SSD and ran the commands to configure the OS using a simple volume. Next I ran the script to install HA, and I think it did not work that well for me. I see no MAC address printed, and there were "command not found" errors interspersed. I have attached the text file from the session. I will start looking for commands to see if it is running or how to obtain the mac address. Mark Update: I went back through and ran the individual commands by placing "sudo" in front of each one, and it installed and set the MAC address. HA Installation on eisy - SSH Capture - 2024-01-18.txt Edited January 19 by giomania Update
tpolito Posted January 19 Posted January 19 (edited) Similar issue as above. Installed my SSD and setup per the instructions (or at least I think I did). Ran the HA script and got this. Alot of messages like "no packages available..." or "command not found". I'm assuming i did something wrong? [admin@eisy ~]$ curl -s https://pkg.isy.io/script/create_ha_vm.sh | sudo bash Password: ifconfig_re0: -> DHCP Make sure necessary packages are installed Updating FreeBSD repository catalogue... FreeBSD repository is up to date. Updating FreeBSD-base repository catalogue... Fetching meta.conf: 100% 163 B 0.2kB/s 00:01 Fetching packagesite.pkg: 100% 14 KiB 14.7kB/s 00:01 Processing entries: 100% FreeBSD-base repository update completed. 162 packages processed. Updating udi repository catalogue... Fetching meta.conf: 100% 163 B 0.2kB/s 00:01 Fetching packagesite.pkg: 100% 9 KiB 8.9kB/s 00:01 Processing entries: 100% udi repository update completed. 11 packages processed. All repositories are up to date. pkg: No packages available to install matching 'vm-bhyve' have been found in the repositories pkg: No packages available to install matching 'edk2-bhyve' have been found in the repositories pkg: No packages available to install matching 'wget' have been found in the repositories pkg: No packages available to install matching 'qemu-tools' have been found in the repositories Prepare /etc/rc.conf vm_enable: -> YES vm_dir: -> zfs:storage/vms vm_list: -> homeassistant Create ZFS fileset for VMs and prepare templates bash: line 41: vm: command not found cp: directory /storage/vms/.templates does not exist bash: line 45: vm: command not found bash: line 46: vm: command not found Downloading image bash: line 49: wget: command not found Extracting... unxz: /tmp/tmp.dzM7itoO/haos_generic-x86-64.img.xz: No such file or directory Creating a VM bash: line 54: vm: command not found Copying image dd: /tmp/tmp.dzM7itoO/haos_generic-x86-64.img: No such file or directory /usr/sbin/sysrc: cannot create /storage/vms/homeassistant/homeassistant.conf: No such file or directory /usr/sbin/sysrc: cannot create /storage/vms/homeassistant/homeassistant.conf: No such file or directory /usr/sbin/sysrc: cannot create /storage/vms/homeassistant/homeassistant.conf: No such file or directory bash: line 64: vm: command not found bash: line 65: vm: command not found bash: line 66: vm: command not found Please wait about 10 minutes and follow instructions at https://www.home-assistant.io/getting-started/onboarding/ to get your Home Assistant setup ISY integration: https://www.home-assistant.io/integrations/isy994/ [admin@eisy ~]$ ifconfig_re0="DHCP" Edited January 19 by tpolito
giomania Posted January 19 Posted January 19 14 minutes ago, tpolito said: Similar issue as above. Installed my SSD and setup per the instructions (or at least I think I did). Ran the HA script and got this. Alot of messages like "no packages available..." or "command not found". I'm assuming i did something wrong? [admin@eisy ~]$ curl -s https://pkg.isy.io/script/create_ha_vm.sh | sudo bash Password: ifconfig_re0: -> DHCP Make sure necessary packages are installed Updating FreeBSD repository catalogue... FreeBSD repository is up to date. Updating FreeBSD-base repository catalogue... Fetching meta.conf: 100% 163 B 0.2kB/s 00:01 Fetching packagesite.pkg: 100% 14 KiB 14.7kB/s 00:01 Processing entries: 100% FreeBSD-base repository update completed. 162 packages processed. Updating udi repository catalogue... Fetching meta.conf: 100% 163 B 0.2kB/s 00:01 Fetching packagesite.pkg: 100% 9 KiB 8.9kB/s 00:01 Processing entries: 100% udi repository update completed. 11 packages processed. All repositories are up to date. pkg: No packages available to install matching 'vm-bhyve' have been found in the repositories pkg: No packages available to install matching 'edk2-bhyve' have been found in the repositories pkg: No packages available to install matching 'wget' have been found in the repositories pkg: No packages available to install matching 'qemu-tools' have been found in the repositories Prepare /etc/rc.conf vm_enable: -> YES vm_dir: -> zfs:storage/vms vm_list: -> homeassistant Create ZFS fileset for VMs and prepare templates bash: line 41: vm: command not found cp: directory /storage/vms/.templates does not exist bash: line 45: vm: command not found bash: line 46: vm: command not found Downloading image bash: line 49: wget: command not found Extracting... unxz: /tmp/tmp.dzM7itoO/haos_generic-x86-64.img.xz: No such file or directory Creating a VM bash: line 54: vm: command not found Copying image dd: /tmp/tmp.dzM7itoO/haos_generic-x86-64.img: No such file or directory /usr/sbin/sysrc: cannot create /storage/vms/homeassistant/homeassistant.conf: No such file or directory /usr/sbin/sysrc: cannot create /storage/vms/homeassistant/homeassistant.conf: No such file or directory /usr/sbin/sysrc: cannot create /storage/vms/homeassistant/homeassistant.conf: No such file or directory bash: line 64: vm: command not found bash: line 65: vm: command not found bash: line 66: vm: command not found Please wait about 10 minutes and follow instructions at https://www.home-assistant.io/getting-started/onboarding/ to get your Home Assistant setup ISY integration: https://www.home-assistant.io/integrations/isy994/ [admin@eisy ~]$ ifconfig_re0="DHCP" I guess the instructions are not as simple as I thought. I went back through and ran the individual commands by placing "sudo" in front of each one, and it installed and set the MAC address. I did get a few errors when things that worked before were already there, but seems to be good. Mark
tpolito Posted January 19 Posted January 19 So you typed in each command listed in the Helper script window? Or saved that script and ran it, but added sudo in front of each line? I'm a good direction follower, but know very little about coding.
giomania Posted January 19 Posted January 19 23 minutes ago, tpolito said: So you typed in each command listed in the Helper script window? Or saved that script and ran it, but added sudo in front of each line? I'm a good direction follower, but know very little about coding. I did type (copy / paste) each command, but now that you mention it, I did not realize that the curl command script at the bottom was a representation of all the commands contained in the helper script window. Now I wonder if just adding sudo in front of the curl command script would have been the solution. When it comes to CLI, I need someone to tell it to me like I am a 3-year old so nothing is left to my imagination or chance, lol.
tpolito Posted January 19 Posted January 19 20 minutes ago, giomania said: I did type (copy / paste) each command, but now that you mention it, I did not realize that the curl command script at the bottom was a representation of all the commands contained in the helper script window. Now I wonder if just adding sudo in front of the curl command script would have been the solution. When it comes to CLI, I need someone to tell it to me like I am a 3-year old so nothing is left to my imagination or chance, lol. sudo before the curl command seems to be working for me. We will see shortly. Thanks!
tpolito Posted January 19 Posted January 19 Worked! Thanks! Now I just have to learn Home Assistant... and get my Eisy showing back up in the IoX Finder... and get my Govee lights working through the admin console 1
Michel Kohanim Posted January 19 Posted January 19 @tpolito, sudo should be before bash and not before curl. So: curl ... | sudo bash With kind regards, Michel 1
tpolito Posted January 19 Posted January 19 Uh oh. I put it before curl and it worked. Should I go back and redo something?
ndfan77 Posted February 2 Posted February 2 On 1/18/2024 at 7:59 PM, tpolito said: Similar issue as above. Installed my SSD and setup per the instructions (or at least I think I did). Ran the HA script and got this. Alot of messages like "no packages available..." or "command not found". I'm assuming i did something wrong? pkg: No packages available to install matching 'vm-bhyve' have been found in the repositories pkg: No packages available to install matching 'edk2-bhyve' have been found in the repositories pkg: No packages available to install matching 'wget' have been found in the repositories pkg: No packages available to install matching 'qemu-tools' have been found in the repositories I ran into these errors as well while stepping through the Helper Script line by line as root on a brand new Eisy. Issuing: pkg install <xyz> would produce the "No packages available to install matching ..." error no matter which (currently uninstalled) package I was trying to install. And issuing: pkg search <xyz> wouldn't return any results (for packages like wget that are definitely in the base repository). What straightened pkg out on Eisy for me was temporarily renaming the /usr/local/etc/pkg folder to something else and then issuing a pkg command (e.g. "pkg search wget") and then pkg recognized it needed to rebuild the repository databases, and then started to work correctly. Then I renamed the /usr/local/etc/pkg folder back to its proper name and pkg rebuilt the repository databases again (for any type of pkg search) -- and worked fine after that. BUT, I suspect that if I had just issued: pkg update -f (Which also rebuilds the repository databases.) It probably would have fixed my issue without temporarily renaming /usr/local/etc/pkg. So my suggestion to UDI would be to consider including "pkg update -f" before the "pkg install ..." in the helper script: echo "Refreshing package databases..." pkg update -f echo "Installing necessary packages..." pkg install -y vm-bhyve edk2-bhyve wget qemu-tools
jec6613 Posted February 2 Posted February 2 On 1/17/2024 at 4:21 PM, Michel Kohanim said: From my own experimentations, the only things that can run BSD in a VM are qemu (which is slow), virtualbox (which is even slower), or bhyve running on BSD. If you know of other solutions, I am all ears! Hyper-V and VMWare. VMWare is for server-class hardware, but Hyper-V is just a feature to install on desktop Windows, or you can run Hyper-V Server for free as a hypervisor on any hardware that supports running Windows. Performance is about 95% that of running on the iron, at least for pfSense, which also runs on FreeBSD.
ndfan77 Posted February 2 Posted February 2 1 hour ago, jec6613 said: Hyper-V and VMWare. VMWare is for server-class hardware, but Hyper-V is just a feature to install on desktop Windows, or you can run Hyper-V Server for free as a hypervisor on any hardware that supports running Windows. Performance is about 95% that of running on the iron, at least for pfSense, which also runs on FreeBSD. Yep, I'm running FreeBSD 13.2 on the free version of ESXi (along with quite a few other Linux and Windows VM's) on a PowerEdge T140 down in the basement. It works fine. I didn't start with the FreeBSD vmdk image though (since I prefer to know exactly how an OS is configured from the start). Instead I started with a new vSphere VM with unformatted storage, mounted the FreeBSD amd64 installation ISO as a virtual DVD, and let it install like it was installing on bare metal, and then used pkg to install open-vm-tools when it finished. My interest at this point isn't in doing a P2V of IoX/PG3 to ESXi, but rather just to have a FreeBSD 13.2 playground with snapshot/restore capability separate from the Eisy (before doing something I'm not sure of on the Eisy). I'd much rather be able to use that Eisy hardware to run a couple other small footprint critical infrastructure services (such as Ubiquiti's UniFi controller) since I can keep the Eisy up a lot longer on UPS/generator during a power outage. Eventually I may fool around with trying to run the UniFi controller in a separate Bhyve VM on Eisy.
Michel Kohanim Posted February 2 Posted February 2 10 hours ago, ndfan77 said: What straightened pkg out on Eisy for me was temporarily renaming the /usr/local/etc/pkg This is not a good idea for a very specific reason: our udx repository includes many packages that are tailored made for us (libwebsockets, wifibox, dns, ...). Once you remove that directory, then the plain ones are installed from FreeBSD repo. This will cause major issues. In most cases, if a pkg is not available it's because of communications OR - if not from udx - FreeBSD repo has not yet built it or had errors in the last build. In all cases, if you have done the above, please restart udx which will take care of reinstalling udx specific packages: sudo service udx restart With kind regards, Miche
ndfan77 Posted February 2 Posted February 2 (edited) On 2/2/2024 at 11:29 AM, Michel Kohanim said: This is not a good idea for a very specific reason: our udx repository includes many packages that are tailored made for us (libwebsockets, wifibox, dns, ...). Once you remove that directory, then the plain ones are installed from FreeBSD repo. This will cause major issues. In most cases, if a pkg is not available it's because of communications OR - if not from udx - FreeBSD repo has not yet built it or had errors in the last build. In all cases, if you have done the above, please restart udx which will take care of reinstalling udx specific packages: sudo service udx restart With kind regards, Miche Michel, I know you have a custom udi repository defined in that folder (along with an additional FreeBSD repository), and I thought I was being pretty clear that I wasn't suggesting anybody else do the same thing (if they were running into the same problem with pkg not installing components), and should instead try issuing a "pkg update -f". There was no communication issue with my Eisy (former lead network engineer [among other things] speaking). The pkg configuration on my (essentially brand new) Eisy was definitely broke (and yes, the Eisy was restarted several times [which I strongly dislike as a 1st response]). To reiterate what I said: Trying to install anything (not currently installed) produced a "No packages available to install matching ..." error, and trying to search for anything that pkg should easily find (e.g. "pkg search wget") produced an empty result every - single - time. Something was clearly "wonky" in the pkg configuration on my new Eisy, so I started comparing it to a fresh FreeBSD 13.2 VM installation (which did correctly return a response for "pkg search wget", even after being installed), and noticed that the clean FreeBSD 13.2 installation didn't have the /usr/local/etc/pkg folder. So the next thing I tried on the Eisy was renaming the custom repository folder to something else (briefly) and viola!: pkg on the Eisy now started returning results for "pkg search wget" (after it automatically rebuilt the repository database). This is not behavior that suggests a communication error was suddenly fixed. This is behavior that suggests that doing something that forcibly causes pkg to rebuild the repository database is what fixed the issue. The /usr/local/etc/pkg folder was shortly renamed back to it's original state (before pkg installed or updated anything), and when another "pkg search wget" was issued again, pkg again rebuilt the repository database (to the correct configuration). I don't have a lot of FreeBSD experience (but do have a lot of enterprise-level Linux administration experience) and I've never seen a package manager downgrade or remove newer packages that came from a repository no longer enabled, unless it was specifically told to. What I'd submit should be of interest (to you) is: Apparently some Eisy's (at least in my case) are starting out in life with a non functioning pkg configuration, because something's wonky with their repository database. That I suspect issuing a "pkg update -f" will fix it (the right way). I say "suspect" because I didn't discover the pkg update command had a -f (force) switch until after my configuration had already been fixed with the custom repository folder rename sequence. (The -f switch is not documented in the FreeBSD pkg man pages.) If others are running into this too (sure seems like there's been a couple already), and refreshing the pkg repository databases with "pkg update -f" fixes it, than you should consider adding that statement before the "pkg install -y vm-bhyve edk2-bhyve wget qemu-tools" statement in the helper script. Edited February 15 by ndfan77 1
mapeter Posted February 11 Posted February 11 Has anyone tweaked the memory and disk values in the create_ha_vm.sh shell script? The script specifies 1GB for RAM and 16GB for disk. However, the HA page at https://www.home-assistant.io/installation/linux states the minimum requirements should be 2GB of RAM and 32GB for disk. The eISY has 8GB of RAM and best I can tell from the top command, there is 6GB free (Mem: 68M Active, 394M Inact, 34M Laundry, 662M Wired, 6476M Free). That said, I am not running many node servers, yet. What are people's experiences with the RAM setting? Is 1GB good enough for HA? Will specifying 2GB steal too much from IoX and PG3? From the script: # Home Assistant VM name and how much resources to allocate HA_VM_NAME="homeassistant" HA_VM_CPU="2" HA_VM_MEM="1G" HA_VM_DISC="16G"
Recommended Posts