I'm on v5.3.4. I encountered what you describe when I tried to use any smtp mail server for notifications other than default. The ISY began rebooting continually.
With the default mail server the test button generates a test notification but no real notifications are sent ever. I've been on this version for a long time and only realized notifications aren't working when a leak sensor responding to a water leak failed to notify me. I then looked into all of the notification messages and created conditions for them to be sent and none did.
I tried two different smtp servers. The test button would result in an error message that the mail server isn't responding. I left one setup and figured I would get back to trying to make this work later. That evening when programmed actions began I noticed the same things happening over and over. When I checked in the admin console the ISY was only up for about 2 min before it would reboot.