You're absolutely right that it could be the fixture, as I haven't tried it without Insteon hardware. My thoughts were leading me in the direction that I'll have to spend some time with the hardware troubleshooting next. However, that's more difficult than trying settings in software, so before I pull the fixture out of the ceiling and try out different rewiring options I wanted to try posting here to get additional input.
Sharing more about the situation, I'm not familiar with commercial lighting fixtures and pros/cons and what brands are good/bad. With the residential fixtures in my home that have sockets for replaceable lamps, I've got a good perspective on what brand bulbs work well for the areas where I use Insteon dimmers. I picked up this LED troffer as it was aesthetically pleasing for the application I needed - filling a big hole in the area above/outside my front door where previous owner placed a 48" fluorescent fixture with a drooping plastic lens tacked up over the cutout (big eyesore). Anyway, the hardware store had this nice looking troffer fixture, it checked the mark of having the ability to dim (0-10V), LED works well in the cold and I knew Insteon had ballast dimmer supporting that.
Here's a link to the manufacturer of the fixture - https://lithonia.acuitybrands.com/products/detail/229368/lithonia-lighting/vtl/vt-series-led-troffer to give some perspective.
I fully understand that I may need to replace the fixture to solve the issue. If anyone has had success with similar troffer fixtures that don't flicker/flash when turning off from 40% brightness, I'm open to suggestions. As part of the renovation for this fixture - I now have a standard 2x4 cutout with a nice metal trim in the ceiling so swapping to another manufacturer's fixture would be possible. I also observed that the LED section behind the fixture lens is a Samsung PCB strip that's attached to the metal frame. So another possibility could be to gut the fixture of its LED PCB/driver and replace with something else that's more friendly with the Insteon hardware. Presently, I don't know what that LED hardware would be, so any suggestions on those lines would also be appreciated.
Thank you for the suggestion and hope the above additional background helps.