I am having the same basic issue. Hitting the set button makes the address show up in start linking. However, it fails every time for insteon engine. My keypads are dual band as are my PLMs. I have tried multiple keypads and multiple PLM and I see the same basic issue both in ISY and Indigo. These keypads are installed in a corporate environment so I can't control what phase they are on and I wouldn't be surprised if there is a power line issue. However, since everything is dual band I don't understand why I need power line for this to work anyway. The keypads and the PLM are in the same room. I have some RF only insteon motion sensor right next to the keypads and they link no problem.
Is good power line communication necessary to link dual band devices? Shouldn't I be able to link using only RF?