Gluing the button to the tact switch may or may not work depending on the switch. If it's the orange squishy ones, that could be fine. If it's a standard black tact switch, that wouldn't work because the switch rotates.
Also as mentioned above, the second solution is to use a joycon housing and frankencase it into your enclosure.
Third option would be to use a 3d printed case which has the orientation blockers built in. Personally I'd go that way, but I don't know how you're building your case.