You may need to connect the joy sticks/buttons to a a microcontroller, then plug that into the rp4's USB, if you use an Arduino Micro, you can use dmcke code for it https://github.com/dmcke5/Ardupad. It's very good and easy to use.
I tried compiling it for the teensy 2 but as you thought the joystick library is not compatible, I think its the PluggableUSB.h and DynamicHID.h that do not work with the teensy, shame!
I
Here's the link to the setup video, note you do not have to connect it to your Wi-Fi as the video says, so even quicker to setup, I will have a look if you can turn the WiFi off to use less power!
Have you thought of ditching the orange pi and using a TP-Link AC750 Wireless Portable Nano Travel Router with smb, works really well and easy to set up, not to large?...well maybe?