You may need to connect the joy sticks/buttons to a a microcontroller, then plug that into the rp4's USB, if you use an Arduino Micro, you can use dmcke code for it https://github.com/dmcke5/Ardupad. It's very good and easy to use.
I tried compiling it for the teensy 2 but as you thought the joystick library is not compatible, I think its the PluggableUSB.h and DynamicHID.h that do not work with the teensy, shame!
I