Ok, so I plugged in the RCA to a TV and this is what I got.
the gray bar to the left of the screen is like what is happening on the G-boy screen, but not nearly as bright or flashing. What I am getting on the RCA output is a black screen with the gray bar that is in the picture. I'm not exactly sure what to make of this.
I have been retracing my steps with the g-boy and I still can't find what is causing the glitchy video. My only guess is that the u10 I have soldered in may be dead, but I do not know how to test if that is true. I tried plugging in the controller board that came with the display in the g-boy kit to see if I could get the screen to display at least something else, and I was able to bring up the screen settings via the controller board.
At least this means that the screen is definitely working and doing what it is supposed to, but its the motherboard which is outputting the glitchy video. I don't know if this video is a side-effect of the trim being bad or not. Besides buying another Wii to take the u10 from to try and replace the one on my board in hopes it will work, I am at a loss at what to do.
The best way to test if you U10 is functioning properly is powering on the portable, testing for 3v3 at the power input pin (top left) of the U10 with a multimeter, testing for the delayed 3v3 signal at the output pin (top right), and then checking for that same delayed 3v3 at the U10 via. If all of that looks good, chances are your U10 is working correctly. If your U10 is busted, you can recreate the effect with an RC delay circuit.
Have you checked all of your main voltages to see if they are correct?
You'll need a multimeter, set to read DC voltage. Put the black probe on any ground point around the Wii (check the definitive trimming guide for ground points, though the rings around the screw holes are a generally good spot). Put the red probe on whatever voltage you are trying to measure (top left pin of U10, top right pin of U10, as mentioned in my last post). You can check the definitive trimming guide and / or G-Boy manual for the main voltage points (5v, 3v3, 1v8, 1v15, 1v).
I just came back from testing the voltages and to my surprise, I am getting all the right voltages. Even on the via from the u10, so I suppose this means the u10 is good? GingerOfOz said that the screen flickering probelm was probably a u10 issue, but if it isn't the u10, then I don't know what else it could be.
Okay, that's pretty much a full charge on the batteries, so that's ruled out. Just to double check, did you test your resistances between your voltage lines on the Wii before wiring it up to everything else? If not, I would recommend disconnecting the Wii from everything else, then following this guide.
Ok, so I'm back from de-soldering everything and testing voltages, and here is what I got. The board I trimmed was the C/RVL-CPU-60 model. Most everything that my multi-meter read did not line up with the spreadsheets on the Wii trim resistances thread, but they only have the voltages for the CPU-40 and CPU-50 boards.
For your resistances between 3v3 and gnd / 1v15 / 1v8, were those readings 16.3 ohm or 16.3k ohm? If they were just 16.3, that would be indicative of a short. It also looks like your resistance between 1v15 and gnd is low (assuming it really is 1.9-2.5 ohms). That's almost a dead short, and could potentially messing with the power to the CPU / GPU, potentially leading to your issue. I would try resanding the edges of the Wii, see if that clears up any shorts.
What grits of sandpaper did you use when you first sanded your board? You should be using 120, 220, 320, 400, 600, 800, and 1000 for the best results.
The 16.3 ohm readings were actually 16.3k, I just messed up typing. I used 150 grit sandpaper initially, and I could see the copper shining, like the g-boy guide specified, so I thought it was good. I'll re-sand the edges with some finer sandpaper. I'll be back with the results soon hopefully.