Bryston SP3 problem
 

HDMI from an Oppo 205 goes to the DVD input of the SP3 and HDMI from a Cox Contour cable box goes to the Cable/Satellite input. The SP3 video output goes to a projector, front LR audio via balanced cable to an Ayre preamp, and LR surround via balanced cable to a NAD 328 class D amp. When I play a Blu-ray I get video with front LR audio from the front speakers as expected, but with cable I get video with front LR audio from the LR surround speakers. What am I doing wrong?

The surprise is that the NAD 328s driving the KEF LS50 surrounds sounds good although it isn't correct, but it is an energy efficient way to enjoy TV without firing up the Ayre preamp and amp. The surround LS50s project the front LR signal in a way you can imagine the voices coming from the projection screen.

Is this a new install?

If not, did it work fine before? Have you tried connecting the hdmi from the cable box to another hdmi input on the SP3?
If sound comes out correctly, the issue would be with the satellite input of the SP3.

Does the satellite box have other digital outputs that you can try out?

If have experienced issues of my own with cable running through the SP3. The issue in my case was that projector would turn itself off randomly when watching cable. After some troubleshooting I have zeroed in on the satellite box. It appears to be that the cable box is sending some odd signal that makes the projector turn itself off. Right now, I am running the cable hdmi through my Oppo 205 hdmi in connection. I can see that the screen going dark every so often but the projector stays on. I think that in my case my cable box is defective and the Oppo is fixing whatever odd signal is coming from the cable box. You could use your Oppo for troubleshooting your issue.

Appears to be a down mixing problem. The OPPO does have an audio test signal you can direct to any of the (up to 7.1) speakers. Perhaps use the HDMI audio out of the OPPO into the cable input of the SP3 and start there.

Bryston SP3 problem
 

If I understand the OP correctly, he has no problem when playing discs from the Oppo. In other words, the Oppo-SP3 connection works fine.
His problem is with the cable box-SP3 connection.

The Oppo 205 has an HDMI-in port so presumably I could connect the Cox cable to that. I wonder which has the better DAC process, the newer 205 or the older but much more expensive SP3.

Bryston SP3 problem
 

Connecting the HDMI from the cable box to the Oppo HDMI-IN is exactly what I am doing and what I suggested in my earlier post. I still think that you have a cable box problem and not a SP3 problem.

The DAC of the Oppo 205 is way better than that of the SP3. The SP3 is fairly old technology for an electronic device.

Thanks, nicoff. I'll try cable to the HDMI input of the 205. I suppose HDMI Main will display video for Blu-ray and cable. Unfortunately, the 205 has no USB out that could go to the Ayre QX-5/20, so I'd use balanced analog to a by-pass input of the KX-5/20 preamp and unbalanced analog to the surround amps. In effect, the 205 becomes the distributing processor, replacing the SP3.

Yes, the Oppo becomes a distributor processor replacing the SP3. To select which source you are seeing, you just change the input on the Oppo using the Oppo remote.
You would select input named "HDMI- IN" to watch cable or the input named "Blu-Ray" to watch a disc.
By the way, the Oppo also has digital outputs (coaxial and optical) that you can connect directly to the Ayre QX-5/20 instead of using balanced analog from the Oppo.

Aren't both coaxial and optical bandwidth limited? The QX-5/20 doesn't have HDMI in and the Oppo 205 doesn't have USB out. The DAC of the QX-5/20 would be the one to use if I could. Most of my listening is to Roon files sent to the QX-5/20 via ethernet, and the sound can be superb, especially with DSD and 192/24 files.

Coaxial and optical can do 192/24. That's the maximum, but it looks to me that this is what you desire. So both coaxial and optical will do what you want.