What is the main thing that is holding back optical computers or quantum computers?

Well other than the price, for quantum computers there is also the tough part of manipulating individual atoms for calculation.  So far the highest number they have been able to control is very insignificant.  Unless they can control a huge number of atoms, quantum computers is still put on hold. In a few decade or so, when silicon computers reach their limit, quantum computer would most likely be ready.

I work in a defense funded company in the US that works on quantum computers. At this moment, there is nothing that even remotely works like a quantum computer, at least not in the concepts proposed back in 1998 when I was snoring through endless highly speculative seminars about this technology. Some prospects have been proposed that show some quantum effects, but nothing like you would expect.

The list of things that are holding this technology are too numerous to mention; some of them are even classified. All I can tell you is that for this technology to sort of work you need to cool down devices to extremely low temperatures (mili Kelvins) and that is a major pain in the rear end.

Edit: Optical computers stop being in the map as something desirable. I remember back in early 90s there was this push towards optical computers and it faded away towards 1998. I became clear that optical computers did not offer any real advantage over conventional electronic computers.