A 1 square meter solar panel that weighs around 10kg contains under half a kg of actual doped silicon photovoltaic cells, with the remaining >95% being mainly aluminium framing, glass to cover the cells, and some backing material and wiring, all of which are trivially recyclable. So even if the doped silicon wasn't recyclable (which it is with a little more effort than the other materials) that's hardly the majority of the material that goes into making a solar panel.