Dual seed semiconductor photodetectors and methods to fabricate thereof are described. A dual seed semiconductor photodetector is formed directly on an insulating layer on a substrate. The dual seed semiconductor photodetector includes an optical layer formed on a dual seed semiconductor layer. The dual seed semiconductor layer includes a seed layer and a buffer layer. The seed layer of a first material is formed on an insulating layer over a substrate. The buffer layer is formed on the seed layer. Next, an optical layer of a second material is formed on the buffer layer. The buffer layer includes the first material and the second material. In one embodiment, the first material is silicon. In one embodiment, the second material is germanium.