A first set of semiconductor devices is formed on a first semiconductor substrate comprising a first semiconductor material having a first melting point. A first via-level dielectric layer containing first contact vias is formed on the first semiconductor substrate. A second semiconductor substrate comprising a second semiconductor material having a second melting point lower than the first melting point is formed either by bonding or deposition. A second set of semiconductor devices is formed on the second semiconductor substrate. A second via-level dielectric layer, second contact vias contacting the second set of semiconductor devices, and inter-substrate vias electrically connecting the first contact vias are thereafter formed. A metal interconnect layer containing a metal interconnect structure is formed over the second via-level dielectric layer to electrically connect the first and second set of semiconductor devices through the second contact vias and the inter-substrate vias.