The apparatus comprises a double walled vacuum insulated vessel defining a cryogen space for holding a cryogenic fluid, a pump assembly comprising a pump with a suction inlet disposed within the cryogen space, and at least one elongated member extending from the pump to a drive unit disposed outside the cryogen space. The elongated member comprises an elongated non-metallic section that has a thermal conductivity that is less than that of a structurally equivalent elongated stainless steel member of the same length. In preferred embodiments, the elongated member can be one or both of a drive shaft or a rigid structural member for supporting the pump and holding it in a fixed relationship to the drive unit. The method employs the apparatus to increase hold times for holding cryogenic fluids by reducing heat leak into the cryogen space.