A composition for a slurry for manufacture of a building panel includes one or more fillers, one or more binders and water. One or more fillers is an expanded perlite at least partially covered with a film-forming, thermoplastic polymer coating Some embodiments use calcined or uncalcined gypsum as a binder or filler. One embodiment is a building panel that includes a solid matrix of calcium sulfate dihydrate crystals bound to expanded perlite. A method of making the panel includes acquiring expanded perlite and coating it with film-forming, thermoplastic polymer coating. After the expanded perlite is prepared, it is combined with water other optional fillers and one or more binders to form a slurry. The slurry is shaped into a panel and allowed to dry.