A computer model of a physical structure (or object) can be generated using context-based hypothesis testing. For a set of point data, a user selects a context specifying a geometric category corresponding to the structure shape. The user specifies at least one seed point from the set that lies on a surface of the structure of interest. Using the context and point data, the system loads points in a region near the seed point(s), and determines the dimensions and orientation of an initial surface component in the context that corresponds to those points. If the selected component is supported by the points, that component can be added to a computer model of the surface. The system can repeatedly find points near a possible extension of the surface model, using the context and current surface component(s) to generate hypotheses for extending the surface model to these points. Well-supported components can be added to the surface model until the surface of the structure of interest has been modeled as far as is well-supported by the point data.