Pre-defined performance tools enable instrumentation and optimization of a heterogeneous program by modifying a platform-neutral intermediate representation (IR) of the program. The intermediate representation is hierarchy of base elements that correspond to instructions, code blocks, procedures and components within the program. Each base element exports an application program interface that provides for navigation, querying and modification of the corresponding element in the hierarchy. An application program interface for the entire intermediate representation includes the application program interfaces for the individual elements along with additional functionality. A user can request instrumentation and optimization of any level of the intermediate representation hierarchy and the performance tools perform the required modifications to the intermediate representation. When a modification requires inserting instructions, the performance tools add prologue and epilog code to connect the inserted instructions with the original intermediate representation instructions. When the intermediate representation instructions are translated into platform-specific instructions and executed, the modifications made to the intermediate representation provide the instrumentation and/or optimization of the heterogeneous program.