An extensible kernel-mode audio (e.g., MIDI) processing architecture is implemented using multiple modules that together comprise a module graph. The module graph is implemented in kernel-mode, reducing latency and jitter when handling audio data by avoiding transfers of the audio data to user-mode applications for processing. In one embodiment, the audio processing architecture is readily extensible. A graph builder can readily change the module graph, adding new modules, removing modules, or altering connections as necessary, all while the graph is running.