A scalable port controller architecture supporting data streams of different speeds. In an embodiment, a port controller contains high speed receptor units and low speed receptor units, and a port routing logic connecting each external device (on corresponding port) to one of the receptors according to various registers. The port routing logic may connect an external device to one of the receptors, which determines the data rate at which data on a corresponding virtual connection from the external device is being received/sent. If the receptor does not have sufficient capacity (based on the data rate) to communicate with the external device, the connection is moved to other receptors, potentially in another control unit.