This paper presents a complete solution for real time, 30 frames per second, object detection and tracking using a SSD architecture for object detection in combination with a proportional controller for tilt-pan camera adjustments for tracking. It addresses several problems including the lack of real time capability of the SSD architecture on low powered devices and the choice of the right feedback controller for this application. The real time aspect is analyzed by comparing different input resolutions at different frame rates across the entire system. It is found, that the resolution size correlates with the inference time of the neural network in a proportional manner, achieving up to 3.5ms at a resolution of 160x120x3 pixels with special hardware acceleration. It is also found that a proportional controller is better suited than a proportional-integral-(derivative) controller in this context by looking at the unit step responses. The interaction between object detection and tracking is investigated in tests of oscillating motions. The system can track a red ball in a pendulum motion with more than 160◦per second. It proves the real time capability of the developed system and builds a foundation that can be adopted in other applications.