This thesis presents methods for automatic scene analysis in surveillance using keypoint cameras, which aim to replace original images with a stream of keypoints for object categorization and recognition, enabling higher semantic scene knowledge. Traditional image-based foreground segmentation is unsuitable for keypoints, prompting the development of new techniques to segment keypoints into foreground and background. The focus is on replacing image-based change detection with keypoint-based motion segmentation, addressing privacy concerns and overcoming challenges like shadows and non-static cameras. The work begins with segmenting keypoint correspondences between images based on common motion. Multiple frame-to-frame segmentations are then combined to form trajectories, which are segmented according to motion over time. An innovative approach is introduced to describe objects lacking dense texture, using a no-feature (NF) method to fill undescribed areas with NF keypoints that can be matched in subsequent images. Two applications are demonstrated: online learning of object keypoints through motion and pixelwise segmentation of images based on motion-segmented keypoints. Experiments reveal that the motion segmentation approach achieves a mere 23% error compared to state-of-the-art methods and operates in real-time. Additionally, NF keypoints outperform regular keypoints in precision and recall, significantly enhancing the a
