This thesis is related to the broad subject of automatic motion detection and analysis in video surveillance image sequence. Besides, proposing the new unique solution, some of the previous algorithms are evaluated, where some of the approaches are noticeably complementary sometimes. In real time surveillance, detecting and tracking multiple objects and monitoring their activities in both outdoor and indoor environment are challenging task for the video surveillance system. In presence of a good number of real time problems limits scope for this work since the beginning. The problems are namely, illumination changes, moving background and shadow detection. An improved background subtraction method has been followed by foreground segmentation, data evaluation, shadow detection in the scene and finally the motion detection method. The algorithm is applied on to a number of practical problems to observe whether it leads us to the expected solution. Several experiments are done under different challenging problem environment. Test result shows that under most of the problematic environment, the proposed algorithm shows the better quality result.