In response to the promotion of smart city and smart home, people pay more and more attention to the quality of life, wishing the technology could change our life. In recent years, with the use of GPU and big data, deep learning has brought revolutionary progress to various fields, especially in the area of computer vision. But the GPU accelerating is limited to power and cost, which makes its product extreme expansive. This paper will implement an unsupervised face recognition system with the deep learning technique, accelerating the algorithm with the combination of FPGA and ARM, in order to realize an access control system. The recognition system can only be executed when there is an object near the camera, and only compute the detected face. Results shows that each face is taken around 0.25 seconds to compute its feature on de10-nano platform to achieve low power consumption. Additionally, our design can retain the 94% accuracy on VGGFACE2 dataset, 99.2% on LFW dataset.