While fully device-independent security in (BB84-like) prepare and measure Quantum Key Distribution (QKD) is impossible, it can be guaranteed against individual attacks in a semi device-independent (SDI) scenario, wherein no assumptions are made on the characteristics of the hardware used are made except for an upper bound on the the dimension of the communicated system. Studying security under such minimal assumptions is especially relevant in the context of the recent {it quantum hacking} attacks wherein the eavesdroppers can not only construct the devices used by the communicating parties but are also able to remotely alter their behavior. In this work we study the security of a SDIQKD protocol based on the prepare and measure quantum implementation of a well-known cryptographic primitive, the Random Access Code (RAC). We consider imperfect detectors and establish the critical values of the security parameters (the observed success probability of the RAC and the detection efficiency) required for guaranteeing security against eavesdroppers with and without quantum memory. Furthermore we suggest a minimal characterization of the preparation device in order to lower the requirements for establishing a secure key.