We introduce a minimal Agent Based Model for financial markets to understand the nature and Self-Organization of the Stylized Facts. The model is minimal in the sense that we try to identify the essential ingredients to reproduce the main most important deviations of price time series from a Random Walk behavior. We focus on four essential ingredients: fundamentalist agents which tend to stabilize the market; chartist agents which induce destabilization; analysis of price behavior for the two strategies; herding behavior which governs the possibility of changing strategy. Bubbles and crashes correspond to situations dominated by chartists, while fundamentalists provide a long time stability (on average). The Stylized Facts are shown to correspond to an intermittent behavior which occurs only for a finite value of the number of agents N. Therefore they correspond to finite size effect which, however, can occur at different time scales. We propose a new mechanism for the Self-Organization of this state which is linked to the existence of a threshold for the agents to be active or not active. The feedback between price fluctuations and number of active agents represent a crucial element for this state of Self-Organized-Intermittency. The model can be easily generalized to consider more realistic variants.