Conversational agents constitute a specific type of ITSs that has been reportedly proven successful in helping students in one-to-one settings, while recently their impact has also been explored in computer-supported collaborative learning (CSCL). In this work, we present MentorChat, a dialogue-based system that employs a configurable and domain-independent conversational agent for triggering students' productive dialogue. After a system overview with an emphasis on design rationale and system architecture, we present a pilot study, where the agent is evaluated in the context of the computer-assisted language learning (CALL) domain. Thirty students collaborated in small groups trying to accomplish three different tasks. MentorChat conversational agent supported each group discussion differently in each task providing (a) "weak"-directed interventions and/or (b) undirected interventions. The study results indicate that "weak"-directed agent interventions can be more effective than undirected interventions by means of increasing the level of explicit reasoning, and thus productive dialogue. Some encouraging results concerning the system usability and user acceptance are also presented.