Computer technology made its way into writing instruction and assessment with spelling and grammar checkers decades ago, but more recently it has done so with automated essay evaluation (AEE) and diagnostic feedback. And although many programs and tools have been developed in the last decade, not enough research has been conducted to support or evaluate the claims of the developers. This study examined the effectiveness of automated writing instructional programs in consistent scoring of essays and appropriate feedback to student writers. It examined the scoring and instructional program called "MY Access! Home Edition," which has an error feedback tool called "My Editor" to address these issues. The study combined a quantitative study of agreement and correlational analyses with an analysis of content and topic. Participants included 114 English language learners who wrote 147 essays to three writing prompts, which were graded by trained EFL raters and "MY Access." From the sample, 15 randomly selected essays were also used for an error analysis comparing "My Editor" with human annotations to examine "My Editor"'s accuracy. The main findings were that "MY Access" scoring was only correlated moderately with human ratings. Furthermore, because "MY Access" scoring is limited to the recognition of content words, not how these words are organized at the discourse level, it did not detect slightly off-topic essays and plagiarism. Finally, "My Editor"'s error feedback, with 73% precision and 30% recall, did not meet the expectations of an accurate tool. In conclusion, the home edition of "MY Access" was not found to be useful as an independent instructional tool. These findings give us pause regarding the effectiveness of "MY Access."