Response time has started to play an increasingly important role in educational and psychological testing, which prompts many response time models to be proposed in recent years. However, response time modeling can be adversely impacted by aberrant response behavior. For example, test speededness can cause response time to certain items to deviate from the hypothesized model. In this article, we introduce a robust estimation approach when estimating a respondent's working speed under the log-normal model by down-weighting aberrant response times. A simulation study is carried out to compare the performance of two weighting schemes and a real data example is provided to showcase the use of the new robust estimation method. Limitations and future directions are also discussed.