According to data from the ASEE, women were awarded 23.1% of doctoral degrees and held 15.7% of tenured/tenure-track faculty faculty positions in 2015 versus 21.3% and 12.7% in 2009, respectively [1, 2]. While promising, the “leaky pipeline” remains a persistent problem in the recruitment of underrepresented people into tenure track positions. To help overcome this barrier, we have created a program to improve the competitiveness of underrepresented applicants in the tenure-track faculty recruitment process at a large public university. The program activities are two-fold. First, seminars and panel discussions led by faculty representing different engineering disciplines cover a variety of topics related to the job search process. Second, peer review sessions over the course of several months allow students to develop their own application materials. Since its founding in 2014, the program has been evaluated by considering four elements: content, format, pace, and climate. The evaluations in the first two years were based on conducting pre-, mid-, and post-surveys as well as voluntary one-on-one exit interviews. For the program’s third year, we made significant changes based on past participant feedback. Specific topics were expanded to increase understanding and improve familiarity with the application process. The evaluation structure was revised to increase immediate feedback. In this paper, we discuss how the program has evolved over the three years as well as how our methods for program monitoring have been revised. By incorporating these changes, we aim to continue to prepare high quality female faculty candidates, thereby diminishing the gender gap in engineering academia.