Technology Assisted Student Assessment Institute NAVIGATION » Home » News & Views » About TASA Research » » TASA Agenda » » Phase 2 » » Clearinghouse » » Potential » Publications » Media Releases » Contact THIRD ANNUAL RESEARCH SYMPOSIUM: Innovations in Testing Technology Click here for information. An initiative of the Society for the Advancement of Excellence in Education

Phase 2 Research Program - An Overview Phase 2  consists of 6 applied research projects coordinated by TASA Director, Dr. Jim Gaskill. A highly qualified principal investigator has been selected for each study as funded, and a variety of partner agencies will be involved in the research. Initial funding has been provided by the Max Bell Foundation. Evaluation of an Online Assessment Pilot Principal Investigators: Jim Gaskill and Mike Marshall Important learnings about large-scale implementation of CBA can be derived from a baseline study of the Grade 7 Foundation Skills Assessment online pilot in 8 BC schools which commenced May 2005. Preliminary lessons from this pilot could be applied to Grade 10 Graduation Program Examinations (100 schools) online delivery planned by the Ministry of Education. The evaluation will include an analysis of the following: start up success and difficulties, expansion rate 2005-07, how tests work with technology platforms in the schools, accessibility, learners' experiences, ability to capture open-ended responses, procedures needed to prepare students and iron out problems that arise, validity issues and the link between assessment and instruction, cost analysis, marking reliability and turnaround time. Jim Gaskill Beginning as a teacher in Vancouver, Jim Gaskill returned to UBC where he got his Ed. D. in Mathematics Education. After teaching at Acadia University, Jim rejoined the K ­ 12 system teaching Mathematics and Computer Science in Surrey. Before joining the Ministry of Education in 1988, Jim spent 8 years as a principal in Queen Charlotte (Haida Gwaii).  His work at the Ministry has included managing provincial learning assessments, being in charge of the development of all provincial examinations and  Foundation Skills Assessments, chairing the consortium that developed a pan-Canadian mathematics assessment, and being the Canadian Representative on the PISA mathematics assessment Forum. Jim has also been responsible for implementing item banking and on-line testing for BC. Dr. Gaskill is Director of TASA Institute. Dr. Mike Marshall Dr. Marshall is Executive Director, Applied Research and Evaluation Services (ARES), UBC, and possesses extensive experience as an educator and in management, technical analysis and planning. Among the positions he has held are Director of Management Planning for the Ministry's Joint Education Management Projects, and Technical and Administrative Consultant to the Learning Assessment Branch. His responsibilities with ARES have included serving as International Director of Administration for the Third International Mathematics and Science Study (TIMSS), Project Director for TIMSS-Canada, and Project Director for PIRLS-Canada (Progress in International Reading Literacy Study), an international reading literacy study carried out in 38 countries, and technical advisor to the BC Ministry of Education. In recent years, the majority of his time has been spent in the development and implementation of UBC's English Language Proficiency Index Program, a computer-delivered suite of Speaking, Listening, Reading, and Writing tests designed to evaluate competence in English. Mike is a Research Associate at TASA Institute. Computer Scored Essays Principal Investigator: Susan Phillips Conventional testing programs spend millions of dollars on human scoring of essay answers. It is now possible for this to be done by computer at significant cost savings. More creative use of optical scan sheets to automatically score responses to open-ended questions could also produce efficiencies. Pennsylvania expects to save half of its $1.6 million cost of scoring writing examinations through automating the process. As there are currently no Canadian examples to study, we will review and synthesize all available reports to advance knowledge about the technical state of the art and the pedagogical implications of moving in this direction. This review could lay the foundation for a future study of a Canadian computer scored essays pilot, including examining teacher confidence in their use. Computer Effects on Student Performance Principal Investigator: Mike Marshall Some evidence suggests that students' previous experience with computers directly affects their performance on computerized tests. Investigation in this area may provide useful guidance relative to which types of tests to computerize, implications for transition from paper and pencil to electronic formats, and issues of comparability across formats. This proposed research will use the Grade 7 FSA to undertake a validity study of paper and online versions of the same test. A survey will be designed to capture data on student prior experience with computers as part of the analysis. Accessibility for Special Needs Students Principal Investigator: Alan Taylor The provision of equal opportunity for students with special needs to access examinations has important policy implications. Some examination programs provide for students with disabilities by allowing additional time, readers, Braille, use of scribes, etc. Many provisions come with serious drawbacks and may introduce reliability issues. Electronic assessment offers potential to offer enhanced accommodations for students with disabilities. This study will identify current testing accommodations offered for special needs students in the provinces and territories, document exemption policies in each jurisdiction and determine their potential impact on special needs students, determine accommodations that could be made through computerized testing, and recommend steps for provinces and territories to take in addressing these issues. It will also construct a research design for testing the validity of alternative accommodations in a follow-up phase. Dr. Alan Taylor Dr. Taylor is internationally recognized as a testing and measurement expert. He designed and implemented numerous program assessments in BC. From 1994-2002 he was the National Research Coordinator for Canada in the Third International Mathematics and Science Study. His administrative experience includes positions as Director of Student Assessment and Examinations for British Columbia and Director of Curriculum, Assessment and Research for a large urban school district. A former UBC adjunct professor, he is currently vice president of Raven Research Associates and working on a variety of international research contracts in assessment design. Co-author of Student Assessment in Canada (2001, SAEE). Establishing a Business Case for Transition Principal Investigator: Barry Carbol ROI (return on investment) evidence is an essential part of government decision making in shifting from paper to electronic assessment. TASA's 2004 pan-Canadian survey of Assessment Branches revealed that costs and related issues such as technical support were significant barriers to entry into the field of online assessments. This feasibility and ROI study will be conducted in conjunction with one interested province (Ontario has tentatively been identified). The province's current assessment programs will be examined and a cost/benefit plan developed for a phased-in conversion of one or more its components. It will include careful examination of all costs, including hidden ones such as professional development and infrastructure. It will also determine potential savings to be realized from distribution, scoring and reporting efficiencies; economies of scale; more extensive use of item banks; the establishment of partnerships and consortia; and pedagogical benefits. Sharing of costs, expertise, and technical support (development and implementation) will be pursued through potential partnerships with companies and institutions already involved in this endeavour. Dr. Barry Carbol Dr. Carbol is an experienced educator having spent 30 years in various capacities in public and private education. He was a senior executive in the BC Ministry of Education from 1984-1997. During his time with the BC Ministry, he had responsibility for provincial examinations, learning assessment programs, international assessments, curriculum development, learning resources, distance learning and education technology policy for schools. After leaving the Ministry, Dr. Carbol held senior management positions with the Open Learning Agency where he was responsible for the operations of the BC Open University, Open College, Open School, and Workplace Training Systems. Among his academic credentials, Dr. Carbol holds a Ph.D. in educational psychology (measurement and evaluation) from the University of Alberta, an M.Ed. in Curriculum and Instruction from the University of Oregon, and a B.Ed. from the University of Alberta. He has presented at national and international conferences on a range of topics from learning assessment to implementation of education technology and has published research and other articles on adult learning, learning assessment, and service quality. Dr. Carbol is currently the CEO of Merit Global Learning, a K-12 online education solution provider and President of Schmidt & Carbol Consulting. e-Portfolio Development Guide Principal Investigator: Kathryn Barker Use of the e-portfolio as an assessment tool is one of the fastest growing areas of assessment. In the province of BC, a portfolio is now a graduation requirement for students. A leading researcher in this field will prepare an overview of the costs and benefits, criteria for evaluating digital portfolios of student work, and sound pedagogical practices for districts and schools considering their introduction. This brief report should be of broad interest to classroom teachers and educators interested in e-portfolio effects on teaching practices and students' metacognition and ownership of learning. Dr. Kathryn Chang Barker Dr. Kathryn Chang Barker is a consulting education futurist with a PhD in Education Administration and Policy Studies from the University of Alberta (1994). She has been engaged at all levels of Canada's education and training system, and having started as a teacher and college administrator in Alberta, she has worked across Canada and internationally as a consultant and writer. Barker provides leadership in changing learning systems in the future for the future. As founder and president of FuturEd Consulting Education Futurists Inc., Barker has provided research and development in the education industry since 1995. Clients include HRDC in Ottawa as well as various provincial ministries and training institutions. Barker has expertise in the fields of eLearning; workforce development and lifelong learning; education reform in the context of global change; accountability, Return on Investment, and quality assurance in learning systems; international education and national standards development; adult and workplace literacy; learning technologies and distance education; prior learning assessment, human capital assets management, and various other fields related to learning innovations. Her current passion is the ePortfolio ­ the electronic record of a person's acquired skills and knowledge ­ and she is working to animate "an ePortfolio for each and every Canadian ­ and one ePortfolio for life." Barker has become a "consumer's advocate" in the field of education and training. She is concerned that the end-users get the information they need to make informed choices, and this is reflected in such FuturEd projects as the Consumer's Guide to eLearning and a Consumer's Guide to ePortfolio Products and Services. In 2002, Barker launched QualitE-Learning Assurance Inc. - a service to assess and certify the quality of on-line learning. Based on the Open eQuality Learning Standards that were created by FuturEd, QualitE-Learning licenses the eQcheck certification mark to products and services that meet consumer-oriented standards of excellence. Barker is the founding Chair of the Learning Innovations Forum d'Innovations d'Apprentissage, and also serves on the boards of the Canadian Education Association, the Innovate online journal, and the European Institute for eLearning. Online Resource Library Susan Phillips has been contracted as research librarian to locate, review, abstract and catalogue appropriate resources to create an up-to-date online clearinghouse of research on programs, innovations, tools, outcomes, best practice and policy implications. Research Symposia Two conferences will be convened in the spring of 2007 and 2008 to promote knowledge exchange and seed new R & D partnerships. They will bring together 50-75 researchers, the technology industry, education assessment officials, and district leaders to examine key issues, progress and barriers in development and implementation of cba. And finally, an ambitious Phase 3 research program and the partnerships, funding and infrastructure to achieve it will be developed in consultation with TASA associates, advisors and research partners. » Return to Top   » Research From the Field