to download the full report (PDF document) click here
Background
Students who have difficulty with a traditional paper examination question paper can apply to SQA to use ‘Assessment Arrangements’ (AA), such as extra time, scribe or reader, transcription of the paper, word processor or ICT, signed interpretation, or an adapted paper (Braille, enlarged, or printed on coloured paper). SQA publish guidance for schools to help them to choose and use the most appropriate assessment arrangements, and a key part of this is ensuring that schools meet duties under equality legislation including the Disability Discrimination Act. The main principles of this guidance are to allow reasonable adjustments to be made where candidates have difficulty accessing questions or presenting written responses, while ensuring that any adjustments do not compensate for a candidate's inability to meet set standards. In addition, these arrangements should be tailored to meet the individual needs of candidates and should reflect, as far as possible, the candidate’s normal way of learning and producing work.
In 2006 there were 43,291 requests to use AA in SQA examinations on behalf of 10,650 students . This represents 7.22% of the total number of candidates and 5.91% of the total number of entries. The use of alternative arrangements varies with the level: a larger proportion of candidates at Standard Grade used AA compared with Higher, for example. The largest single identifiable group of candidates who requested AAA are those with specific learning difficulties, with 6,965 candidates using arrangements in 29,002 examinations. Pupils can use more than one type of support in exams (e.g. extra time, plus reader, say) – the 43,291 requests resulted in 77,374 individual instances of support.
Why Digital question papers?
There were 16,815 requests for the use of a reader, and 15,059 requests for a scribe for SQA exams in 2006 in Scotland: 16,815 instances where pupils were sitting in separate rooms in schools in Scotland, with a member of staff scribing and/or reading for them, with a personal invigilator. Such widespread use of scribes and readers raises questions about independence and objectivity of assessment. In addition, schools have difficulty finding accommodation and staff, and there are significant costs for staff and invigilators.
ICT is used by pupils requiring AA and there were 3,063 requests to use ICT in SQA examinations in 2006, but in the main these were to support writing and recording. Given that more pupils seem to need a reader than a scribe, it is clear that we should ask ourselves how ICT can be used to support both reading and writing in examinations.
In 2002, CALL carried out an investigation for SQA into the use of ICT for candidates with disabilities and a key recommendation was that papers should be made available in a digital format, so that candidates could easily open the document and use a range of different technologies to both read the paper and write answers. (Although SQA do permit staff to open the paper slightly in advance of the start of the exam, and scan the paper(s) into the computer using a scanner and optical character recognition software, it adds an extra layer of complexity and uncertainty which is not helpful during an examination.) CALL researched the features that were required with digital question papers and found that staff wanted papers which:
- had the same appearance and layout as the paper, so that students could use both digital and hardcopy during an exam;
- could be magnified and adjusted (for example by changing foreground and background colours on screen);
- would allow students to type directly into the paper on screen;
- were compatible with assistive technologies such as screen or text reader programs, specialised keyboards or access systems, or speech recognition software.
Digital Question Papers
The next step was to develop digital papers to this specification, and to evaluate them in schools. We chose to create the papers in Adobe Portable Document Format (PDF) because this format is low cost (for both SQA and schools); interactive (unlike digital book formats such as Microsoft Reader or Daisy); and accessible for the majority of students with additional support needs.
We aimed to use cheap, easy, scalable ICT tools that would fit into the existing SQA procedures and working methods and that could be implemented quickly. The digital papers are another type of adapted paper, like enlarged, colour or Braille papers, to be distributed to schools on CD as discrete PDF file, and used and marked using the same procedures as all other papers. Anything which would add cost or complexity for pupils, schools or SQA was avoided where possible because this would reduce reliability and make it less likely for digital exams to become eventually available. The digital papers were delivered to schools on CD (rather than ‘online’) so that they could be accessed both on networked and standalone computers. In addition, networked computers in schools can be unreliable and are often inaccessible for pupils with additional support needs because network managers do not permit satisfactory access to the accessibility tools in the Control Panels, and network restrictions can prevent installation of specialist software. By using Acrobat and the free PDFAloud text reader, cost to schools is very low and since we aiming for eventual implementation on a large scale (i.e. uptake by a reasonable proportion of the aforementioned 10,650 pupils sitting 43,291 examinations) it is important to keep costs and complexity to a minimum.
Use in 2006 examination diet
In Autumn 2005 CALL and SQA created a bank of 163 digital past papers from 2003, 2004 and 2005 for students and schools to use for practice and revision. The trials using the digital past papers went well, and 31 students used digital papers in 105 examinations in May/June 2006. The pilot trials were very successful: the papers themselves were reliable; staff and students were enthusiastic; and results obtained in line with other support methods. SQA therefore intend to offer digital question papers on a wider scale in 2007.
The full project report is available here <INSERT URL HERE>
The report summary, together with comments from students, is given below.
Feasibility Study Report Summary
- ; Candidates with disabilities or additional support needs can request permission to use one or more Assessment Arrangements in SQA examinations i. The number of students for whom assessment arrangements have been requested has increased by over 300% in the last decade; from 3,094 in 1995, to 9,814 in 2005 . In 2005 there were requests for Assessment Arrangements for use in 41,454 examinations. Students may request more than one method of support for the examination (e.g. extra time and also use of a scribe), resulting in 74,720 individual instances.
- The most common type of request is for extra time (33,146 requests in 2005), followed by a reader (15,740 requests) and then a scribe (14,505 requests) v. Each examination sat using scribe or reader requires separate accommodation, a member of staff to read and/or write, and an invigilator, and so there are significant resource costs attached to this type of support. More importantly, the student is not independent when sitting the examination.
- Digital question papers in accessible PDF were successfully trialled in a previous project in 2004-05 . Candidates with reading or visual difficulties can click on the text or questions on the digital question paper and the computer will read it out, and answers can be typed into the question and answer papers on screen.
- The aim of the project described in this report was to build on this experience by creating a bank of digital past question papers for schools and students to use for revision and practice. If these proved successful and practical to create and use, SQA undertook to consider offering digital question papers for students in the 2006 diet of examinations.
- The digital question papers were produced in Acrobat PDF because of relatively low production costs for SQA, good functionality and reliability, and low cost for schools. SQA already produce question papers in PDF, and so there is no need to re-design the papers. It is necessary to manually draw ‘form fields’ into the question and answer papers, and to check that these form fields are accurate and functional. Each paper was also ‘speech-enabled’ using TextHelp Systems’ PDFAloud Publishing Toolkit: a process which requires a single mouse click (TextHelp 2006). The PDF papers are reasonably accessible for most candidates: they can be magnified; colours altered; accessed using the keyboard instead of the mouse; and the PDFAloud text-reading software is helpful for candidates with reading and visual difficulties. Cost to schools are relatively low: to use a digital question paper the candidate must have access to a computer with Acrobat Standard (at around £25 per licence) and the free version of the PDFAloud text reading software installed.
- A bank of 163 digital past papers across 28 different subject areas was created and used by 73 students in eleven centres in the 2005-06 session. Question papers were requested at all levels: Standard, Intermediate 1 and 2; Higher and Advanced Higher. Feedback from staff suggests that the students found the digital question papers very helpful for independent revision and practice.
- Eight of the centres went on to request 152 digital question papers for use by 34 students in 111 examinations in May 2006, and SQA staff created 57 separate digital question papers across 18 subjects. SQA estimate that the production time to create digital question papers is slightly less than the time to produce large print hard copy papers.
- 31 out of 34 students used the digital question papers in 105 examinations (three students chose to use other arrangements for six examinations).
- SQA analysis reveals no significant difference between the estimated and actual results obtained by the candidates using digital question papers, although the small numbers of candidates involved makes analysis difficult. Digital question papers do not appear to have an impact upon results achieved compared to other types of assessment arrangement.
- Students were asked to complete a questionnaire after each examination. 76 questionnaires were returned in respect of 92 out of 105 examinations.
- Most of the students who opted for the digital question papers chose to use them because of reading, writing or spelling difficulties. A small number of students had physical, visual and other issues.
- 10 students out of 31 used PDFAloud text-to-speech software to read 35 out of 105 question papers. The use of text-to-speech to read the question paper varied greatly between different schools: all the candidates at two schools used PDFAloud to read every question paper they sat whereas it was not used at all by any candidates in five schools. In one school, PDFAloud was used to read question papers in 25 examinations, while human readers were used for 27. If this level of use could be extended across other schools in Scotland it would have a significant positive impact upon the ability of thousands of students to demonstrate attainment more independently, and upon staffing and resources in schools.
- In 68 of the 92 examinations, digital question papers were in question and answer format where candidates could type their answers into the question paper on screen. Candidates used this option in 60 cases. Feedback from candidates suggests that digital question and answer papers are easier to use and less confusing than using a word processor or scribe. Again, there is considerable potential for digital question papers to increase candidates’ independence and to reduce demands on resources.
- Students were enthusiastic about the digital question papers, with 28 out of 31 stating that they would use them again (3 were not sure) and all students who responded felt that SQA should provide digital question papers alongside other types of adapted question papers. Candidates found typing into the digital papers faster, easier and in some cases less painful than handwriting; more private than using a reader; and less stressful. This is what we would expect given that the students elected to use the digital question papers themselves, after practicing with past papers. Note that digital question papers will not suit every candidate who might require assessment arrangements because a reasonable degree of ICT and keyboarding/text production expertise is required, and because some candidates will feel more comfortable with other types of support.
- Interviews were conducted with staff involved in the pilot. Most staff felt that most students were more confident, independent, motivated and skilled when using digital papers, than with traditional papers and accommodations. All felt that SQA should provide digital adapted question papers in future. Centre staff found the digital question papers to be reliable (average of 4.75 out of 5 for reliability). Some schools had difficulty finding and organising sufficient numbers of accessible computers, and this may be a barrier to increased use of digital question papers in future. Support from local authorities and senior school management is necessary to ensure that satisfactory hardware and software can be made available.
- Overall, staff felt that the demands on resources were lower when using digital question papers compared to traditional assessment arrangements (average of 2.88 compared to 4 (on a scale of 1 to 5) in terms of staffing; 2.25 compared to 3.5 in terms of accommodation; 2.25 compared to 3.5 in terms of invigilation). There are of course increased demands for ICT resources, and for technical support.
- Staff were positive about support and communication from SQA, giving the effectiveness of the information and guidance an average of 4.5 out of 5, and the effectiveness of support 4.875 out of 5.
- Adapted digital question papers appear to offer considerable benefits. The question papers produced by SQA are reliable, relatively inexpensive to produce and staff estimate that demands on accommodation, staff and invigilation are lower than traditional support methods. More importantly, the students who used them in May 2006 found them effective; students using digital question papers are far more independent than those using other types of support such as scribe and reader; and results obtained using digital papers are in line with results achieved using other methods.
Comments from students on why they wanted to use a digital paper for the exam?
To help me along with spelling and writing
Because it's easier to read than a paper copy. It's easier for me to describe my answers directly rather than using a scribe.
Typing into the paper is much easier than writing. Prolonged writing is difficult and causes a lot of pain.
Writing for long periods is painful. My writing is illegible - even to myself! It avoids the need for a scribe. I don't like using a scribe.
It is easier and quicker. It is preferable to see the question when typing in an answer, rather than typing into a blank document using a Word Processor.
Comments from students on the digital exam
It's easier and less stressful than having to write it yourself.
Papers are simpler to use. You do not have to fiddle about with different papers.
By switching windows, you can see both the question and the text at the same time.
Digital paper is easier to use. It is easier to type answers in rather than write them in.
It saves time. It is simpler to use. You don’t have to fiddle with lots of paper. You can see the text and the questions at the same time.
It makes it easier for many pupils to do their papers this way.
Because it is much easier to use than a reader.
Much easier than trying to write!
Prefer it to using a scribe. Easier to give answers.
Easy to do. You can see both the questions and the text at the same time.
Enjoyed using it!
