In my last post on this topic I explored the ‘conditioning’ and ‘coherence’ effects that Tim Oates claims at state level in his policy paper ‘Why Textbooks Count’. In that post I set out my concerns about the way the paper deals with the idea of ‘coherence’, and how this causes Tim Oates to overstate the effect of state or central control over textbooks’ content. This is a theme that I will return to in this blog-post. I’ll also be developing my critique of Oates’ treatment of the idea of ‘coherence’.

I want to focus on the effects that Tim Oates claims for textbooks at the classroom level in key jurisdictions.  Along the way I’ll suggest more ways in which the papers suffers from evidential problems, and point out the rhetorical devices which, in my opinion, are the sine qua non of a policy paper, rather than a piece of ‘New research’ as is claimed by the page title on which it is posted.

According to Oates, at a classroom level well-designed textbooks ‘free up teachers to concentrate on refining pedagogy and developing engaging effective learning’ (p.4). In ‘key nations’ they have also ‘been developed to support highly effective pedagogic practices’, where they also ‘encourage clarity regarding key concepts and core knowledge, [and] provide clear learning progressions’. This support enables teachers to provide ‘enhanced responsiveness to individual learner need’. None of this is controversial. I recognise these advantages from my own practice – and it would not surprise me if ‘high performing teachers [were] most supportive of the use of well-designed textbooks’ as Oates claims Reynold’s and Farrell’s 1996 study suggests (though I must admit I can’t find this claim in that study). I am also reminded of my own search as teacher and HOD for textbooks that were coherent with the curriculum aims that I had for my students.

An ambiguous comparison

One of the key rhetorical devices that Oates uses is a sense of crisis, of a problem with a daunting scale in the quality and use of textbooks in England.  In order to support the idea that the use of textbooks is ‘dauntingly’ low and problematic a confusing picture is then painted in which responses to the TIMSS 2011 questionnaire items about the use of textbooks from English teachers are compared with those of Finland and Singapore. Oates sets out the very low numbers of English teachers reporting that they use textbooks as ‘a basis for instruction’, and contrasts these with the much higher proportion of teachers from Finland and Singapore who report that they use textbooks as ‘a basis of instruction’.

However, the questionnaire itself asked whether teachers use textbooks as ‘basis of instruction’, or as ‘supplement’, not as ‘the basis for instruction’ nor yet as ‘a basis of instruction’ as Oates sets out in his table headings. I’m aware that this might seem like taking close reading to a pedantic level, but as I hope we’ll see – it’s important.

It is suggested that this comparison will ‘confound many common assumptions about these three countries’, though only two such assumptions are hinted at, which arise out of ‘high level messages regarding high school autonomy and learner-centred pupil support’ in Finland. I’m forced to presume that Oates thinks that from these high level messages many people assume that the Finns don’t use textbooks. This is not something that I’m aware of. If people did think that then they’re obviously wrong. Finnish teachers obviously value textbooks, as ‘basis of instruction’ and as ‘supplement’ (or whatever they were asked in Finnish).

In which case what else are we supposed to do with this comparison? Well we’re told that the problem (the one with the daunting scale) is ‘low use and low quality’ of textbooks in England (p.4), and here we’re asked to compare the use of textbooks in England with that of Finland and Singapore. Both are bywords for high performance in international comparative tests, whereas English performance in such international comparisons is commonly assumed to be dire. Is Oates comparing dire England’s ‘low’ use of textbooks with the high use of textbooks in high performing Finland and Singapore?

Why doesn’t Oates join these dots more directly? Possibly because there is no correlation between the proportion of teachers reporting that they use textbooks as ‘basis for instruction’ or as ‘supplement’ and a country’s TIMSS 2011 scores at any of the benchmarks that TIMSS used. In fact we might be able to confound a few assumptions by pointing out that in maths, far from being dire in comparison to Finland’s performance, they come out at similar levels, despite the ‘problem’ of low use of textbooks. Performance in Singapore was stratospherically better than in both England and Finland, yet in Singapore teachers report that they use textbooks as ‘basis for instruction’ less than they do in Finland.

A flawed comparison

What we can say from the TIMSS 2011 questionnaire is that in England 74% of 4th grade teachers of maths report that they use textbooks (either as ‘basis’ or as ‘supplement’) in their lessons, whilst 88% report that they use workbooks or worksheets, either as ‘basis for instruction’ or as ‘supplement’.

The omission of an ‘a’ or a ‘the’ in the question about ‘basis’ or ‘supplement’ is important and makes it harder for us to understand the responses of the TIMSS teachers. When I taught classes I used a textbook in the vast majority of my lessons. I would often use it as ‘a basis of instruction’ but not ‘the basis of instruction’. Sometimes I would also use a second textbook as a supplement if I felt that material was better covered therein. If I was asked whether textbooks were ‘the’ basis of instruction in my lessons I would say “no”. My own knowledge was “the basis of instruction”, though often a textbook would be used as ‘a basis of instruction’, with my knowledge informing how this instruction was done.

So, all in all we can say English teachers are less likely to report that textbooks are ‘basis of instruction’ in their lessons than in all other countries, although 78% report that they use textbooks in lessons. This is less than other countries, but there may be special reasons for this, as when English teachers teach maths and science they often use bought in worksheets.

So, what was the point of this comparison, if we can’t join the dots and say that using textbooks as ‘basis for instruction’ is more likely to lead to better outcomes? To understand the use that Oates puts the comparison to we’ll have to pick over this sentence:

With levels of use lower than other jurisdictions, and very low levels assigned to ‘as a basis for instruction’ [sic] what is interesting in England is the existence of an underlying ‘anti-textbook ethos’, and its location in teacher training and educational research communities,

It’s not clear whether Oates is suggesting that the TIMSS survey is evidence of an anti-textbook ethos. The two things are put side by side and, again, it seems it is up to us to join the dots.

However the ‘existence’ of this ethos is assumed. As we have already seen, we don’t know if in the TIMMS survey there were low levels assigned to ‘as a basis for instruction’ because that’s not what was asked. As we will see, it’s location in teacher education is only perilously supported by anecdote.

Oates cites Marland as evidence. Marland’s analysis is described as ‘comprehensive and penetrating’ (p.8). Oates seems to be suggesting that a survey in which 78% of science teachers responded that they used textbooks in lessons supports Marland’s findings of an anti-textbook ethos. If we read the extract from Marland this ‘comprehensive’ analysis suggests that that there is little evidence in the literature for the proposed anti-textbook ethos other than anecdote. My own anecdotal experience as a teacher trainer at university and as a mentor in school would suggest that trainee teachers might be discouraged from using textbooks as ‘the basis for instruction’, and positively pushed into using it as ‘a basis for instruction’.

Context and Historical Analysis

The paper is very critical of those wishing to explain Finland’s success in simple terms of contemporary school and professional autonomy without exploring the historical context of the country’s success, or of its current relative autonomy. This would imply that any analysis of the seemingly low use of England textbooks in maths and sciences which did not consider the historical context of this pattern of use would also be flawed. This paper makes no attempt to consider such a history, despite finding space for a (also flawed) consideration of Finland’s recent educational history. The TIMSS survey in 2011 came after the end of the New Labour years of national strategies, specified schemes of work and centralised resources distributed in packs of booklets or downloaded from the DCSF website. In these circumstances it is not surprising that teachers in science and maths report that they are less likely to use textbooks as ‘basis for their instruction’.

Furthermore we need to consider Oates’ own definition of ‘textbook’ in his paper, which he gives us as:

rigorously-designed paper-based materials which can include textbooks for teachers’ use, textbooks for pupils and pupil workbooks.

This is a wide definition, and elements of it could easily fall within the ‘worksheets or workbooks’ as well as within the ‘textbooks’ that English teachers were asked about in the TIMSS questionnaire. So, workbooks mean textbooks to Oates, but the TIMSS treats them separately. We know that in many English primary and secondary schools science materials are often purchased in the form of photocopy master worksheets. These too could be defined as ‘rigorously designed paper-based materials’. So, another reason that so many teachers report low use of textbooks could be because their school doesn’t buy in its ‘rigorous paper based materials’ in the form of books, but in the form of worksheets.

Fundamental flaws in Oates’ claims

However, there are real problems with Oates’ claims about (1) the quality of English textbooks, and about (2) the potential for centralised control over the content of textbooks to combat these perceived shortcomings. It may be that England’s textbooks are of an inferior quality, though there is not enough evidence in this paper to support such a claim and such evidence that there is suffers from important methodological problems.

In our discussion at http://www.cambridgeassessment.org.uk/news/new-research-shows-why-textbooks-count-tim-oates/ Oates presents us with a ‘method statement’. Interestingly this statement is not in the paper itself – the ‘New research’ that was published on the site.

In the paper itself we are told that 200 textbooks were collected and used ‘as part of the transnational curriculum content mappings’, which then allowed for ‘further analysis of the qualities of the textbooks themselves’. We are also told that the textbooks were ‘documented for the different kinds of information elements which [they] contained and the manner in which [they] presented these elements’. The assessment of these textbooks was based on ‘the coherence of the text based on either correspondence to a stated model (eg spiral curriculum) or to an obvious form adopted in the text.’.

Sampling

The treatment of the issue of sampling is a key problem in this paper. In the ‘method statement’ on the website above we are told that:

“Just over 200 books were examined, obtained from Singapore, Hong Kong, Finland, Alberta and Massachusetts. […] Books in Science, Maths, History, Geography, and English/Native Language and Literature were collected and scrutinised, covering both Primary and Secondary. Books on early reading were included and were of course focussed on early Primary.[…]”

In the paper we are asked to compare 3 textbooks – one from each jurisdiction. The problem with this is obvious. Each textbook is made to stand for the textbooks of its jurisdiction. If we were drawing conclusions only about those particular textbooks then these conclusions might have weight, but instead the each one is held out as representing all the books in each jurisdiction. The choice of a KS4 book in England is an interesting one and brings us back to the definition of ‘textbook’. KS4 textbooks vary widely in audience and in purpose which leads them to vary widely in content and tone. Is this a class text? Is it a revision textbook? Is it one that has been approved by the examination board (and which therefore has to contain information and examples which relate directly to the examination)?

Method

The method for the study is also not outlined in detail, and the method itself seems to have been forgotten when data was collected and analysed. There are two steps in the method: – ‘documentation’ of the different kinds of information elements, followed by ‘assessment’ of the coherence with a ‘stated model’ or ‘obvious form adopted in the text’.

Turning first to the idea of ‘documentation’. This is an interesting choice of word. I would take this to imply the production of a non-judgemental record of the elements of the books, so that this would allow further analysis and supported judgement. What we are not told is the framework by which the documentation will take place. Beyond the aim of documenting ‘key elements’ We don’t know if the civil servants who did this analysis were looking for visual elements, explanatory elements, information elements.

When we look at the Case study texts ‘extracted from the textbook analysis’ we can assess how closely the method stuck to the principle of documentation, and an idea of the un-written analysis framework starts to emerge. Documentation in fact seems to involve a fair amount of evaluation, which would not be problematic in itself, if we were told the framework around which the evaluation of the different types of ‘element’ were taking place.

So, the ‘Elements’ in the maths textbook from Hong Kong starts with what seems like documentation. We are given a list of elements as follows:

• Statement of Pre-requisites
• Review activity to determine whether pupil is ready for the chapter
• Different forms of the equations of circles
• Features of circles from the equations
• Equations of circles from the different given conditions
• Intersection of a straight line and a circle Inclusion of a series of problems
• Check through assessment: 6 problems, 1 practice exam Q, 1 lively maths problem

What we have is a list of things that are in the textbook. We could quibble about the ‘lively maths problem’ and ask about what makes it ‘lively’. A similar list is given for the textbook from Singapore. By the time we get to the ‘documentation’ of the English textbooks we are told that the IGCSE textbook has:

• Clear statements of mathematical ideas
• Clear statements of operations
• Some sample activities

The GCSE English textbook ‘documentation’ is cursory at the least, and seems to have been done with a pre-conceived judgement in mind:

• Extremely diverse content within diverse structure – complex
• Divided into Higher Tier and Lower Tier elements to match examination 299 pages long
• Sample full GCSE exam paper very early in the text: p11

Are the ‘sample’ activities set out in the IGCSE textbook divorced from the type of questions that will be asked in the final exam? I don’t know, and we are not told. What makes the IGCE textbook statements of ideas and operations ‘clear’? I don’t know, and we are not told. How do they compare with similar statements in the GCSE textbooks. We don’t know, because we are not told. What does it mean that the English GCSE textbook is ‘diverse’ in content and structure? How is ‘complexity’ judged? We go from a list of elements with some implicit approval of the Asian books, to clear but un-supported critique of English GCSE textbooks which abandons any attempt to ‘document’.

This tendency to eschew documentation and skip straight to judgement is best illustrated by the recording of the ‘Key Features’. These consist of value judgements, rather than documentation or even of an analysis of ‘coherence’. So, the Hong Kong book contains ‘important’ evaluation, ‘good’ elaboration, and the Singaporean book ‘extremely clear’ statements and ‘good’ elaboration, whereas the English textbook is ‘rather incoherent’.

Furthermore the second stage, or the analysis of coherence with a ‘stated model’ or ‘obvious form adopted in the text’ is not further explained, though we are assured of the Singaporean and Hong Kong texts that coherence was ‘impressive’. In the paragraph which seems to draw conclusions instead we are told that texts from Hong Kong and Singapore had ‘extremely clear presentation, explanation and reinforcement of key concepts and ideas’. We are not told which stated model they cohere with, nor any obvious forms adopted in the text around which they cohere with. We are not told what the difference or similarities are between a ‘model’ or ‘obvious form’, whether this is a model of presentation, learning, cognition or whether we are looking at obvious forms of ‘style’ or otherwise. This lack of definition makes it impossible for anyone attempting to look at the same sample of 200 books (should this list be made available) in order to repeat the analysis.

Coherence

All of this brings us back to the central issue of coherence. Even in those jurisdictions which Oates chooses to focus on, where there is (or was) state control over the content of textbooks, there is evidence that the force of ‘coherence’ is (or was) provided by a wider agency than a central curriculum authority and that a good deal of it came ‘upwards’ from the classroom and from teachers, as well as from society as a whole. In Hong Kong we are told that textbooks tend over time to become similar as they follow the market leader, and thus are affected by teachers’ views as well as by the self- censorship that publishers undertake to please the actual censor. Not only that, but attempted innovations fail because they are rejected by the teachers – interestingly it is the teachers who are providing coherence, as well as the government approval system. In Singapore we also read about the ‘panel of professionals’ drawn from ‘curriculum specialists, teachers and academics from universities’ who approve textbooks. In my last post on this topic I discussed the societal shifts and coalescence which resulted in the education reforms of Finland in the 1960s and which provided wider coherence to the implementation of these reforms.

As we also saw in the last post in this series, Schmidt’s paper on coherence suggests that teachers will place authority and trust in curricula materials which cohere with the curriculum as it is enacted. If we accept Oates’ argument about narrowness and the instrumentalist approach of English textbook publishers (though I don’t think that we have to on the basis of the poor evidence that Oates supplies), and his evidence that teachers’ views and practices have caused the failure of state sponsored changes to textbooks in other places then we must look at the curricula system as a whole in order to find out why textbooks are like this in the English system, not apply state level approval of texts as the primary guarantee of ‘coherence’.

This all boils down to my basic problem with this paper – that it does not reflect the critical-realist methodology that Oates has claimed for it. Of course textbooks count – but the important questions are the extent and the mechanisms by which they count in particular circumstances. My worry about the analysis in this influential policy paper is that these circumstances are only referred to in just enough detail to provide legitimacy for the threat of state intervention.