Measuring and debating effectiveness

Reaching the target

As I’ve noted, Cooney and others in CTW were concerned that the ‘particular’ target audience – the ‘bullseye of the target’, as Cooney put it[i] – would not be reached. This was partly because such viewers were seen to be less interested in educational television than their middle-class counterparts. However, there was also a particular technical problem: African-American viewers were more likely to have old-model TV sets, which made it difficult to tune in to the UHF channels on which such programmes were broadcast. As we’ll see, an extensive range of outreach activities was devised in order to get the word out to such viewers, and then to enable them to make best educational use of the show.

Initial audience data, collected in 1969 by the Nielsen Corporation, suggested that Sesame Street was being watched somewhat less in low-income homes; and concern was expressed that, if this was the case, then it might actually widen rather than narrow the educational gap.[ii] CTW’s Head of Research Edward Palmer acknowledged that there was ‘increasingly heavy viewing as we go up the economic scale’, although he also expressed some justified scepticism about the reliability of such ratings research.[iii]

This was a sensitive issue, and CTW went on to commission the Yankelovich Organisation to conduct some more detailed research on ‘ghetto audiences’ in five areas the following year.[iv] The reports showed fairly conclusively that the programme was indeed reaching its target, among Puerto Rican as well as African-American households; and there was a good deal of positive (although not detailed) feedback on the educational aspects specifically. Follow-up studies in 1971 on the series’ second year, and then again in 1973 and 1978, showed penetration above 90%, and regular viewing between two thirds and three quarters of the available audience, with the exception of areas where the programme was only available on one UHF channel (such as Washington DC). Interestingly, the show was also being watched by significant numbers of older children (aged 6-11). By 1973, Yankelovich was able to conclude that Sesame Street had become ‘virtually an institution with ghetto children’, with rising figures year by year.

A related issue here was to do with the cost of support materials that were intended to supplement the programmes. CTW published and distributed its own free newsletters; but commercial publishers, seeing the programme’s growing popularity, also wanted a piece of the action. With an eye to its own longer-term financial viability, CTW struck deals with publishers like Time-Life, although it was concerned that the cost of their books and learning ‘kits’ should not prove prohibitive for the key target audience. One early set of large-format books cost as much as $3.95 each, well over $20 in today’s terms; while a Time-Life ‘kit’ (including filmstrips, audio cassettes, games, activity books and ‘assessment cards’) cost $19.95, more than $100 today. Although the kits sold well[v] and the company agreed to distribute some of this material for free, Cooney acknowledged that the deal was ‘a terrible gaffe – we backed into it, and it’s the wrong image. $19.95 is not even middle-class.’[vi] She told Variety that, while they wouldn’t discourage middle-class parents from watching, ‘we’ve made Sesame Street for the poor people and the ghetto communities’, and any ancillary publications would need to be at a price ‘people in the ghetto can afford’.[vii] Later publications were indeed significantly less expensive: a series proposed with the publisher Scholastic included books retailing at less than a dollar.[viii]

Assessing effects

Sesame Street may have been hitting its ‘bullseye’, but was it effective in teaching them – and particularly in narrowing the gap between disadvantaged and middle-class children? Despite later disclaimers, there’s no doubt that this was a key question for CTW. In addition to funding the Yankelovich studies specifically in ‘ghetto’ neighbourhoods, it also tasked ETS to look at this issue in its key large-scale evaluations of the first and second seasons. Much of the fieldwork focused specifically on low-income areas, which were often harder for researchers to access than middle-class suburbs: 731 of the 943 children involved in the first year study were classified as ‘disadvantaged’.

As I’ve noted, the educational objectives that were actually assessed by ETS were much narrower than those identified in the official curriculum – which were in turn much narrower than those that had been discussed at the early seminars, or (more loosely) identified in Cooney’s initial reports. Loretta Long’s PhD, written in 1973, reviews the research in some detail; but when it comes to evidence, for example about the show’s contribution to improving racial tolerance, or raising the self-esteem of disadvantaged children, she is bound to resort to anecdotes and viewers’ letters. Even when we look across the enormous body of research that has been conducted on the series over the past fifty years, there is much more about basic literacy and mathematical skills, and about factual recall of aspects of science and civics, than there is about the broader social or ‘non-cognitive’ aims with which the series began.[ix]

In terms of these basic skills, the first ETS studies showed that children were indeed capable of learning from the show. Children who watched the most, appeared to learn the most; the things they learnt best were the things that were shown most frequently; and they learnt more with adult intervention or support, although they were capable of learning without it. However, when it came to the achievement gap, the findings were rather more equivocal. Interestingly, ETS agreed with CTW that they would not report comparisons between children along ‘racial’ lines (blacks vs. whites vs. Hispanics), not least because they feared that this would be too sensitive to discuss in public settings.[x] However, ‘race’ was a variable in the study (for example in composing the sample), and there were comparisons between advantaged and disadvantaged children (abbreviated as ‘M.C’ and ‘W.C.’). In these terms, it appeared that disadvantaged children who were frequent viewers learned more than advantaged children who watched less frequently – although of course what was being assessed here was not learning or cognitive development in general, but learning of the particular selection of Sesame Street’s educational objectives that were chosen for analysis.

As far as CTW was concerned, the success of Sesame Street in narrowing the educational gap was pretty clear. Cooney’s first year report to the Carnegie Corporation in 1970 suggested that, on the basis of the ETS studies, the programme was being successful in meeting ‘one of the primary goals, namely, putting the disadvantaged child on an equal footing with his more fortunate and better motivated middle-class peer when they arrive together at the doorway of formal education’.[xi] CTW’s glossy publicity brochure from 1971, cited above, repeated the comparison between disadvantaged regular viewers and advantaged infrequent viewers, and suggested that the research had ‘led to the conclusion that television can reduce the distinct educational gap that usually separates advantaged and disadvantaged children’. These broad conclusions were duly reported in the press, and were widely taken as a ‘good news story’.

Challenging effects

A little further down the line, however, they came to be challenged, both by independent researchers, and by a major re-evaluation of the ETS studies. My interest here is not in the technicalities of the research itself, or even in the actual findings, so much as what the debates reveal about the fundamental aims of Sesame Street.

One early opponent of Sesame Street was Herbert Sprigle, director of the ‘Learning to Learn School’ in Jacksonville, Florida. Sprigle conducted a series of studies in 1970 and 1971 comparing the use of Sesame Street with face-to-face methods in his own schools. He argued that the programme failed to prepare ‘poverty children’ for the work they were to do in first grade, and that it did not narrow (and in fact increased) the achievement gap between them and their middle-class counterparts. Sprigle claimed that non-viewers were getting a more solid foundation that would enable them to advance faster in school; and that there were ‘no permanent educational benefits’ for those who watched the programme more frequently. There was a danger, he argued, that ‘politicians will think all you have to do is fund Sesame Street and then forget about education for poverty kids’.[xii]

These findings clearly contradicted those of the ETS studies, and CTW’s Head of Research Edward Palmer published an extensive rebuttal, which took a highly confrontational tone. Palmer argued that Sprigle’s methods were ‘so thoroughly flawed as to be meaningless’: he was looking at the wrong age group, using inappropriate measures, failing to match the groups in his study adequately, and introducing all sorts of additional biases.[xiii] Furthermore, in failing to share all his data with CTW, he was guilty of a lack of ‘professionalism’. Without going into all the technicalities, there’s undoubtedly some truth in Palmer’s response, but its vehemence is nevertheless quite extraordinary: however idiosyncratic his approach may have been, Sprigle had evidently touched a nerve. Perhaps the most significant aspect of Palmer’s response in this respect, however, is his denial that Sesame Street was ever attempting to narrow the educational achievement gap in the first place – an issue to which we’ll return shortly.

In fact, these concerns appeared to be more widely shared. In 1973, a report by Robert Yin, funded by the Markle Foundation and published by the Rand Corporation, pointed to some of the difficulties in assessing the effectiveness of CTW’s work (which by this point also included a new series for older children, The Electric Company).[xiv] Yin asked some fundamental questions about what and how such evaluations should measure: for example, should they be confined to declared cognitive objectives, or should they consider ‘non-cognitive’ ones as well? Like Palmer, Yin effectively demolished the Sprigle studies; but he also questioned the validity of the ETS work, arguing that it used unreliable measures and limited samples.

However, it was a later re-evaluation of the ETS studies by Thomas Cook and his colleagues that provoked the most striking debate. Cook’s study was funded by the Russell Sage Foundation, and published in 1975 (although it was circulated to CTW and ETS well before then, and publication was delayed by the ensuing disputes between them). As with Robert Yin’s Markle/Rand study, I haven’t been able to identify the reasons why the grant was made, but it may reflect the rivalry between the various foundations operating in the field. Cook and his team conducted an extensive and very detailed re-analysis of the ETS data, and many of their criticisms of these studies are highly technical: there is much discussion of the relevance and validity of different outcome measures, the composition and weighting of the samples, and so forth. In a sense, Cook and his team were playing ETS and CTW at their own game, and (unlike Sprigle, for example), this made him harder to dismiss. What is most relevant here, however, is the way the re-analysis, and the debate that followed, exposed the incoherence or inconsistency of CTW’s original aims.

Cook focuses particularly on one of the key variables in the ETS study, the issue of ‘encouragement to view’. In some instances, parents were encouraged to tune in, and also urged to encourage their children to watch; and comparisons were then made with households where there was no such encouragement. Cook points out that, without such encouragement, the learning gains from viewing were confined to basic letter and number recognition and ‘relation skills’; but even in those cases, the differences were either very small or statistically insignificant. Encouraging some parents and children to view obviously created an artificial situation, which could not necessarily be replicated under normal circumstances. Indeed, it seems likely that, in reality, middle-class children would have more parental support than disadvantaged children, at least in relation to preparation for school. On the basis of ratings data (which were admittedly less reliable than ETS), Cook argued that advantaged children were more likely to view than disadvantaged children, and were thus likely to gain more, at least where gains were to be had. On the other hand, disadvantaged parents in the ETS study who were encouraged to view spent less time reading with their children. As a result, Cook concludes, ‘Sesame Street is probably increasing achievement gaps in those domains where it effectively teaches’.[xv]

Of course, these points were disputed by CTW; but for Cook, the key issue was a broader one – it was to do with ‘the difficulty of pursuing compensatory goals by means of a universalist strategy’[xvi]. Cook argued that compensatory approaches may stigmatise their recipients as in some way deficient (via a deficit model); and that universalist approaches are unlikely to be equally used by all segments in society. In other words, CTW could not hope to teach all children via a universal medium like television, while simultaneously aiming to narrow the gap between rich and poor. If it was aiming to close the gap, the programme would have to be more available to disadvantaged children, or they would have to be more actively supported in using it, so that they learnt more; or alternatively, it would have to be less available and less supported for advantaged children. These two aims, Cook argued, were incompatible: Sesame Street could not have its cake and eat it.

Going back to Cooney’s original funding proposal from 1968, Cook drew attention to the contradiction between the ‘general aim’ and the ‘particular aim’ of Sesame Street. Cooney’s claim that the programme would stimulate ‘the growth of preschoolers, particularly disadvantaged preschoolers’ was at least ambiguous: it did not clarify whether the programme would specifically target this group, or whether they would learn from it, or whether they would learn more than their economically advantaged peers. To this extent, ‘particularly’ would seem to be a conveniently evasive term.[xvii]

As Cook and his colleagues argued, this apparent inconsistency raised much broader questions about the aims and methods of social action initiatives that are intended to reduce inequality. Advocates of such initiatives needed to look harder at who actually made use of such programmes, and who got the most out of them. Was it more effective to target resources particularly at the disadvantaged, or to attempt to raise the overall level of achievement? How should efficiency in delivering a service (such as education) be balanced against the requirement for social justice?

Backtracking

Cook’s re-analysis was fiercely challenged by both CTW and ETS.[xviii] Collectively, they accused Cook of incompetence, of lacking relevant credentials, and of being ‘petulant’, unfair and misinformed. Again, there is a fair amount of technical discussion, but it is their response to his broader argument that is more striking. Both CTW and ETS effectively sought to disclaim the idea that Sesame Street had ever sought to narrow the gap in the first place. After some intense debate, Samuel Ball and Gerry Anne Bogatz of ETS secured a right of reply in the form of an appendix to the book; and after disputing many of the technical points, they made the key claim that ‘closing the gap was not a goal of Sesame Street’ – not least because, in their words, this was something ‘no other universally available education [provision]… has been able to develop’.[xix]

Gerald Lesser’s account of the early years of Sesame Street appeared before Cook’s study, but he had certainly seen the re-analysis (and done the best he could to prevent its publication). He too offers the ‘both/and’ version of the story that is apparent in Cooney’s various reports and public statements. Yet here again, there is also a direct denial: ‘Since we hoped that all children would watch and learn, we did not aspire to reduce the differences between poor and middle-class children. But we did hope that our series would help to prepare poor children to do well in the schools as they are now organized and operated.’ Lesser appears to accept the logic of Cook’s argument that a ‘universalist’ strategy like a national TV show could not simultaneously reduce inequalities; but nevertheless he says he ‘hoped’ that the gap would not be widened. He agrees that CTW wanted to know whether children would learn regardless of their social background, but he also suggests that the comparisons in the ETS study between disadvantaged and middle-class children were ‘not of interest’ to CTW.[xx]

These are extraordinary claims. As I have shown at considerable length, closing (or attempting to close) the educational gap between advantaged and disadvantaged children was in fact a major aim identified in countless published and internal documents. It was a key dimension of the wider public context, and was clearly part of the reason why Sesame Street was funded in the first place. A substantial proportion of its early budget was spent precisely on attempting to reach out specifically to disadvantaged children (as I’ll describe in the following section). The comparison between advantaged and disadvantaged children was also implemented in the research itself, and reported in the published findings. To argue that this was not an aim in the first place was quite implausible.

Much later, Joan Cooney seemed to acknowledge that the aim had actually changed. Despite asserting that Cook’s analysis was a ‘misinterpretation’, she appeared to have accepted at least some of his arguments. In an interview conducted in 1998, she presented the findings of the ETS research in a rather different way: ‘it turned out that middle-class [children] moved faster, so there was always that gap… We brought both poor and middle-class children up higher than where they were, but we couldn’t close the gap because everyone was running faster.’ As a result, she said, ‘we decided the aim was not to close the gap, but to get all children up above the literacy line, so they could go to school and hit the ground running, and learn to read.’[xxi]

Reaching the target

Assessing effects

Challenging effects

Backtracking

Share this:

Like this: