1 2 3 We are still interested in assessing a depth and breadth of - - PDF document

1

2

3

We are still interested in assessing a depth and breadth of the NGSS and other 3-D / Framework-based standards, but we (as a field) need to define what that will mean and what that will look like in this context – because it will be different from what we’ve typically expected. 4

5

How will we know if new assessments – ones that purport to assess Framework- based standards – do what they purport to do (for purposes of this conversation – specifically in terms of the complexity question). 6

Again, we certainly need to define what is “acceptable” and what is appropriate. But the goals of the NGSS and other 3-D standards are truly compromised if we neglect to hold assessments accountable for assessing students at an appropriate level of complexity – as corresponds to the expectations of the standards. 7

DOK is successfully used to interpret standards and add clarity to conversations about complexity in our education system. Based on work with hundreds and hundreds of reviewer panels comprised of thousands of educators, we have evidence (in the form if interrater reliability and other statistical measurements) that the DOK language system can be used effectively and reliably to evaluate the complexity of educational materials. 8

9 9 DOK is a tool that educators use to inform the process of interpretation and implementation of standards – to ensure that the complexity of engagement expected by the standards follows through in the other components of the system. It is used throughout the US as well as beyond our borders.

Our program recently hosted a 2-day meeting to gather stakeholders from all different parts of the system (and who are taking all different approaches to working with NGSS complexity) to try to calibrate our understanding and interpretation of complexity within the NGSS and other Framework-based 3-D standards. 10

It’s important to acknowledge that there’s not necessarily a shared understanding within the education world in general -- nor within the science education world in particular – about what complexity is all about. W/r/t NGSS, we’ve heard a broad range of perspectives on and interpretations of complexity – this was evident from discussions in our NGSS / DOK stakeholder summit. This is a key part of what is so powerful about DOK (or any common language system) – it allows all of the stakeholders in different parts of the system to communicate. 11

From the DOK perspective, content complexity is multidimensional; multiple factors contribute to the complexity of engagement required (by a learning expectation or task, etc). The ideas of “knowledge in use” and “sense-making” – often cited in 3-D science standard-related materials – relate strongly to some of these factors, e.g. suggesting an expectation that there is some degree of processing of concepts and skills in authentic contexts. The “context” factor is directly related to the selection of a phenomenon: different contexts afford different opportunities for engagement. –observing organelles in the context of a cause-effect relationship or some other fairly routine interaction may

• ffer different opportunities for engagement than a novel problem that requires

students to grapple with a variety of information and data to figure out how to even start to approaching making sense of the problem. W/r/t novelty – a main consideration for 3-D C&I and assessment is the phenomena

• used. if we are using a phenomenon in the classroom that is already hugely familiar

to students then the opportunities for engagement that we think we are offering might not actually be providing the opportunities we think we are providing – same

• n assessments – if we are providing a very familiar vs novel context, it affects the

complexity of the subsequent questions b/c it might be that instead of working through the question during the assessment, students are actually recalling the answer. ========= 12

We find that difficulty and complexity are often conflated. The are certainly related, but are different. It is important to differentiate between these as we consider NGSS (and other 3-D) assessments. 13

Clarification: This “DOK wheel” (AKA “the wheel of misfortune”) is NOT related to Norman Webb and/or the WebbAlign program. We strongly disagree with the message of this wheel – we emphasize that it is NOT possible to rely on a verb to interpret the complexity of a task. We noted that we sorted tasks that all used the same verb (e.g. “draw”) at all different levels of

• complexity. “Defining” a word is different from “defining” policy. Etc.

This wheel can be a good thesaurus but is not related to DOK. 14

15

When working with educators and content developers, we find that expectations for [cognitive] complexity are not always clearly interpreted, even when using the language of Appendix F. For example, the two SEP statements on this slide are ones that we see applied at [what we would call] DOK 1, 2, 3, (and even 4). This is an experience I’ve heard echoed multiple times in the last day and a half of this conference. 16

This is something that was recognized by the writers of the framework – in this quote here, they are talking about the range of types of models – note that they are talking about the complexity of the MODEL and I’m talking about the complexity of ENGAGEMENT with a model – but a more complex model likely provides more

• pportunities for complex engagement. So this is exactly how we use DOK – to add

clarity to the work we’re doing, to have a tool we can use to reflect on whether a task is doing what I’m intending it to do. 17

We can’t rely on Appendix F to tell us about complexity because the language can be interpreted in very different ways. And similarly we can’t rely on Appendix F when we are evaluating the complexity of tasks because the application of any practice depends on the context and specifics of the task. 18

In practice, we have found that practitioners and even folks who were involved with Framework-influenced standards are able to differentiate between and among the expectations (for assessment) of the different PEs/Standards. But if we accept this assertion (that everything is/should be maximum complexity), then we need to reconcile it with another common assertion that we want to assess the NGSS (and other Framework-based standards) at a “range of complexity.” 19

Per state directive, the standards used for a recent alignment study were in this

• format. Georgia educator expertise contributed to the interpretation of the

standards, but “the standards” were presented as the statements shown here. A panel of 4 internal educators (Georgia-based) and 4 external educators (from various states) went through a consensus discussion process to determine the DOK of each of 24 high school biology standards. Panelists’ consensus coding for the subset

• f standards shown on this slide is included above.

For 5.b, for example, the panelists thought that this was an expectation for students to really demonstrate a conceptual understanding of the cycling of matter and flow

• f energy – to make sense of the relationships and interactions in the system. They

expected students to use typical models, typical representations of these processes. They did NOT think that it was an expectation to engage in abstract and non-routine thinking to develop a novel representation of these concepts. The panel decided that 16 of the 24 standards used in the alignment analysis were DOK 2, emphasizing engagement with questions and tasks that build conceptual understanding of science, including developing one’s own understanding of relationships and interactions between and among ideas and concepts, etc. Two standards were considered to be DOK 4. Six standards were considered to be DOK 3, requiring hypothetical and strategic thinking about non-routine problems. 20

Although those closest to the standards agree that the PE statements or any statement in isolation should not be used or considered as consistent with the vision

• f the framework, it is anyway happening in some states -- and as we know there are

all sorts of reasons for why we do things – there’s practical and political reasons, for

• example. NOTE that per WA correction post-session: although the OSPI RFP stated

the above, the foundation boxes and other aspects of the standards were supposed to be considered for alignment judgment and NOT “the specific language in the PE.” 21

Speaking of political reasons…we heard in our recent meetings that this statement that PEs might be adopted in isolation was something that “had to happen” but wasn’t the intent – but – in any case -- there’s lots of messaging that the statements

• f the PEs are intended to guide assessment (see underlined statement here, e.g.)

and next slide. 22

So -- if the NGSS is intended to support assessment development – as is explicitly stated in multiple places within the standards, and the PEs are explicitly supposed to “be clear about which practice students are responsible for…” then 1) that’s really different from what many states are doing but 2) it means that if states do this, it gets back to the idea that we need to interpret the complexity of a PE (if we were to do that at all) and of tasks by evaluating the practice within the context of that PE/ task – e.g. think back to the model – modeling isn’t just modeling – so what is the complexity of the engagement in that particular expectation/task. 23

One thing that I do agree with is that we need to think a little differently about NGSS in the classroom and NGSS on a summative assessment. So in the classroom I may have multiple PEs that I am addressing over the course of 3 or 4 weeks in a phenomenon-based learning context – and NO, I’m not going to slap a single DOK “level” on that unit – on that entire set of lessons. That doesn’t make any sense. But it will be very helpful for me to use DOK to reflect on the learning objectives, tasks, prompts, questions, that I use throughout those 3-4 weeks – THOSE are the units of analysis for which I can use DOK to analyze complexity – and determine if I’m doing what I’m intending to do in the classroom. On a summative assessment, we have specific claims, a specific assessment construct, specific assessment targets. In this case, it might not be the PEs themselves but we have some set of something that we should be able to pretty directly identify. If we can identify what we are assessing, we should be able to evaluate if the complexity of the tasks in the assessment are, indeed, as intended. 24

Not assessing DOK 1 is a choice a decision, but it doesn’t make DOK 1 disappear. If we don’t want any DOK 1 items, why do we have so many DOK 1 items? Understanding and application of DOK can ensure avoidance of DOK 1 items. We’re seeing a WHOLE LOT of assessment items that are put out as NGSS exemplar items that we would call DOK 1. 25

To be clear – there’s actually no inherent value judgment to DOK. DOK is just a descriptive tool. Then we decide “is this what I want?” If we are trying not to have DOK 1 items, but we do have DOK 1 items, it suggests we need to think about what makes a DOK 1 item (?) vs a DOK 2 (and DOK 3) item. In this example, the information presented in the phenomenon does not provide enough information for the student to answer the question. Rather, the student must recall information about the topic to identify the correct statement. Through the lens

• f DOK, it’s immediately evident that this item does not afford an opportunity for

sense-making about the phenomenon shown. Notice that not only does the phenomenon not provide the information necessary to answer the question, the image and stimulus does not need to be used at all to answer the question. The distractors are statements that are False, False, False, and True – no matter the context. 26

Again, the information presented in the phenomenon does not provide enough information for the student to engage in sense-making about the phenomenon shown, or to “figure out” the answer to the question. Rather, the student must recall information about the topic to identify the correct statement. And again, notice that not only does the phenomenon not provide the information necessary to answer the question, the image and stimulus does not need to be used at all to answer the question. Only one distractor is a true statement; the others are false. 27

Now we are making a model. We’d still call this a DOK 1, because there is a very straightforward path and a visual representation (rather than a written representation) of a “fact.” This is an example of “developing a model” that is consistent with (at least some interpretations of) Appendix F specifications but, in the context above, is DOK 1. Is that as intended? 28

Here’s the SAIC Prototype, which was referenced numerous times over the course of NSCA 2018 presentations. This publication contains mostly DOK 1 items, perhaps a few DOK 2 – is this as intended? Here, there is a fairly elaborate presentation of a phenomenon 29

30

In this example, a fairly elaborate presentation of a phenomenon is followed by DOK 1 questions that involve definitional labeling and identification of inputs/outputs – which must be recalled; cannot be “figured out” based on the interaction with the

• phenomenon. This comes from an often referenced (as a model; exemplar)
• prototype. Is this what we are aiming for?

31

Here students must recognize the process that is represented. This type of question, requiring a definitional recognition of a particular process, is DOK 1. [ASIDE: Note that this representation of phytoplankton as conducting photosynthesis (only) “vs” zooplankton conducting cellular respiration helps to perpetuate a common misrepresentation of metabolic processes and matter/energy cycle/flow in science materials (-- that photosynthesis and cellular respiration are parallel processes and that plants conduct PS while animals conduct CR).] 32

[with answers shown] 33

This question does not depend on any interpretation of the phenomenon; DOK 1. 34

One problem is that we can’t determine the complexity of something that we can’t

• identify. I think we need to keep working on that – what exactly is the assessment

construct? What are we assessing, exactly? For DOK in particular, a critical issue is that we have outdated science definitions. So if you are steeped in DOK and really understand the conceptualization of complexity that underlies the DOK language system, then it’s pretty easy to use it to interpret framework-based standards and assessment tasks. But if you are using the definitions based on the words in the current definitions, there’s a lot of potential for confusion. 35

36

We’re taking what we’ve learned from conversations with folks in the field, and our experiences with educators and with the standards, and working on revising the language of the definitions. Revised definitions will be circulated for input/feedback before being finalized. The revised definitions would NOT be NGSS specific but they WOULD use the type of language that we are using to talk about NGSS as well as other framework based and 3-D and new science standards. (They might call out specific NGSS considerations – like, if you’re using NGSS, don’t’ forget to look at Appendix F -- which was something that came out of our recent meeting – the importance of calling out that folks need to attend to Appendix F.) 37

There are some ideas that NGSS assessments might evade the requirement that assessment tasks “match” any type of expectation. This is a risky proposition that could compromise the potential for NGSS (and other framework-based standards) to shift the system, as intended. As was often acknowledged over the course of NCSA 2018 (and is often acknowledged in other venues), assessments send a message to teachers and others in the system – and influence what happens in the classroom. If we are sanctioning assessments that are comprised primarily of DOK 1 items, we will not achieve the most basic of goals of the NGSS to shift away from rote learning and the requirement for students to recall science “facts.” The DOK lens provides an evaluative tool that can help hold assessments to an expectation that items require students to engage in sensemaking, require knowledge-in-use vs in isolation, etc. 38

Questions/comments welcomed at this email address (sara.christopherson@wceps.org). Please contact me if you’d like to review the DOK draft revised definitions for science. 39

40

41

42

43

Bottom line: a great deal of information is already embedded within the standards documents, so a “complexity of the PE” is both unnecessary and inappropriate. (note that this is true for all 3 dimensions). What IS true, however, is that there is still a range of complexity that can be associated with a given standard or bundle of standards, depending on how that standard is operationalized within a task—and our complexity framework focuses on complexity of tasks, not standards. PEs represent the floor, not the ceiling. 44

45

These can be driven by different degrees/foregrounding backgrounding of the dimensions, and this is a place complexity can help us be discerning. 46

Go through one-2 examples Boils down to the degree and nature of application—sense-making—and how the dimensions engaged support this. 47

48

49

50

51

52

53

54

55

56

57

58

59

60

task; while most items are intended to be 2 or 3D, the complexity contributions can tell a different story. —this model provides a comprehensive approach that can be tailored to different needs. 61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

78

79

80

81

82

83

84

85

86

87

88

89

90