Saturday, February 12, 2011

My struggles with the FSAs

Earlier this week I was sitting in the computer lab watching the students complete the grade 4 numeracy portion of the FSA  (Foundation Skills Assessment - British Columbia's standardized tests for grade 4 and 7). There were many thoughts going through my head during this time.  I had looked at the rankings that the Fraser Institute published and saw that my school had jumped significantly in the standings.  I knew that we would be moving up from where we were last year, our focus has been on writing for the past few years.  We have enjoyed some rich professional development, had great discussions, found different ways to emphasize and promote writing in the school and continued to work with different writing programs.  These have all lead to improvements.  I do believe that the students have improved in writing because of these changes, but that does not tell the whole story.

Our results are directly impacted by the number of eligible students who write the different tests and the number of students who have been excused from writing it for a variety of reasons including parental request that their child not write the tests.  If a number of very strong students are excused from writing the test, our results will drop, conversely if the number of students excused from writing are struggling learners our results will rise.  If there is not a flu bug running through the school around that time then the results go up, if there are a few kids who families all go on vacation at that time our results could drop.  There are so many external factors that impact the results of the test beyond the teaching that if they fall in your favour as they mostly did last year, the result is a rise in the rankings, if they do not, as was the case two years ago and this year, they will drop.

To give you some insight as to how it works I will include a brief description.  We received the package a few weeks earlier and I decided to wait to start.  Over the past few years the system has had problems where students will be kicked out of the electronic portion and have to log back in.  For some students this would happen 3-4 times over the course of the time they were working on one portion of the test.  This caused a lot of stress and caused some to panic and become very anxious, worried that they would have to start again.  (I cannot even begin to imagine what would be going through a high school student's mind when writing a provincial test and this happens!).  I hoped that by waiting there would be fewer students using it at the same time, thereby reducing the possibilities of crashes.

During the couple of weeks leading up to us starting I had some conversations with other schools, people on my soccer team and elsewhere discussing the FSAs.  A couple of friends with students at schools outside of our district were telling me that the kids had been practicing the tests for a couple of weeks and were a bundle of nerves because they were told how important these tests are and that they had to do well to represent the school.  One of my colleagues was telling me that she had 3 different parents come to her and ask whether or not their child should be writing the test because they were worried that their kids would not be successful because of their IEPs (individual education plans) and would therefore bring the school results down.  She told the parents that it was their choice, but that they should not be excluding their children for that reason.  Would all principals have said the same thing?  At our school all students write the test unless we are concerned about the impact that this could have on the student, and we do not spend any time practicing.

I had a few parents come to me a little before we were going to start wondering why we had not started our tests when when some of the neighbouring schools had.  I explained my rationale and they felt that it made sense.  One then asked me if we all received the tests at the same time and I let her know that I believed that to be the case.  The next questions caught me off guard as it was not something that had entered my mind.  If you held onto the tests for a couple of weeks, is there anything stopping schools from practicing with the actual tests before starting?  Could they all do a rough draft and then copy the good draft into the booklet?  If they were not all doing the electronic portion at the same time, would it be possible for a class to begin the reading portion and then copy the stories, print them and be able to go over them with the other students to prepare them for the questions?  When I answered that hypothetically speaking this was possible she was stunned.  How was this standardized?  In order for it to be standardized should all the kids in the province be doing the tests at the same time, or at the very least, all of the kids in the same school at the same time?

There are other factors that concern me as to how the data is used.  Two years ago our school received additional staffing partly due to the fact that we had a larger number of students with learning challenges than we normally had.  That year, when the students were writing the FSAs, the flu went through the school and some got sick, 1 missed a lot of time from school so I decided to not have that student finish the FSA. Result, not meeting expectations in all areas.  2 students went on vacation part way through the test, result, 2 students not meeting expectations.  2 students broke down in tears because they were frustrated so we took the tests away, result, 2 students not meeting expectations.  I believe that 4 of those 5 students would have fully met expectations.  I had two students who would not be writing the tests for a couple of reasons and I used their identifications to show the students to log on.  Result- 2 more students not meeting expectations because I had logged them in and answered 1 question but did not click on submit thinking the results would not show, this was a boneheaded moment which I did not repeat.  When you have a cohort group of about 50 kids and 7 are not meeting expectations for reasons other than the results of the test itself is one thing that a school can address and explain to its community, when an outside agency is ignorant of what happens in the schools and publishes results is another.

Four years ago we had the exact same number of identified gifted students as student with learning challenges two years ago.  The school went from double digit number of students who were gifted to double digits with learning difficulties.  The cohort was completely different and something outside the control of the school.  The result was a huge drop in the number of students not meeting expectations.  I go over the results with the parents, give them the numbers as well as some background information.  I am able to explain why we have gone up or down.  Our community is aware of our context, an outside agency examining and working with the data is not.  When a school has their test data compromised because their electronic results have gone into an Internet abyss and suddenly 10 students results are automatically not meeting expectations and those results are published, the public perception of the school can be compromised.

From my vantage point, the FSAs can provide useful information for a staff to look at and plan our school goals.  We can use it as a measuring stick to chart our progress, knowing our context.  This is not shirking responsibility.  I still stand in front of our parents and share the data and listen to concerns that they may have.  My concern is that when a non-informed group that ignores or does not care about the contextual situations that can arise in schools, and publishes results that are challenged statistically it impacts the perception of the school.  Why are schools punished because H1N1 hit their schools hard one year?  Why are schools punished for not practicing for the tests for weeks and choosing to focus on the curriculum and using the FSAs as a snapshot as they were intended to?  Why are schools punished because the cohorts each year can be significantly different from year to year and give a false increase or decrease in the students' perceived success?  The number of students writing is also compromised, based on parent willingness to have their children write the tests.  There are some schools where only 30% of the eligible kids are actually writing the test.

If everybody is not writing the test at the same time, in the same way, under the same conditions, with the same number of kids in a room writing it together, with the same preparation, with ALL students writing the test regardless of abilities, with one large group of independent markers marking all of the tests in the same way and removing the possibilities of possible manipulation of the testing situations, intentionally or accidentally, the test is no longer standardized even if all of the questions are the same.  There are just too many outside factors possible to make this testing equitable.  Let schools use the information for the way that it was intended and not let outside agencies compromise the validity of a non-standardized standardized test.

6 comments:

  1. Good observations Remi. And when you consider those variables beyond the control of the school, it seems miraculous that all those independent schools can achieve 100%. Divine intervention might explain the faith based schools but what accounts for the complete lack of errors in the secular schools?
    In those hundreds of children not a single one clicked the wrong choice in 70 multiple choice questions. Nor did one of them missinterpret the reading selection or have a bad day writing their long essay. How are we to interpret such perfection? Clearly the mortal environment that you so ably describe is not the state in Bountiful and the rest of these elite schools.

    ReplyDelete
  2. Thanks for speaking up about this Remi. We have a number of students pulled from writing our tests and often they are students who would do well on the tests. I have witnessed students in tears sitting in front of a computer screen for an hour trying to choose the best answer. It is never an easy time at our school and I believe that you only need to observe the students writing the tests to begin to question if this is best for our kids. The key is that parents want to know how their child is doing - so I think we need to focus on parent communication and performance standards. Maybe if we communicate with parents better, the need for tests like these would be less necessary?

    ReplyDelete
  3. A very thoughtful post Remi. I think there are two separate issues. First is the tests themselves, and while I am not about to defend standardized tests, I think it is unrealistic to imagine a system of 4 billion dollars not having some third party checks-and-balances. I think we need 5 billion, but that's a different issue altogether! We expect politicians, mechanics, doctors, and others to be accountable...we should be too. Again, not defending the FSA...maybe it's the wrong method of "checking", however, I don't believe the schedule of tests (only gr. 4 and 7) is excessive. The kids I've seen stressing over the FSAs are the same ones who stress about school-based tests. The other end of the spectrum is whether kids (especially Gr. 7s)take it seriously since it doesn't count.

    However, the way THAT Institute uses the results is the real issue. If the results could be used by the province and school districts for what is was intended I don't think we would have near the controversy. We have K-5 elementary schools in our district, however, the grade 7 students count in their results even though they have been in Middle School for almost 2 years. Their math at best is inaccurate; at worst it's fictitious.

    I think year-to-year comparisons are pointless. We need to begin looking at data longitudinally. Some groups are stronger than others, but sometimes the weaker groups improve at a greater rate. I think that would be a more useful way to monitor student achievement, especially if we are going to do it on an individual basis.

    Admittedly I am a bit more moderate on the FSAs than most I know. Not a fan but I do accept the notion of some accountability to the public when it comes to our system's effectiveness. For me it's more of a question of method.

    Great post...made me think...those are the best kind!

    ReplyDelete
  4. Kevin,

    I don't think it's fair to accuse the independent schools of cheating on the FSA's.

    I'm pretty sure that the perfect 10s that you can see in the Fraser Institute report don't represent perfect scores from the students. The Fraser institute has norm-referenced the scores, which mean a 10 really means "you did better than everyone else" and a lower score just means "you did worse".

    I could go into further analysis and try to create a more useful z score from the Fraser institute data, which would mean we could measure in terms of standard deviations from the mean, rather than a relatively useless ranking system.

    Not that the scores mean anything, as Remi has pointed out.

    ReplyDelete
  5. Chris, I agree with 100% about communication. If parents are satisfied with the communication with respect to their children's progress, much of this broohaha goes away. When that piece is not there then the call for accountability becomes loud.

    Tom, you raise really good points. I personally do not have any issues with the original reason for the FSAs to be there. The information can be quite useful. I dislike what it has become because of THAT institute. The purpose and validity have been altered in an unsatisfactory way in my opinion. Thanks for the thoughts.

    ReplyDelete
  6. Thanks David for your thoughts. I think one of the biggest issues around this test is the fact that the results, regardless of whether it is a public or private school, could be questioned because of the lack of control over the product. Just the sheer fact that those questions can be posed is a problem. If some institutions have had their curriculum questioned in one year suddenly make great improvements, because of the lack of reliability of the test results, questions will be there. It is almost like the Lance Armstrong debate, was his performance juiced or not? No evidence has been found that was the case, but there are people convinced that he could not do it on his own. Maybe he just worked really hard.

    ReplyDelete