When should evidence change practice? Part 3 levels of evidence

Further to the previous post which considered claims that not all the relevant evidence had been included in Novak’s systematic review, and why that was. The Children’s Bobath tutor and others Bobath supporters raised further methodological issues about the research. This related to whether evidence from other sources than high quality RCTs should be considered, given that there was so much of this low grade evidence and the results were so +ve.

 

Levels of evidence

Picking up on the notion of levels of evidence., a current Bobath student added that  “you only have to look at the multitude of case projects on the BBTA website to …. to recognise a beneficial impact of this approach. These projects are not by the “converted” but by students who typically have mixed views as to this approach and are exploring the techniques with a wide variety of patients in a wide variety of settings. In terms of quality of evidence, I recognise case studies rate poorly in comparison to other methods, however, feel the multitude of these case projects, all demonstrating very positive effects as a direct result of a patient’s treatment with specific Bobath techniques can be underestimated”.

I, of course, was agog to find out more about this new evidence that should not be underestimated, so I had a good look at the case studies on the website. There are 18, and they were submitted as the assignment for a Basic Bobath Course and were uploaded between 2010 and 2015. There have, of course, been many more than 18 people who have attended these courses in this time period but we don’t know why or how this select group were chosen for wider dissemination – perhaps they were just the few that showed Bobath in the best light, or perhaps there was another reason, we don’t know. Unfortunately the ‘evidence’ they provide is negligible.  Their number and the beneficial impact they appear to demonstrate does not overcome the poor quality evidence they provide.

The good news is that all the reports should positive results – Yes, every single one and sometimes (literally) incredibly positive results. So it works, right? ‘Fraid not. The ‘results’ presented are at the higher end of scientist bias – almost as biased as it is possible to be. This is why.

Bias in study design

Observer bias

Obviously the authors have a vested interest in reporting +ve results, it’s an assignment! One is not going to submit something that shows they tried using Bobath and it didn’t work – how would that be marked?

The assessments were undertaken by the authors who were aware of the purpose of the study and the treatment received which involves an inherent observer/ assessor bias, one of the greatest causes of scientific bias. Having a blinded assessor who is unaware of (and has no vested interest in) the treatment received would reduce this.

Performance bias

The patient also knows that s/he is getting special treatment and is involved in something important to the therapist- so they are going to try their hardest and be positive about the treatment received.

Reporting bias

The reporting is selective. It is a principle of research quality that all outcomes are presented – so what happened to all the assignments that aren’t uploaded? How many of them reported less spectacular results?

Only immediate effects are studied, we don’t know if these changes were permanent or wore off as soon as the treatment finished. For any treatment to be useful the effects need to be sustained.

Selection bias

The sample of patients involved is highly selective – most are young, do not have serious co-morbidities, were previously fit and well, are positive and motivated and have motor-sensory impairments only. So there is a lot of cherry picking going on. If the treatment works for these ‘best case scenario’ patients, can we say it would work for everyone? No.

Ineffective use of outcome measures

Most of the assignments have attempted to use objective measurement tools to demonstrate effect but many are not used effectively. None provide data to demonstrate that the patients’ performance on the measures is representative of their ability. They are just one-off measures. The strength of their argument could be increased if multiple measurements had been taken and mean values presented (one or two folk tried this). Or they could use a single case design and take multiple measurements over time before, during and after the intervention. If the treatment is effective then it should show an improvement clearly above the variability of baseline that is sustained once treatment has finished

Lack of control

There is nothing that indicates any changes observed were down to the Bobath treatment received as there no control. So Bobath may have had an impact and/ or it could be down to spontaneous recovery and/or any of the other treatment during the same period.

The reason that good quality randomised controlled trials (like the ones selected by Novak et al in their review) are superior and provide stronger evidence than these ‘low level’ case reports and those listed by the Children’s Bobath tutor on 17th March, is that they have been designed to overcome sources of bias. They are designed to ensure the participants are representative of the clinical case load and have broad and inclusive recruitment strategies (a so called pragmatic trial) – so they include the complex, diverse population of potential patients. They have also included a sample size calculation which works out how many participants is needed to show an effect if one exists, so the trials is done on just enough participants– not too many and not too few.

So that is why one should take note of the good quality, unbiased trials when it exists – rather than the low quality evidence. The only time one can justify relying on such low quality evidence is when nothing stronger exists, which is not the case in this situation.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s