I am writing this post after Earth Day and March for Science 2017 for two reasons:
- Science is under more and more scrutiny (as it should be!) and we scientists need to be careful what we put out to be read or heard;
- I have just started a Journal Club for the academic research group I belong to at my university and the first article I chose to present to them was a great example of ‘Science gone wrong’.
In this post, I will be using this article as an example. I am not trying to single out this article, nor the authors, but I thought it was a good example of what we should be avoiding. And, for those of you reading and citing scientific publications, I have a message: it is not OK to just read the abstract and conclusions of a scientific article (even if it is peer-reviewed) without critically evaluating its methods and results. It is not OK to cite an article you have not read completely. I have been tempted to do this myself when under time pressure, but we need to stay alert! And this post aims to show you WHY.
Before I get into the article itself, I thought I would share with you the main ideas behind our Journal Club:
- The meetings are fortnightly
- Everyone in the research group is encouraged to present at least one article during the year
- Presentations are also encouraged from postgraduate students (including Masterate students) and undergraduates at the stage that they will benefit from presenting and critically evaluating an article
- Attendees can read the article before the meeting, but presenters should be aware that not everyone will have had time to read it and should address this in their presentation of the article i.e. begin with a background to the study before the critique
- Critical appraisal is encouraged. Presenters are welcome to present their own work, but a more balanced critique is likely to occur when the author(s) is not present.
You will have noticed how much ‘critical appraisal’ has been emphasised in this list. As scientists, it is our job to be critical of everything we read and hear. The need for this became obvious when we evaluated our first article:
Sandøe, P., Nørspang, A. P., Forkman, B., Kondrup, S. V., Lund, T. B., & Bjørnvad, C. R. (2017). The burden of domestication: a representative study of welfare in privately owned cats in Denmark. Animal Welfare, 26(1), 1-10. doi: 10.7120/09627286.26.1.001
I chose this article because I am interested in the indoor versus outdoor cat debate, the issue of neutering, and selective breeding. This study was published in a well-respected peer-reviewed journal. The aim was to investigate how indoor confinement, neutering, and selective breeding may impact on the health and behaviour of domestic cats.
Their statistical analysis involved prevalence and odds ratios i.e. measures of association between the outcome variables (behavioural problems OR health issues) and the exposure variables (breed, neuter status, housing).
The abstract would lead you to believe that the major findings of this study were: (1) confined cats had ‘significantly’ more behavioural problems than free-roaming cats; (2) entire cats had ‘significantly’ more behavioural problems than neutered cats; and (3) ‘significantly’ more purebred cats had diseases than domestic shorthair cats.
However, we found major issues with the methods and results. The study involved a questionnaire, but this was available in different formats to different age groups (a ‘mixed mode design’) – with the assumption that those over 65 would prefer to answer by telephone interview and those 18-64 would be happy to complete it online. This difference was not dealt with in the authors’ later analysis.
The original questionnaire was available in the supplementary material online and, after closer inspection, the questions they asked were not always well reflected in their results. They re-defined ‘How cats are kept’ using a range of responses given by the pet owners – and not all of these made sense. The question they asked to get these results gave confusing options to owners where ‘indoor’ was defined in various ways not necessarily consistent with most people’s definition of indoor cat housing. The only behavioural problems that were significantly different between how the cats were kept were: ‘house-soiling’, ‘damage furniture or things’, and ‘displays signs of boredom’. These could have been associated with, for example, other cats in the household, availability of toys and environmental enrichment. However, despite these follow-up questions being in the questionnaire there was no evidence of them being used in the analysis.
For neuter status, the only behavioural problems that were significantly greater in non-neutered cats were ‘aggressive behaviour towards guests’ and ‘other problems’. These were poorly defined categories and/or could have been the result of natural feline behaviours.
When it came to looking at the prevalence of health issues, the authors did not adjust for the number of vet visits. Owners are potentially more likely to know about their cat’s health issues if they visit the vet more often. Instead they controlled for vaccination status – claiming it to be a proxy for concern about cat health. They reported that more purebreds had one or more of the diseases of interest. But this was compared to ‘domestic short-hair’ cats, with a separate ‘mixed breed’ category. Purebred cats were grouped together as there were not enough of them in the study for them to be separated by breed. They did not appear to use the information they had about the amount owners spent on their cats in this analysis either.
The discussion section was interesting, if irrelevant at times. The section on body condition scoring (BCS) underestimating the level of body fat percent was a very good review. But as they did not use either BCS or fat percentage estimates it felt a little redundant. Probably the most important conclusion they reached was that we need to do more research on cats. Research into purebred cats lags behind canine research.
In conclusion, I hope I have shown that just reading the abstract and discussion/conclusion of a peer-reviewed article is not sufficient to cite it and/or propagate its findings. After our Journal Club meeting we all felt that this article was trying to prove a set of conclusions (neutering is good, purebred cats are bad, and outdoor cats are better than indoor) rather than fully investigate them. Importantly, we left with it clear in our minds that critical evaluation is important for academic writing. We felt that this article is, unfortunately, going to be highly cited due to: (1) the journal it has appeared in; (2) the authors; and (3) the subject matter i.e. for the very same reason I chose to evaluate it in the first place.