The endless quest for biomarkers: thoughts on IMFAR 2016

imfarA week ago I returned from spending a few days in the USA attending the International Meeting for Autism Research – an annual academic conference, which took place in Baltimore this year. IMFAR, as it is known, is one the highlights of my  year.  An opportunity to get together with researchers from all disciplines who share a common interest in understanding autism and helping people who have a diagnosis, and their families. Last year I blogged about the focus which I feel some of this ‘help’ should have, inspired by conversations and experiences at IMFAR, and I thought I’d do a similar post this year.

This post will centre on the specific topic of biomarkers. To get us started, let’s have a think about what a biomarker is. This article attempts to provide a clear definition and opens by saying:

The term “biomarker”, a portmanteau of “biological marker”, refers to a broad subcategory of medical signs – that is, objective indications of medical state observed from outside the patient – which can be measured accurately and reproducibly. Medical signs stand in contrast to medical symptoms, which are limited to those indications of health or illness perceived by patients themselves.

As a quick translation for anyone put-off by the jargon there (the paper was not written for a general audience), this basically means biomarkers are signs linked to a specific condition which are can be measured by a doctor or scientist with little chance of making a mistake.  Another feature is that measuring the same biomarker over and over should give pretty much the same result each time. These are distinct from ‘symptoms’ which rely on patient report – e.g. “my knee is hurting” – or professional interpretation, as in an IQ test. As the name implies, biomarkers tend to rely on biological samples.  For example, for a genetic condition such as Fragile X syndrome, diagnosis is via a DNA blood test which is objective (not open to interpretation) and, if repeated, the result will come back the same each time.

DNA code analysisWhat are biomarkers for?

Biomarkers provide reliable, concrete diagnosis which doesn’t rely on ‘shakier’ methods like clinical opinion (see below for more on this). In autism, the quest to find a biomarker largely focuses on a desire to be able to categorically state whether an individual has autism or not.  This aim has a few sub-goals as well. First, if we had a biomarker for autism based on, for example, a blood test, this would make it easier to identify autism early in life which would allow young children and their families to get useful support right from the start (but see this post on ‘useful support’ and early diagnosis). Second, this method of diagnosis might be a lot cheaper, quicker and less stressful for families than the current array of clinical assessments. In particular, we might be able to reduce the number of people (especially women) who reach adulthood before discovering that undiagnosed autism explains many of the challenges they have been experiencing in their lives. Third, a collection of biomarkers might help us to define useful sub-types of autism. One of the few things I think almost everyone connected to autism can agree on is the wide variety among people with a diagnosis.  This ‘heterogeneity’ within a single diagnostic label makes it hard to talk about autism, hard to provide useful support for autistic people and their families, and hard to do research. Being able to identify valid sub-types within the autism spectrum might help us gain better understanding, and provide targeted supports which match to specific needs.

So far, so good – the idea of finding an accurate, quick, cheap diagnosis, which can identify meaningful sub-types of autism, sounds pretty good. And sure enough, at IMFAR 2016 it felt ike 9 out of 10 talks I attended ended with “and this could be used to develop a biomarker for autism“. Biomarkers are clearly all the rage. So…

What’s the problem with biomarker research in autism?

Sensitivity / specificity : Developing a biomarker is really really really hard. It is also quite hard to explain why without taking ages about it, but I will do my best.  People interested in picking apart one particularly egregious example of poor biomarker science ought to read this briliant post by Jon Brock. But for the time being, let’s imagine I have developed a new Biomarker for Autism Detector Test (or BAD Test) which uses the length of your little finger of your left hand as a measure.  This test is based on the hypothesis that autistic adults all have left little fingers longer than 6cm*.

Senstivity is a measure of how well a specific test detects people who do have a particular condition.  If I test 100 autistic people with my BAD Test will all 100 of them have long little fingers? Specificity is a measure of how well a specific test can rule out people without a condition. If I measure the little fingers of 100 people who don’t have autism, will every single one measure below the 6cm cut-off point?  The answer to both of these questions is almost certainly No. Some autistic people will score below the 6cm cut-off, some non-autistic people will have long little fingers. So now we have to work out how many of these so-called false positives and false-negatives (people mis-diagnosed by the test in one way or another) are acceptable. 1%? 5%? 10%? What if I run my test on a group of 100 people with autism and only ‘miss’ seven of them – that’s pretty good, right?  But then if I run it again on 1000 people, and miss 70 of them… That doesn’t sound so good. So I try another 1000 people and this time I miss 106 of them. Oh no!  Did something go wrong with my test?  No, this is probably because any one estimate of sensitivity (or specificity) is in itself unreliable. As always in research, running the same study repeatedly will yield slightly different results each time.

As I noted above, I would need a lot more time and word count to go into this issue in detail (and you can read more on the sensitivity and specificity aspect here).  The broad point I’m trying to make is that collecting the data required to check the sensitivity and specificity of my BAD Test will take a while. It requires large groups of people, robust methods (e.g. a way of measuring finger length which is identical every time and can’t be biased by researchers seeing what they expect to see), complex statistics, and a strong a priori hypothesis.  This last one means a clear statement of what we expect to happen, stated before we start the research study, and not just an interpretation of what we found which was made to fit the data after we collected and anlaysed it. Most of the so-called biomarker research I’ve seen at IMFAR and in journals clearly fails to meet these criteria.

Differental diagnosis:  Another key feature of a biomarker is that it should be able to distinguish not just between people with autism and neurotypical people, but also between people with autism and (for example) people with social anxiety.  Or people with attention deficit hyperactivty disorder (ADHD).  Or people with developmental coordination disorder (DCD). One of the major flaws of current biomarker claims is that the evidence nearly always relies on comparison of groups of autistic people with neurotypical peers.  In reality, young children (or adults) presenting at clinical services for a diagnostic assessment do so because their parents (or they themselves) are worried about something.  In some way, they are different from whatever is considered ‘normal’. When evaluated by an expert to see whether an autism diagnosis is appropriate, they might be diagnosed with autism or they might get another diagnosis. For a biomarker to be truly useful it needs to be applicable to this situation of “differential diagnosis”.  The question being asked in a clinic is not “Is this person autistic or neurotypical?” but “Which of the many available diagnostic categories best fits this person’s profile and needs?“. In nearly every case I’ve seen lately proposing a new biomarker for autism, the biomarker’s relevance to the second question is at best limited, but usually non-existent.

Better than a clinical opinion? We’ve already seen that one of the goals of biomarker research might be to streamline the diagnostic process – making it faster, more precise and easier for families. The idea of simply having a blood test for autism must sound appealing to those who have experienced the long waiting times and uncertainty of current diagnostic practice.  However in adopting a streamlined process based on a simple biomarker, we risk losing much of the benefit of the current clinical pathway system. When someone with suspected autism meets over time with a range of clinical practitioners (e.g. paedatricians, speech & language therapists, occupational therapists) we find out a lot more than just their diagnostic status.  The clinical team get to know the individual and their familiy.  They may have some insight into how a child is learning and changing over time. They will be aware of a young person’s life stage and context – what kind of school are they in? what do they like to do in their spare time? what are their goals for the future? To be clear, I haven’t heard any biomarker proponents recommend that a simple test completely replace a comprehensive clinical assessment. But I am concerned about some aspects of the current drive for streamlining and efficiency in diagnosis.  Autism is just not a simple thing and our diagnostic practice, and support services, must embrace that fact, rather than wish it away.

eye-tracking-750x404What about the ‘bio’ part? The term biomarker is a contraction of biological marker.  The strong implication here is that biomarkers shoud rely on biological samples – like DNA. Instead in the world of autism research, increasingly people are assigning tentative biomarker status to responses to behavioural tasks, such as patterns of eye-movements. The major problem I have with this is that, in many cases, it would therefore be possible to fix the test.  An autistic person who doesn’t want to be labelled as such could do their homework and just choose to look at the part of the screen they know autistic people don’t normally look at. And vice versa – a family in need of extra support might be able to acquire an autism diagnosis for their child by instructing them to look at a particular part of the display. If these examples seem a little extreme, consider instead the perhaps more likely scenario that a particular individual repeatedly experiences the same test and their responses therefore adjust over time – as they begin to be more confident with content they intially found off-putting, or get bored of the features that initially captured their attention. A biomarker which is subject to this kind of intentional or inadvertent manipulation by the individual would be clinically meaningless.

Translational laziness: This is my biggest bugbear with the repeated “this could be a biomarker” refrain at IMFAR 2016. At the moment, quite rightly, there is a strong emphasis on so-called translational research.  Developing findings from basic science (i.e. information collected in a controlled laboratory environment) into practice – whether that is diagnosis, intervention, education or social services. One fundamental and common approach in psychology, and other disciplines, is the two-group comparison. We take a bunch of people with autism and a bunch of non-autistic people and we compare their responses to an experimental task, interview questions, online survey etc. When a difference between the groups is found, increasingly the temptation, desperate to prove translational applications of the science, is to say “this could be a biomarker for autism“.  No.  No it couldn’t. A mathematically significant difference between the average score of two groups (e.g. one group’s average is a good bit higher than the other group’s average) has none of the statistical or methodological features required for biomarker development or evaluation.  In the vast majority of cases the finding is unlikely even adeqately to justify someone going on to do the further data collection and analysis necessary to robustly assess a candidate biomarker. No funders would pay for research built on such a tenuous foundation, no scientists would give it a favourable peer review. Not to mention the question of how the supposed ‘test for autism’ would operate in clinical practice ^. But still the same point appears again and again: “This could be a biomarker for autism”

To conclude…

The biomarker obsession among autism researchers at the moment is motivated by a desire to do something useful and practical with their research.  To create a better diagnostic experience for people on the spectrum and their families, to provide a foundation for more personalised support and accurate science. This would be super.  However researchers need to practice far more responsibility in their attempts to locate and report on candidate biological markers. As well as meeting the very high standards required scientifically, they should consider more carefully the clinical and social context in which their proposed biomarker might operate. As it stands, the current restricted and repetitive interest in biomarkers among autism researchers looks like it requires an intervention.

*in case it isn’t completely clear from the context, this test is entirely made-up and bears no resemblance to any autism biomarker research that I am aware of in the real world.  It is ridiculous for a whole host of reasons.

^ for example, a large number of proposed biomarkers for autism rely on recording and analysing patterns of eye-movements but it isn’t clearto me how feasible it is to integrate the specialist equipment and the skills necessary to operate it into clinical services.

 

4 thoughts on “The endless quest for biomarkers: thoughts on IMFAR 2016

  1. Varun

    IMHO, one of the issues that surround biomarker discovery in the field is the sample size. Most of what I saw at IMFAR was done in small samples – ofcourse, I work in genomics, and so I’m biased. But I do think that sample size really needs to go up to get both the sensitivity and specificity needed for biomarkers. Going by the ~1% estimate, in UK and US combined there would be ~ 3.8 million individuals with autism – the size of Panama. Would n = 50, or even n = 500 be representative of Panama?

    The other issue was the sheer number of different measures/scales. Now, this per se isn’t necessarily bad. But then if you have ten different scales on, say, sensory issues, all of which have been validated in small samples, are they really helpful?

  2. Gavin Buckingham

    Speaking as someone who recently invoked the biomarker potential in my most recent publication (http://link.springer.com/article/10.1007/s10803-016-2701-0), it was never at the forefront of my mind that any single measure from a behavioural psychology experiment could ever serve as a viable biomarker. Indeed, the outcome in my study only accounted for <5% of the variance of AQ scores in non-clinical populations.

    However, I could imagine a battery of perceptual, motor control, and cognitive tests, from which a multivariate outcome could become a suitably good diagnostic predictor. I (perhaps optimistically) assume that most scientists who refer to biomarkers do so in the same vein. Furthermore, perhaps there is an appropriate trade off between the specificity of a test and ease of which is it administered – blood-based genetic tests are obviously excellent biomarkers in many pathologies, but they aren't viable to be administer to the general public as a first warning sort of thing (cf. the automatic blood pressure cuffs in some large pharmacies). That being said, I fully agree that everyone could be a bit more straightforward about the logic here…

    1. admin Post author

      Thanks for this comment. I think you and I will have to agree to disagree. I recognise your point that a single cognitive test or experimental task might have more value as a biomarker if incorporated as part of a complete assessment battery. But I can’t see how this would improve on the current system of expert clinical review. A battery of assessments tapping cognitive skills or perceptual approaches would take just as much time and expert input to administer and interpret, while failing to provide rich information which maps onto current educational and therapeutic practice. After all, if I know that a specific person is great at visual search, but poor at reading emotions from the eye-region, even if that could be powerfully associated with autism such that the information could be used in diagnosis, how would I then offer them appropriate support? I would need a lot more knowledge about things like their communication preferences, motor development, and current level of support if I wanted to do more than just diagnose.

  3. rose

    The thing that bothers me about the eye tracking…it seems to be a one size fits all answer. There are presently studies to overcome eye tracking differences by behavioral intervention 25 hours a week for very young children (12-18 months). It is unrealistic to imagine that changes in eye tracking will overcome epilepsy, LD, ID, ADHD, Dyspraxia, health problems, anxiety, depression…HEY, but we have eye contact!

    ““The lack of preferential attention toward biological motion is consistent with diminished attention to the eyes and diminished expertise in social action and interaction found in later life,” Klin and his fellow researchers concluded in a 2009 study published in Nature.” http://www.emory.edu/EMORY_MAGAZINE/issues/2013/winter/features/autism.html

    This obsession is so unrealistic as to be almost laughable. Me thinks the doctor ignores the forest for the veins in the leaves of the trees. Not even taking into consideration that adult autistics say they find eye contact painful. It seems like another excuse for maltreatment of autistics by science. Sigh…

Comments are closed.