Ioannidis, J. P. (2018). Meta-research: Why research on research matters. PLoS biology, 16(3), e2005468.
According to John Ioannidis of the Meta-Research Innovation Center at Stanford (METRICS), whatever else that the words meta-research may convey to the reader, this relatively new young discipline is the study of bias (Ioannidis 2018:1/6). What I find interesting is that Ioannidis (2018) is very clearly and firmly situated within the positivism/positivism (a research paradigm, the oldest worldview on ‘science’ and its method). Thus, unlike my other reflective reviews, this one will intersperse more articles on the topic to round out the authorial stances and viewpoints. While research on research matters indeed (Ioannidis 2018), the how and why it matters will differ according to what, where, with whom, and from which viewpoint (research paradigm/worldview) one approaches the very notion of research as scientific engagement. For instance, in my school of design research, scientific acts (Sato 2000) are undertaken with the aim to improve design performance in the world (Poggenpohl & Sato 2009); and, if a better means to understand and represent our ‘users’ may be necessary to overcome embedded bias in the method or tool or practice implemented to elicit understanding of a people, then it follows that one’s user-centric design practice will inevitably improve with illumination of blindspots. Here, for this blogpost I do not quibble between user-centric and human-centric although there are strands of design research discourse which do. Here, I am using the best word for clarity of communication to the widest audience, not just within the confines of academic discourse.
McGee, R. G., & Dawson, A. C. (2020). Fake news and fake research: Why meta‐research matters more than ever. Journal of Paediatrics and Child Health, 56(12), 1868-1871.
“Meta-researchers study, analyse and critique the research pathway”(McGee & Dawson 2020:1868). And, because my own meta-research sits outside the dominant research paradigms of the scientific method espoused by McGee & Dawson (2020) I will reverse the order of the words used by the authors’ to describe the elements which meta-researchers focus on, such as how to conduct, how to test, how to communicate, how to verify, and how to reward. That is, meta-research focuses on the most important set of How To questions in the pursuit of systematic knowledge production (“research”). What I discarded from the original sentence structure is the word choice made by McGee & Dawson (2020) such as “reproducibility” with “how to verify” next to it in brackets. These are reflections of the set of assumptions encapsulated by their research paradigm; positivism or postpositivism which underpins ‘the’ scientific method. This is not to imply that research using an interpretivist or constructivist paradigm is not science but that there has been a long and contentious history where qualitative methodologies (originally called naturalistic inquiry by Egon Guba) had to battle for legitimacy, recognition, and respect in the corridors of knowledge production. Thus, my preference for the clarity yet flexibility of the “how to” explanations used by McGee & Dawson (2020) to describe the interests of meta-research, including my own. They are more flexible for application across a range of research paradigms.
Meta-researchers, for better or for worse, simultaneously examine and experience research culture. For this reason, among others, the positionality of meta-researchers is never fully independent, nor is it totally objective, in the pursuit of different forms of truth and knowing. This does not, as one might readily assume, diminish the validity, legitimacy or impact of its outcomes. Instead, it is precisely this variety of perspectives and interpretations offered by its practitioners that makes meta-research so powerful and worthy of a sustained contribution to knowledge, as well as to how knowledge is produced and then disseminated. (Derrick et al. 2024:2)
Thus, it was not surprising to find my own thesis advisor (external) who had cited Derrick et al. (2024) – a Handbook on Meta-Research – for her co-authored paper (Vienni-Baptista & Herbig 2026), given her focus on interdisciplinary and transdisciplinary research at the td-lab, ETH Zurich. Nor, I guess, is it surprising that I would find my way here, albeit too late to include a chapter on meta-research in my doctoral dissertation in design. Article-based doctoral theses, as prevalent in the Nordics, are not the same beast as a monograph, and one’s overarching framework is closely situated within the publications which comprise two-thirds to three-quarters of the manuscript. However, from the very beginning, I make clear that limitations in my available tools for design research as experienced once I stepped outside the well-documented structures of the formal economic system are what led me back to school for R&D to help improve my own practice. Design methodological development work entailed asking How To questions, such as described by McGee & Dawson (2020) above. Ergo, meta-research for human-centric and participative design approaches (such as “co-creation” for example) would describe my field of study. Hence, I explore this newly emerging discipline and feel extremely encouraged by encountering one of my own thesis advisors in the citations.
Causadias, J. M., Korous, K. M., Cahill, K. M., & Rea-Sandin, G. (2023). The importance of research about research on culture: A call for meta-research on culture. Cultural diversity & ethnic minority psychology, 29(1)
For a while there I was worried that only the natural and hard sciences were interested in meta-research and that I’d only find positivism and its metrics in discussions. That would make it very difficult for me to extrapolate to my own meta-research on design research methodologies which must involve just, equitable, and plural user engagement. I cannot yet link to the thesis itself for why this might be so, but a very broad answer is that such work is more intersubjective, more participatory, and, thus, more qualitative. That is, the word choice used above by McGee & Dawson (2020) which I discarded – “reproducibility” – is not a metric used by any of the research paradigms common to such human-centric design methodologies.
It is impossible to recreate the outcomes of a co-creation workshop even if the very same group of participants will be involved. The second time they will be inspired to brainstorm slightly or very differently from the knowledge products generated the first time the workshop was instantiated. The method/tool/practice/agenda of co-creation (Lee et al. 2018) sits outside the notion of “reproducibility” but the question of “how to verify” must remain in place for those who will use it as part of their scienfic engagement rather than as part of their design practice. That is, “co-creation” is by its very nature a multi-paradigm inquiry methodology, where syncretisation of assumptions of ontology and epistemology must necessarily inform evaluation and judgement. Why? It is at once “essentially unique” yet “generally applicable”. You can plan, design, and run a co-creation workshop on any topic with any set of participants (generally applicable) but what happens is not reproducible, but “essentially unique”. In fact, if reproducibility is enforced then the notion of co-creation is disabled (Bhan forthcoming). This is supported by empirical evidence from a sequence of three workshops run with 8 different groups and 4 different facilitators (Bhan forthcoming). The only thing which we can focus on maintaining consistently over a total of 27 workshops is the participant experience (UX or CX), and that is directly correlated to facilitation (Bhan forthcoming; Light & Akama 2012).
Therefore, discovering the work of Causadias et al. (2023) is very encouraging since it is clearly outside the box of the hard sciences. In fact, there is already what could be called meta research in design scholarship on culture as discerned in Duan, Zhipeng’s (2024) dissertation in service design. My own work, for instance, as seen in the featured image used in this blogpost is an example of purposeful and goal-oriented meta-research to inform the design of user research for a very specific use-case – that of multi-epistemological settings where multiple knowledge systems abound and the bias is embedded in the systems design of the academy (see Saami scholar Kuokkanen 2007 or Torres Straits Islander academician Nakata 2007 for this argument). Causadias et al. (2023) open the systems view of such knowledge production and its praxeology.
Featured image: The development of user research derived from the touchpoints of co-creation as identified through field study, reproduced from Bhan, N., & Tedjokusumo, L. Z. (2025). Knowledge mapping: User research for plural, just, and equitable co-creation interface design. International Journal of Design, 19(3), 83-99