Please enable javascript to access the full functionality of this site

Benchmarking critical technologies

PB53-BenchmarkingCriticalTech-banner
Dark
@ASPI_ICPC

Benchmarking critical technologies.

Building an evidence base for an informed critical technologies strategy

Kitsch Liao, Dr Samantha Hoffman and Karly Winkler, with Baani Grewal, Cheryl Yu, Saki Kikuchi, Tilla Hoja, Matthew Page and Jackson Schultz.

What’s the problem?

Technology policy formulation has recently gained a renewed importance for governments in the era of strategic competition, but contextual understanding and expertise in deciding where to focus efforts are lacking. As a result, decision-makers might not understand their own national strengths and weaknesses. It’s difficult to judge whether a country’s R&D outputs, no matter how advanced, and its development of production capacity, no matter how significant, align with the country’s intended strategic objectives or can be used effectively to achieve them.

The ability to measure the relative strengths and weaknesses of a country by weighing specific strategic objectives against technical achievements is of paramount importance for countries.

This is especially true as nations seek to resolve supply-chain resilience problems underscored by the Covid-19 pandemic. China’s rejection of the Quad’s vision of a free and open Indo-Pacific, willingness to use economic coercion and the resulting strategic competition, call further attention to multiple technology sectors’ heavy reliance on a single source. A solution must be found that can exploit synergy across multiple technology sectors among collaborating countries while ensuring supply-chain resilience.

What’s the solution?

Governments’ ability to ensure that strategic objectives pertaining to critical technologies are both well articulated and achievable, and researchers’ and industry’s ability to collaborate in meeting those objectives, would be greatly enabled by the development of an objective and repeatable methodology for measuring technical achievements against clearly defined strategic goals for the critical technology sector. The most pressing challenge should be a relatively straightforward one to resolve: standardise metadata about national objectives and R&D efforts to enable business analysis.

The Quad Critical and Emerging Technology Working Group is an important step towards building collaboration in the research, development and production of critical technologies among like-minded governments. While in nascent stages, the group is gathering momentum and working towards addressing the September 2021 objective to monitor trends in critical and emerging technologies for cooperation, with an initial focus on biotechnology. We recommend as follows:

  • Conduct detailed analysis to understand current and emerging gaps in critical and emerging technologies, starting with biotechnology, among like-minded countries.
  • Develop a partnership between like-minded countries with advanced technological capabilities to deliver a secure technology supply chain for critical tech. This should include a commitment to a set of core principles for technology development and delivery, including ‘baking in’ democratic principles to the technology and agreeing to share any civilian advances on market terms and refrain from coercion.
  • Establish a Quad or Quad Plus critical technologies fund to which participating states pledge investment funds that are then disbursed to address current and emerging critical technologies gaps.

1Introduction to the Benchmarking Critical Technologies Project

Benchmarking Critical Technologies is a pilot project at ASPI ICPC that examines the development of a handful of critical technologies in the context of strategic partnership and strategic competition.

‘Critical technologies’ broadly refers to strategically important technology areas.1 Australia, for example, defines ‘critical technology’ as ‘technology that can significantly enhance or pose risks to Australia’s national interests, including our prosperity, social cohesion and national security’.2 For this pilot study, we focus on the biotechnology and energy technology sectors in China and in the Quad— the quadrilateral Indo-Pacific diplomatic network consisting of Australia, India, Japan and the US.

This project will be expanded over the course of 2022 to include more technology areas and countries.

During the Quad Leaders’ Summit in March 2021, the Quad Critical and Emerging Technology Working Group was announced. The communiqué from the summit said that the working group was intended to ‘ensure the way in which technology is designed, developed, governed and used is shaped by the Quad countries’ shared values and respect for universal human rights’.3 The communiqué didn’t directly name China, but China was clearly implied in its pledge to recommit to ‘promoting the free, open, rules-based order, rooted in international law and undaunted by coercion, to bolster security and prosperity in the Indo-Pacific and beyond’.4

It’s clear that China is the key strategic competitor that the Quad countries are hedging against. They’re technology and manufacturing powerhouses with strong geopolitical influence in the region, which makes the competition both more important and more difficult. As the Quad works to develop capabilities in a range of critical sectors, the Quad members will need to also understand how to leverage each other’s strengths and overcome collective weaknesses to guarantee supply-chain resilience, among other strategic objectives.5 They will also need to triangulate the effects of each nation’s digital enmeshment in Chinese supply chains and the net effects of that in particular sectors.

There’s a lack of empirical data to ground decision-makers’ advice on everything from capability gaps to priority investment areas. This project is an attempt to begin to bring additional empirical data to the decision-making process. Our intent is to offer improved clarity on each country’s strengths and weaknesses in each critical technology. After consultation with the Australian Government, we decided to focus on hydrogen energy and solar photovoltaic (solar PV) technologies from the energy sector, and genetic engineering and vaccines and medical countermeasures in the biotechnology sector.

The broader technology areas that these specific technologies sit within are of clear strategic importance. The Quad Leaders’ Summit communiqué established that biotechnology would be the starting point for the Critical and Emerging Technology Working Group’s collaboration. It also highlighted, in the context of the recent COP26 conference, that the Quad would coordinate to ‘establish responsible and resilient clean-energy supply chains’.6

To assess national capabilities, we measured each country’s R&D and infrastructure development efforts using patent and patent impact data and academic impact data, and compared those results against the country’s technology-specific policy goals. For patents, we collected two measures for each critical technology: the quantity and quality of the patents. IP Australia provided ASPI ICPC with patent data to analyse the quantity of patents for each critical technology. Additionally, using the commercial product PatentSight developed by LexisNexis, we assessed patent quality with the Patent Asset Index (PAI).7 The tool assesses patent quality across various measures in the overall ecosystem of a technology field. Those measures are technology relevance (TR), indicating how much future patents in the field depended upon the patent; market coverage (MC), indicating how much of the global market the patent offers protection of; and competitive impact (CI), the aggregate of TR and MC indicating the economic value of the patent. The aggregate economic value of all patents in the field then constitutes the field’s PAI. For academic impact factors, we used the CiteScore (CS) methodology for measuring impact factors embedded within Elsevier’s Scopus commercial database product.

We also drew on background interviews with industry specialists and senior officials in relevant government departments. Budget data was more challenging to collect, normalise and assess.

Consequently, it isn’t treated as a separate metric, but included with general policy analysis. (For more on our methodology, see the Appendix.)

We recognise that both the policies and technologies on which we base our assessments are evolving. Technology development doesn’t always move in a linear trajectory, and current capabilities aren’t the only indicator of future outcomes. Moreover, the strategic interests and desired policy outcomes one country seeks might not align simply or easily with those of another. Therefore, it isn’t possible to directly compare countries against each other. Rather than arbitrarily rating each country’s progress against the others, we’ve rated each country’s progress in achieving the strategic objectives that it has outlined for each technology area (Figure 1). The progress indicator’s location should be interpreted as being dynamic, given that both policies and technologies will evolve.

Figure 1: Rating scale—country progress in meeting national policy objectives

Rating scale legend

  1. Some high-level policy objectives specific to the technology area have been set, but there’s little evidence of efforts making progress towards meeting those objectives.
  2. Despite the articulation of some policy objectives pertaining to the technology area, those are still relatively unclear. The country’s R&D and production capabilities don’t appear to be sufficient to contribute to realising the country’s stated policy objectives.
  3. There’s some evidence that the country is developing actionable policy in the technology area. There’s clear progress in the country’s ability to contribute to the R&D of the technology, or production capacity. It isn’t clear, however, whether this progress aligns with the country’s stated policy objectives.
  4. There’s evidence that stated policy objectives, research and investment are beginning to translate into aligning capabilities.
  5. There’s strong evidence that stated policy objectives, research and investment have already translated into aligning capabilities.

Source: Image produced by ASPI.

2Overall assessment

  • Quantity doesn’t mean quality, at least in terms of the way patents and research shift global knowledge and capabilities in the overall ecosystem of a technology field. Our findings on patent impact—measured by how often a patent is cited or purchased—highlighted that China, with the highest number of patent applications filed, didn’t have a correspondingly high impact factor.
    Australia and India, and to a lesser extent Japan, filed far fewer patents, but those few patents had impact more on par with US patents, which were high in both number and impact. One patent can significantly influence the evolution of a technology; others might incrementally advance knowledge or create offshoot fields. Impact factors in these types of analysis can be an objective measure for determining scientific advances or commercial success but aren’t necessarily useful in indicating whether national capabilities support policy objectives. If the point of benchmarking critical technologies capabilities at a national level is to understand what makes a country capable of meeting national policy objectives, competitive in a strategic competition and well placed to work with like-minded partners, then the ability of individual researchers or organisations to advance a technology field doesn’t tell us how competitive a country is in translating concepts to capabilities that align with its strategic objectives. For example, ASPI ICPC believes that in China, the disproportionately large number of patents filed internally is most likely attributable to companies patenting specific applications of technology. In the Quad, countries such as Australia and India have been more impactful for a fewer number of patent applications filed and research papers significantly advance the field.
  • Success in connecting policy objectives to outcomes isn’t yet entirely measurable. Our comparison of national policies pertaining to each critical technology we research shows that China, followed by the US, tends to have more clarity about what it seeks to achieve by investing in R&D and production capabilities, and following that up with actions that will achieve those objectives. India, Japan and Australia don’t lack policy development or innovative capacity, but we believe they have been less effective at connecting concepts to capability. This assessment is no doubt at least partially because the development of policy objectives postdates most of our data.
  • Metrics don’t explain the context in which innovation is taking place, including incentive structures, and how that affects a country’s ability to meet specific objectives. In China, the incentive structure is designed so that researchers are working to meet specific policy objectives. In fact, companies closely collaborate with the state in technical standards development. According to the revised 2017 Standardisation Law,8 the Standardisation Administration of China (an agency under the State Administration for Market Regulation) is required to oversee standards initiation and implementation, and in practice technical committees for standards setting under the Standardisation Administration tend to consist of both companies and research institutes.
    We believe the knock-on effect of the incentive structure in China is that the R&D base is disadvantaged, while companies and researchers focus on implementing specific applications of technology that meet policy needs. China’s National Patent Development Strategy (2011–2020) was designed as a ‘long-term and comprehensive plan to use the patent system and patent resources to enhance the country’s core competitiveness’.9 The strategy document prioritises ‘encourag[ing] and supporting[ing] enterprises to upgrade the core technologies and key technologies with patent rights in China’s advantageous fields to national and international standards’.10 We believe companies are seeking to achieve those objectives by owning the market first, and patents support that approach. They’re adding economic value by increasing the quantity of applications, and owning the market comes before efforts to refine the product. Many PRC-originated technologies are being exported globally (see ASPI ICPC’s Mapping China’s Tech Giants project), no matter what the overall quality of the product in comparison to competitors, and that proliferation is probably achieving some market power and incumbency. It’s a cumulative and individual challenge for the Quad nations to move more rapidly from concept to capability in order to avoid the PRC leading in meeting strategic objectives with that technology.

Download

Readers are warmly encouraged to download the full report to access the detailed sector by sector analysis.

The Patent Search Strategy used in the formulation of the report is available for review here

ADF

Australian Defence Force

ACSC

Australian Cyber Security Centre

IEC

the International Electrotechnical Commission

IEEE

Institute of Electrical and Electronics Engineers

IoT

Internet of Things

IoTAA

Internet of Things Alliance Australia

ISO

International Organisation for Standardization

USB

universal serial bus

IIOT

Industrial Internet of Things

ASD

Australian Signals Directorate

CCP

Chinese Communist Party

MERICS

Mercator Institute for China Studies

PRC

Peoples Republic of China

VPN

virtual private network

AI

Artificial Intelligence

SCS

Social Credit System

BRI

One Belt, One Road initiative

CETC

China Electronics Technology Group Corporation

NGO

nongovernment organisation

RFID

radio-frequency identification

CFIUS

Committee on Foreign Investment in the US

SVAIL

Silicon Valley Artificial Intelligence Laboratory

UTS

University of Technology Sydney

ATO

Australian Taxation Office

COAG

Council of Australian Governments

DHS

Department of Human Services

DTA

Digital Transformation Agency

FIS

Face Identification Service

FVS

Face Verification Service

TDIF

Trusted Digital Identity Framework

NUDT

National University of Defense Technology

PLAIEU

PLA Information Engineering University

RFEU

Rocket Force Engineering University

STEM

science, technology, engineering and mathematics

UNSW

University of New South Wales

ZISTI

Zhengzhou Information Science and Technology Institute

AFP

Australian Federal Police

ACIC

Australian Criminal Intelligence Commission

NATO

North Atlantic Treaty Organisation

A4P

Action for Peacekeeping

ASEAN

Association of Southeast Asian Nations

C-34

Special Committee on Peacekeeping Operations

CTOAP

Peacekeeping Training Centre (Timor-Leste)

F-FDTL

Timor-Leste Defence Force

MFO

Multinational Force and Observers

MINUSCA

UN Multidimensional Integrated Stabilization Mission in the Central African Republic

MINUSMA

UN Multidimensional Integrated Stabilization Mission in Mali

MONUSCO

UN Stabilization Mission in the Democratic Republic of the Congo

PNGDF

Papua New Guinea Defence Force

PNTL

National Police of Timor-Leste

RAMSI

Regional Assistance Mission to Solomon Islands

RFMF

Republic of Fiji Military Forces

RPNGC

Royal Papua New Guinea Constabulary

RSIPF

Royal Solomon Islands Police Force

UNAMI

UN Assistance Mission for Iraq

UNAMID

UN–African Union Mission in Darfur

UNAMIR

UN Assistance Mission for Rwanda

UNAVEM

UN Angola Verification Mission

UNDOF

UN Disengagement Observer Force

UNIFIL

UN Interim Force in Lebanon

UNIKOM

UN Iraq–Kuwait Observation Mission

UNIOGBIS

UN Integrated Peacebuilding Office for Guinea-Bissau

UNISFA

UN Interim Security Force for Abyei

UNOSOM

UN Operation in Somalia

UNMHA

UN Mission to Support the Hodeidah Agreement

UNMIBH

UN Mission in Bosnia and Herzegovina

UNMIK

UN Interim Administration Mission in Kosovo

UNMIL

UN Mission in Liberia

UNMIS

UN Mission in Sudan

UNMISET

UN Mission of Support to East Timor

UNMISS

UN Mission in South Sudan

UNMIT

UN Integrated Mission in East Timor

UNOTIL

UN Office in East Timor

UNSMIS

UN Supervision Mission in Syria

UNTAC

UN Transitional Authority in Cambodia

UNTAES

UN Transitional Administration for Eastern Slavonia, Baranja and Western Sirmium

UNTAET

UN Transitional Administration in East Timor

UNTSO

UN Truce Supervision Organization