What Cybersecurity Research Funding Covers (and Excludes)

Measuring Impact in Science, Technology Research & Development: A Guide to NSF Grants Measurement is a critical component of research and development projects, particularly those funded by the National Science Foundation (NSF). The NSF provides grants for various research initiatives, including the 'Funding for Cybersecurity Innovation for Cyberinfrastructure' program, which supports research on securing scientific data, workflows, and infrastructure. To effectively measure the impact of such projects, it is essential to understand the scope, trends, operational challenges, risks, and reporting requirements associated with NSF grants. ### Defining Measurement in Science, Technology Research & Development Measurement in the context of Science, Technology Research & Development refers to the process of assessing the effectiveness, efficiency, and impact of research projects and programs. This involves tracking various metrics and outcomes to determine whether the project is meeting its objectives and contributing to the broader goals of the NSF. For NSF grant recipients, measurement is not just about reporting; it's about understanding what works, what doesn't, and how to improve. Concretely, measurement in this sector involves tracking research outcomes, such as publications, citations, and patents, as well as the development of new technologies or methodologies. Applicants should be aware that the NSF has specific requirements for measurement and evaluation, and not all projects may be a good fit. For instance, projects that are more focused on basic research may not be expected to demonstrate immediate practical applications. In terms of scope boundaries, measurement for NSF grants typically involves assessing the project's progress against its stated goals and objectives. This may include tracking metrics such as the number of researchers involved, the amount of data collected or analyzed, and the development of new tools or methodologies. Who should apply? Researchers and institutions with a strong track record of conducting research in areas relevant to the NSF's priorities. Who shouldn't apply? Those without a clear plan for measurement and evaluation, or those whose projects are not aligned with the NSF's priorities. One concrete regulation that applies to this sector is the requirement to comply with the NSF's Proposal and Award Policies and Procedures Guide (PAPPG), which outlines the policies and procedures for proposal preparation, review, and award administration. A verifiable delivery challenge unique to this sector is the difficulty in quantifying the impact of basic research, which may not have immediate practical applications. ### Trends in Measurement for NSF Grants The NSF is increasingly emphasizing the importance of measuring the impact of its investments. This includes a focus on outcomes-based evaluation, where the emphasis is on what the research achieves rather than just what it produces. The NSF is also promoting the use of metrics that can capture the broader impacts of research, such as its effects on education, diversity, and societal outcomes. To stay ahead, researchers will need to develop their capacity for data collection and analysis, as well as their ability to tell compelling stories about their research. Capacity requirements include having access to robust data management systems, as well as personnel with expertise in evaluation and assessment. The NSF is also prioritizing the development of new metrics and methods for evaluating research impact, including the use of altmetrics and other non-traditional metrics. ### Operational Challenges in Measuring NSF Grants Measuring the impact of NSF grants can be operationally challenging. One of the key challenges is ensuring that data collection and reporting systems are in place from the outset. This requires careful planning and coordination among team members, as well as with the NSF. Staffing requirements include having personnel with expertise in evaluation and assessment, as well as data management and analysis. Resource requirements include having access to robust data management systems, as well as funding for personnel and other expenses related to measurement and evaluation. Another challenge is ensuring that measurement and evaluation are integrated into the research process from the start, rather than being treated as an afterthought. ### Risks and Compliance Traps One of the key risks associated with NSF grants is non-compliance with the NSF's policies and procedures. This can result in delays or even termination of funding. Eligibility barriers include failing to meet the NSF's requirements for measurement and evaluation, as well as a lack of alignment with the NSF's priorities. Compliance traps include failing to report on research outcomes and impacts, as well as not having adequate systems in place for data management and stewardship. What is NOT funded? Projects that do not have a clear plan for measurement and evaluation, or those that are not aligned with the NSF's priorities. ### Reporting Requirements and Outcomes NSF grant recipients are required to report on their research outcomes and impacts through a variety of mechanisms, including annual reports and final project reports. The NSF also requires recipients to participate in evaluation and assessment activities, such as surveys and case studies. Required outcomes include advancing the state of the art in their field, as well as contributing to the broader goals of the NSF. KPIs may include metrics such as the number of publications, citations, and patents, as well as the development of new technologies or methodologies. When applying for an NSF grant, it is essential to understand the reporting requirements and to plan accordingly. The NSF Career Awards and NSF SBIR programs are examples of initiatives that have specific reporting requirements. FAQs Q: How do I ensure that my project is compliant with NSF's measurement and evaluation requirements? A: Familiarize yourself with the NSF's Proposal and Award Policies and Procedures Guide (PAPPG) and develop a clear plan for measurement and evaluation. Q: What kind of metrics should I use to measure the impact of my research? A: The NSF is promoting the use of metrics that can capture the broader impacts of research, such as its effects on education, diversity, and societal outcomes. Q: How can I find out more about the NSF's priorities and how my project aligns with them? A: Review the NSF's strategic plan and the solicitation for the specific program you're applying to, and consider reaching out to a NSF program officer for guidance.