Guides - Evaluating

Communicating Impact of Distributed Science 2: Measuring Impact

admin — Mon, 14 Nov 2016 19:04:36 +0000

Communicating Impact of Distributed Science 2: Measuring Impact admin Mon, 11/14/2016 - 14:04

“All the good business leaders I know are maniacal about measuring things. They know their sales data and customer-satisfaction numbers, which divisions of their company are beating expectations and which are lagging behind… Measurement is a big part of mobilizing for impact. You set a goal, and then you use data to make sure you’re making progress toward it.” – Bill Gates

Distributed science leaders are increasingly required to evaluate their impact. To do so, they need to measure their impact. But this is not easy – in business contexts, outcomes are eventually reflected in a company’s revenues, but science has no such clear outcome. So measurement is increasingly important to science and in this guide we introduce the basics of measuring impact.

Measures and the Logic Model

The first important thing to remember is not to measure for measurement’s sake. Measure only the things that matter – measure performance with respect to specific goals. Any measures that are not associated with particular goals in an enterprise are likely not necessary. A common way to think through goals of an organization and associated measures are through a logic model.

The model for evaluation of a project - often described as a “logic model” or “input-process-output model” - is typically the starting point for evaluation of a complex enterprise. The whole idea of a logic model is to represent the enterprise in terms of a “theory” about how it will lead to good results. What is the logic of this theory? That is, what are the causal mechanisms for how the project will transform inputs into the desired outputs? Logic models explicitly take a systems view of enterprises.

Logic models are developed by the leadership of an enterprise. The typical approach is the input-process-output model, whereby the leadership team describes the enterprise in terms of a system with inputs, key activities, and outputs and then each of these is evaluated separately according to goals for the enterprise.

The figure below describes a simplified, generic example of a logic model for a research enterprise. Inputs and processes represent resources required and the activities which transform these resources into outputs. There may be goals for each of these resources (say costs of facilities, or number of humans), and goals for the activities (say deadlines for data collection, or number of iterations of analyses). Measures of these goals are operational metrics.

Figure 1: Example Logic Model

Some goals have to do with the outcomes of these activities for those involved in the activities. This could include publications for scientific activity and building capabilities for graduate students in the project. Publications are a clear outcome that is important, but does not necessarily translate into impacts to specific stakeholders. What do publications mean to funding organizations, for example? To get at impacts of enterprises and their activities, it is important to think in terms of the value proposition for a particular stakeholder. [Note – see guidelines for thinking through value propositions.]

Impact Measures

It is important to remember that all measures are not measures of impact. In order to measure impact one must take the perspective of a key stakeholder and somehow determine the impact on that stakeholder. There are many measures an enterprise might use to assess performance with respect to particular goals – but only when those goals are associated with outcomes from the perspective of particular stakeholders do they become measures of impact. The way to decide what matters is to focus on goals of the science enterprise, and a good place to get at the goals associated with overall impact of the enterprise is through the value propositions for each stakeholder.

Reporting Meaningful Measures

Finally, simply reporting a number for a measure is rarely adequate for providing a meaningful measure of impact. If an enterprise reported that the publication garnered ten citations in its first year, is this good or bad? In order to make measures of impact meaningful, they must always be reported in relation to some context. Following ar three tactics for introducing context.

Target & Benchmark Values. A critical part of every key performance indicator (KPI) for an enterprise involves finding a target value for the metric for comparison. Oftentimes this target can reflect the typical value of similar efforts. This comparison to typical, similar situations is referred to as benchmarking. By comparing a score to a target/benchmark, stakeholders can immediately understand whether the impact is more or less positive than expected. For example, 10 citations in the first year is excellent if the target is 3 citations. Whereas this score would not be very strong if the benchmark for similar efforts was 20.
Trends. Sometimes specific targets or benchmarks are either not readily forthcoming or do not make sense. In such cases, simple comparison across previous values is a good way to provide context. 10 citations this year, if a similar effort received 5 the year before indicates a positive trend.
Ratios. Another way to standardize impact measures across dissimilar impacts is to report them in terms of a ratio. For example, a common ratio is “return-on-investment” (ROI), which is the cash value of the returns on the investment (R), divided by the initial investment (I). ROI=R/I. By taking the value of return per investment, one can immediately understand when an impact measure is positive (e.g. ROI=5) or when it is negative (e.g. ROI=.5)

One final comment is that useful impact measures for each dimension of value are scarce. This is an emerging area and one where few useful impact measures have gained widespread acceptance.

“In the War on Disease, Measurement Matters,” Bill Gates, Time, September 30, 2013

Communicating Impact of Distributed Science 1: Value Propositions

admin — Fri, 04 Nov 2016 14:59:20 +0000

Communicating Impact of Distributed Science 1: Value Propositions admin Fri, 11/04/2016 - 10:59

Why should we continue to fund you?

Why should we fund you in the first place?

Why should we increase your funding?

Why shouldn’t we decrease your funding?

The questions above are questions about value. Distributed science enterprises are increasingly asked to justify themselves – to communicate the value they provide. Providing value is about solving a problem or satisfying a need for a stakeholder. It is important that value is always from the point of view of the other party. The answer to “why should we fund you?” should never be “because I need the money.” The answer should always involve the value that the distributed science enterprise provides to the stakeholder. The answer should involve a value proposition.

A value proposition is a science enterprise’s statement of benefit, importance, or worth. Since different stakeholders benefit in different ways, a value proposition should always be in relation to a particular stakeholder. In general, a value proposition involves satisfying a need or solving a problem for a particular stakeholder – providing benefit or mitigating harm. It is critical to remember that value propositions are always from the perspective of the stakeholder.

Stakeholders should be distinguished from each other according to the value propositions that the enterprise offers. For example, although federal funding agencies and Congress are both parts of the federal government, the value proposition for a funding agency might weigh more heavily on scientific findings, whereas the value for Congress may involve a focus on return-on-investment for the country in terms of science, innovation, and workforce development. Congress may be further divided between Congress overall and any particular congressperson who might be most interested in economic return to the district.

There are multiple dimensions along which CI enterprises might offer value propositions. Four dimensions that we have found useful for understanding this value are: science, innovation, workforce, and economic:

Science - A fundamental way that science enterprises provide value to stakeholders involves their contribution to science in terms of scientific findings. The same findings might provide different benefits to different stakeholders. A university may be interested generally in number of high-quality publications that come from the study, whereas a foundation may be interested in specifically how the finding moves their scientific agenda forward.
Innovation - An increasingly visible element of the value science enterprises generate involves the development of innovations like new technologies and startup organizations that often significantly impact industry. Many of the technologies that are fundamental to the Internet, for example, have their roots in U.S. science spending, and many high-profile startup organizations have their roots in science (including Google, SpaceX, etc.) – driving programs like SBIR and NSF’s I-Corps.
Workforce - An important area for the value of science enterprises to a variety of stakeholders involves the impact on the workforce. Oftentimes science enterprises do measure and communicate the courses they provided or supported, students who worked on projects and increased skills, and researchers who upskilled themselves and then moved on. Science enterprises impact the science, government, and commercial workforce through upskilling knowledge workers in unique and significant ways.
Economic - Of course, much of the impact on science can be summed up in terms of economic impact. On the one hand science enterprises can have an economic impact over longer time horizons as a catalyst for economic growth. On the other hand innovative and workforce outcomes can impact stakeholders on an intermediate time-frame. In still other cases direct and immediate funding to a science enterprise may be a direct economic return to a stakeholder (such as federal grants and a university stakeholder, for example).

In combining the focus on stakeholders with the four dimensions of value, one can construct a matrix (see Figure 1) to begin identifying value propositions for each stakeholder group. The process involves listing all possible stakeholder groups along the vertical axis and then determining which dimensions of value apply to the value proposition for each stakeholder. This determination can involve extensive stakeholder analysis efforts.

	Science	Innovation	Workforce	Economic
Stakeholder 1
Stakeholder 2
Stakeholder 3

Figure 1: A Matrix of Stakeholders and Dimensions of Value

Once the dimensions of value are understood, this is just the beginning. Next the value propositions should be stated using the stakeholder’s terms. What goals, benefits, or outcomes does the dimension satisfy for that stakeholder? This may involve interviewing and surveying stakeholders to understand what brings them value. A useful way to think about value for a particular stakeholder along a specific dimension is to think in terms of whether it provides short or long-term value, and whether the value involves bringing benefits or mitigating harm (see Figure 2).

Figure 2: Different Conceptions of Value

When a science enterprise generates benefit for a stakeholder, one can think of this benefit in terms of a continuum of timing of how this value is reached. Benefits that involve proximal, relatively certain outcomes would be akin to buying a product or service - one knows what one is getting. However, many science enterprise outcomes are distal, uncertain, and relatively unpredictable. This sort of value is akin to an investment. A good investment strategy is likely to bear fruit over longer time horizons, but are often fairly difficult to predict or assess in the short-term.

Further, science enterprises can produce value by mitigating harm. In the near term this could involve cost avoidance. Cost, conceived broadly, could involve any number of negative situations or excess resource expenditures that are mitigated by a science project. In the long-term, particularly when conceived in terms of future risks, science efforts can act more as hedges against potential but uncertain issues that may arise in the future.

Thus, herein we provided brief guidance for thinking about value in science enterprises. Value propositions are always with respect to a particular stakeholder, and science enterprises provide value across four dimensions: science, innovation, workforce and economic. This value takes the form of either generating benefits or mitigating harm, and can have fundamentally different implications depending on whether value is generated over longer or shorter time horizons.

Guide to Finding Online Conversations About Distributed Science

admin — Wed, 28 Sep 2016 18:11:13 +0000

Guide to Finding Online Conversations About Distributed Science admin Wed, 09/28/2016 - 14:11

There are many online forums and question-and-answer websites which provide further discussion about online scientific practices, organization, and team building. We provide a list of these websites as well as some interesting discussion threads on these sites.

Quora

Created in 2009, Quora is a general Q&A site where the majority of questions, answers, and organization is performed through its users. These questions are then organized into overarching topics. Some relevant topics and questions from Quora include:

Collaboration

Scientific Research

StackExchange

Originally created as StackOverflow in 2008, a Q&A site specifically for programmers and programming related questions, the website was expanded to become a network of Q&A websites encompassing many different topics, such as beer, the Japanese language, and website development. Some relevant StackExchanges are:

Academia - for questions about academia and higher education

How is a paper with multiple authors usually written? by Math Newb
How to achieve successful collaborations? by user102
Any suggestions to create and maintain a good research atmosphere by seteropere

Community Building - for questions about online community management and moderation

How do you justify moderator actions that are based on secret information? by Mad Scientist
Detecting and preventing hostility to women? by Monica Cellio

The Workplace - for questions about working and professionalism

Why can't many teams be self-organized and how to help them to self-organize? by MainMa
Should I feel morally obliged to stay at a job where I'm an appreciated key player? [on hold] by npace
Workplace gossip makes me uncomfortable - how do I deal with it? by Lucas 'Paul' Kauffman