Canada Is Actually Pretty Good at Intelligence Forecasting

While America spends billions every year on sophisticated intelligence gathering so that the CIA can brief the president with timely information on precarious situations in the Middle East, nobody knows exactly how accurate the information truly is. But in a much smaller, little-known Canadian intelligence forecasting unit, excellent and reliable forecasts are churned out regularly, according to a new analysis.

The study, published in the Proceedings of the National Academy of Sciences, found that forecasts issued by the analytic teams of the Middle East and Africa divisions within the Intelligence Assessment Secretariat (IAS) of Canada, matched 76 percent of the actual "variance in geopolitical outcomes" the unit forecast.

“The accuracy of 1,514 strategic intelligence forecasts abstracted from intelligence reports was assessed,” explains the report, authored by David Mandel and Alan Barnes, two defence experts. “The results show that both discrimination and calibration of forecasts was very good.”

Barnes and Mandel were using IAS as a case study for figuring out how accurate internal intelligence forecasting within government agencies really is, generally. The report contends that though forecasting is a vital part of strategic intelligence, giving key decision makers at the highest branches of government geopolitical advice, “intelligence organizations seldom keep an objective scorecard of forecasting accuracy.”

Intelligence forecasts have been under a microscope in the US, with their effectiveness constantly questioned. After missing a number of warning signs about the hijackers who would eventually fly planes into the World Trade Center towers and Pentagon, many accused the intelligence community of not “connecting the dots,” or in other words, spotting the types of trends that lead to predictions of attacks or growing threats. Agencies like the CIA are not fans of internal reviews of their intelligence gathering.

For the IAS, the results of an internal, all-access study is way more flattering than damning. After monitoring the intelligence unit over a six year period (roughly March 2005 to Dec. 2011), the report found several key factors contributing to the accuracy of their forecasts.

Senior analysts tended to produce more accurate work than junior analysts, while “miscalibration” of events “was mainly due to underconfidence such that analysts assigned more uncertainty than needed.”

The researchers, however, caution “tempered optimism about the accuracy of strategic intelligence forecasts,” noting that intelligence producers aim to promote informativeness, avoiding overstatement. Frankly, it's a reassuring finding, just in the fact that policy directives from intelligence experts in Canada caution against extreme decision making.

The analysts generally had postgraduate degrees and were experienced operators with on-the-ground knowledge of regions within question. Their reports were geared toward policymakers looking for quick advice on hot spots, with probabilities assigned to forecasts to better translate their confidence in an incident.

“It is very unlikely [1/10] that either of these countries will make a strategic decision to launch an offensive war in the coming six months,” said an analyst in one of the studied forecasts, with the names of geopolitical actors removed from the report to maintain government secrecy.

In another forecast, the analyst is more confident of their prediction: “the intense distrust that exists between Country X and Group Y is almost certain [9/10] to prevent the current relationship of convenience from evolving into a stronger alliance.”

Analysts within the division determined whether their forecasts were unconditional, conditional, or explanatory judgments that all ranged in degrees of certainty, with numerical probabilities assigned to unconditional forecasts. The analysts weren't alone in their determinations either: reports would be passed to superiors and other government analysts for internal review.

Mandel and Barnes used geopolitical experts to appraise the effectiveness of the forecasts assigning values and designating them to produce statistical evidence.

To be clear, the IAS is the Canadian government’s "strategic intelligence analysis unit" and an internal government agency supporting several branches. It offers policy-neutral assessments on geopolitical events and trends under the Privy Council Office. It’s not a counter-intelligence agency like of CSEC or CSIS (or CIA), but the agency provides detailed forecasts finding their way to the desk of major government actors.

Either way, the study is a portal into the intelligence gathering apparatus of Canada, giving a sense of what bureaucrats see day-to-day when they're judging whether or not to intervene in Mali, or if Syria is getting worse.

In the end, it’s a rare bit of positive news for the Canadian intelligence community given the rash of bad press in the last year. Whether it be CSEC airport spying, or unlawful requests for user data from ISPs, knowing the base level intelligence in the Canadian government is accurate goes a long way in reassuring Canadians that, at the very least, our intelligence is legit.