New technology in open source investment intelligence

The acronym OSINT stands for “open source intelligence” and refers to the process of researching publicly-available, non-secret information to find answers to specific questions that, in turn, can result in some significant action.

This is a continuation of a series on New Technology in investment research:
See: Crowdsourcing investment research: opportunities in OSINT
http://capital-flow-analysis.com/capital-flow-watch/crowdsourcing-investment-research-opportunities-in-osint.html

and Free information and the Efficient Market Hypothesis
http://capital-flow-analysis.com/capital-flow-watch/free-information-and-the-efficient-market-hypothesis.html


and Crowdsourcing investment research: Capital Market Taxonomy
http://capital-flow-analysis.com/capital-flow-watch/crowdsourcing-investment-research-capital-market-taxonomy.html

and Innovation in investment research; dealing with free information
http://capital-flow-analysis.com/capital-flow-watch/innovation-in-investment-research-dealing-with-free-information.html

and Modern technology for institutional investment research
http://capital-flow-analysis.com/capital-flow-watch/modern-technology-for-institutional-investment-research.html

Note: This article requires attention and time.


CIA emblem



The term OSINT is usually associated with government spy organizations, like the CIA or MI5, and to analysts such as that portrayed by Robert Redford in the 1975 thriller classic film, “Three Days of the Condor”.



However, OSINT techniques can be used for any purpose, including investment research.

(For a list of OSINT tools on the Internet, see Radolph Hock’s list, including the interesting “web-chatter tool”.

The newly-launched Capital Market Wiki combines OSINT techniques with crowdsourcing methods in a semantic wiki context.

With information now available on the Internet, OSINT methods can be used to gain competitive advantage in security analysis, based on the disparity between the relatively sparse data available through traditional sources (Standard & Poor’s, Bloomberg, Moody’s, etc.) and the apparently unmanageable tsunami of free data published on the Internet and elsewhere.

In this article, I go into what OSINT entails.


The five steps in the OSINT process




Anyone can surf the Internet, looking for information about anything.

However, to get practical “actionable intelligence” out of the process and really get at “the good stuff”, you need to go about it in a systematic way.

Specialists in open source information intelligence have, over the years, developed a logical method of processing such data which is just as applicable to investment data on the Internet as it is when looking for terrorist plots.

The five steps in gathering open source information and converting it into “actionable intelligence” are as follows:

Requirements Definition: Define the questions to be answered.

Collection Management: Define the way the data gathered is to be organized and handled.

Source validation and discovery: Locate sources of information and determine validity.

Multi-source Fusion: The technique for bringing information together from multiple sources in such a way as to be useful.

Actionable Presentation: Delivery of the harvested information in a format such that the requirement definitions are met and decisions may be made.

Requirements Definition: Putting the Question

Traditional securities analysis, as in Graham and Dodd, focuses on a relatively few, well-defined documentary sources that present information in an expected format, such as annual reports or SEC 10-K forms.

The analyst can often go directly to certain pages, such as the income statement in an annual report, and start analyzing according to a familiar procedure.

Open source informational analysis is quite different in that the information is not standardized and usually consists mostly of irrelevant matter that must be excluded before the analyst can process the kernel of information that is useful.



Know the question

Browsing the Internet can be a tremendous waste of time if one doesn’t have a specific goal. The OSINT technique is to have a defined set of questions that serve as a template in determining whether certain material is relevant or to be discarded.

For example, Capital Market Wiki defines a standard set of questions used when sifting through information about securities, such as stocks, bonds, or derivatives:

1.What are the legal rights and obligations of issuers and investors with regards to this instrument?

2.If this is an specific instrument of an issuer, what is the medium and long-term outlook for investors in this instrument, both in terms of legal performance on the part of the issuer and in terms of the valuation of this instrument in the marketplace?

3.What are the tax and accounting implications of this instrument, both for investors, foreign and domestic, and for issuers?

4.What investment strategies and techniques, and financial methods and operations are associated with this instrument?

5.How is the value of this instrument calculated and how is it derived from other values, such as commodity prices, indices, interest rates, or exchange rates?

There are other sets of questions that are applicable when investigating legal jurisdictions, financial institutions or issuers, and various types of financial operations, rules, or regulations.

Information that is not relevant to the questions you are trying to answer can be disregarded.

Collection Management: Every fact in its place


Open source information often comes in dribs and drabs — little bits here, little bits there.

These bits of information must be carefully gathered, sources annotated, and filed in such a way that information on the same topic easily comes together from many different sources.

If you are working alone, you can devise anyway that suits you to manage information. However, the real opportunity in open source investment research is crowdsourcing, where many analysts who may not even know each other work together on the same subject at the same time.




Typeical outline format

This requires a standardized, easy-to-understand, intuitive system that allows everyone collaborate smoothly.

Software such as the Zotero extension to the Firefox browser is part of the solution. The other part is specialized collaborative editing systems, such as the semantic Capital Market Wiki that employs Capital Market Taxonomy to manage data as it is collected.

Capital Market Wiki also features “recommended formats” for over seventy classifications of financial topics. (See example on right — an outline for an article about preferred stock.)

These “recommended formats” contain pre-packaged outlines on many topics, annotated with popup help-boxes with hints to analysts as to questions that might be answered under specific sub-headings.

This makes it easy to collect related information from many sources in one place to check for consistency and accuracy, while developing coherent statements of fact.

Source validation and discovery

Much information on the Internet is simply not true. Much is obviously opinion, not purporting to be fact. Other supposed “facts” are easily identified as “ranting” or common misconceptions. Most information, however, is not relevant to the questions the analyst is trying to answer.





Separating truth from fiction


Some sources, however, such as SEC archives and annual reports, are supposed to be true, or, at least, “official”.

Information from “non-official” sources, such as blogs or investor forums that are represented as factual, need to be checked against other sources for corroboration and consistency.

Sources of all data that is collected need to be carefully documented, so that other analysts may also judge the reliability.

Capital Market Wiki contains a system for citing sources that is an essential part of the OSINT process.

The wiki system invites further corroboration of facts by publishing the data on the Internet for all to see, inviting correction and comment from any reader.

The system also has a form of open peer review, based on the concept of wiki reputation and merit awards.

Multi-source fusion

Because of Capital Market Taxonomy and the use of “recommended formats”, with the concept “every fact in its place”, data posted from many different sources and from different analysts comes together on same place on the wiki, more or less automatically.


Many eyes on the problem ...



Because a wiki is open to all, every analyst can see all the information posted on a particular topic, check the sources, and, if necessary see additional related information.

This feature of “many eyes on the problem”, combined with the policy of “perfectible accuracy”, allows the development of information that can be more complete and accurate than would be possible with a single analyst, working alone, using only “official sources”.

Actionable presentation



The NRO research


Because articles in Capital Market Wiki consist of facts intended to answer a certain set of questions of interest to investors, organized and presented in a standard format, each article has the potential of being an “actionable presentation”.

However, articles are never actually finished or complete. On a wiki, articles are never “approved” and “published” in the sense that a CIA agent might prepare an intelligence report.

Each reader must determine whether an article is complete or reliable enough to use as the basis for a decision. However, it is not difficult to determine the status of an article, since tabs on each article show all editors that have worked on the article, the editors that have contributed the most, and by going to the editors own page, the reader can see all the other articles on which the editor has worked, comments and criticisms of other editors, and any awards granted to an editor by peer review.

The transparency of the editing process on a wiki is far greater than on analysis published by commercial sources.

See the NRO research for an example.


Anonymous editing

A feature of Capital Market Wiki is anonymous editing , which is encouraged.

Because facts posted to the wiki must be verifiable from independent sources and are subject to correction and revision by any reader, the real world identity of a particular editor is not relevant. Instead there is the “wiki reputation” of the editor in terms of the number and quality of past edits, discussions with other editors, and general behavior.

Because of the anonymous editing, postings to the wiki can be made by government officials, employees of companies, and active professionals in the market. This makes it easier for, say, a company official to correct errors and misinterpretations that have been published, citing the correct official source, without having to check with legal counsel.

Anonymous editing also facilitates the use of the wiki system by investment research departments, universities, or exchange officials.

For further information, see Capital Market Wiki.
Photo credit: Truth: flickr by Jean-François Chénier
Phptp credit: Metro audience: flickr by Liz Noise
Tags: capital market wiki, investment research, OSINT, Technology

http://capital-flow-analysis.com/capital-flow-watch/new-technology-in-open-source-investment-intelligence.html