Introduction

In this article one of the most debated intelligence company is presented: the Palantir Technologies, a firm that is recognized as one of most brilliant providers for data analysis solutions. It is also known for its collaborations with US Intelligence agencies, and according to part of the security community it could be also involved in the US surveillance program known as Prism.

About Palantir

Stellar Wind, Prism, EvilOlive and ShellTrumpet are all names that have emerged after the revelation by the press of the US's massive surveillance programs.

All these programs have the primary intent to control any form of communication within the United States, according to the source of intelligence, the surveillance activity collected an impressive quantity of metadata for at least a decade. Information acquired due the massive analysis of emails, voice records and any kind of users' online experiences.

The authorities declared that the content of conversations and emails was not analyzed but intelligence analysts are sure that the reality is quite different, and that the surveillance program is extended to global scale thanks to systematic computer hacking on foreign networks.

The amount of information to analyze is impressive and the US Government is supported by companies specialized in this type of activities. In some cases it has directly financed the growth of private entities providing millionaire contracts. Palantir Technologies is one of most highly debated companies, many sources accredit it as one of the principal technological partners for the PRISM program. But it is obvious that extensive surveillance programs are the result of intense work of intelligence and of the collaboration of many similar entities.

The company was founded in 2004 by Alex Karp, Joe Lonsdale, Nathan Gettings, Peter Thiel and Stephen Cohen, curious is the origin of the name that comes from the "seeing stones" in J. R. R. Tolkien's fantasy epic The Lord of the Rings.

"I never learned to drive because I was busy reading, doing things, and talking to people," he says. "And I'm coordinated enough to bike, but the problem is that I will start dreaming about the business and run into a tree."

These are the words of Alex Karp, CEO of Palantir.

Today each contract signed by Palantir is estimated in hundreds of million dollars, the company is in continuous expansion conquering an increasing portion in a market once monopolized by companies such as Raytheon, Northrop Grumman, Lockheed Martin and IBM.

Back to the origins

As I have written before Palantir history seems to be linked to the PayPal Company. Paypal in fact has had from its inception serious problems with cybercrime. The giant of online payments still attracts cybercriminals and hackers and it is constantly at work to prevent online frauds and money laundering services.

According Peter Thiel, a PayPal co-founder, there was a moment in the history of PayPal, (exactly in 2000) in which the cost sustained to contain cyber threats had a serious incident of overall business, that risked the stop of the company's activities. The principal problem was that the company was using human resources to control the majority of suspicious activities, and the lack of total automated systems to recognize illegal activities. PayPal decided to create a corporate software to analyze any transaction searching for illegal activities, the need was to build an AI solution able to predict suspicious events basing its analysis of different datasets.

Just after the acquisition of PayPal by ebay in 2002, Peter Thiel, one of the Palantir founders, started a hedge fund. Clarium Capital Management and Joe Lonsdale decided to make use of the experience gained in fraud matters at PayPal. They designed a first system to conduct complex data analysis based on an artificial intelligence engine. Later in 2004 Thiel, Lonsdale, and a couple of former colleagues created the Palantir Technologies with the principal intent to offer the government their data analysis solutions.

In effect the company attracted the attention of many capital firms such as Kleiner Perkins Caufield & Byers and Sequoia Capital but the funds were finally provided by In-Q-Tel, a not-for-profit venture capital firm that invests in high-tech companies for the sole purpose to provide best technology solutions to the CIA and other Intelligence agencies in the country.

Within the Intelligence environments many officials were skeptical about the capabilities of Palantir solutions, but soon they discovered a new way to approach big data world. Michael E. Leiter, the former head of the National Counterterrorism Center adopted Palantir solutions using the following words to describe the eclectic Karp:

"There's Karp with his hair and his outfit—he doesn't look like me or the other people that work for me,"

Today Leiter is a Palantir consultant.

Figure 1 - Alex Karp CEO for Palantir Technologies

"In April 2010, Palantir announced a partnership with Thomson Reuters to sell the Palantir Metropolis product as QA Studio. On June 18, 2010, Vice President Joe Biden and Office of Management and Budget Director Peter Orszag held a press conference at the White House announcing the success of fighting fraud in the stimulus by the Recovery Accountability and Transparency Board (RATB). Biden credited the success to the software, Palantir, being deployed by the federal government. He announced that the capability will be deployed at other government agencies, starting with Medicare and Medicaid." Reported Wikipedia.

Palantir solutions are practically usable in every context in which it is necessary to analyze huge quantities of data to propose an intelligence analysis. The excellence of its products makes it possible for the company to double in size every year.

The company offers a suite of software applications for integrating, visualizing and analyzing a huge quantity of information, supporting the analysis based on: structured, unstructured, relational, geospatial and temporal data.

The products of Palantir are mainly focused on security, they are broadly deployed in the defense, intelligence, law enforcement and financial communities. Many other sectors are demonstrating an increased interest in its solution to respond to the need to analyze a quantity of information even greater.

But the story of Palantir is also made of shadows, early 2006, Palantir's Shyam Sankar and Dr. Asher Sinensky constituted the shell company SRS Enterprises that licensed several i2 software products. The software included widely used intelligence analysis program that was later used to support Palantir for the development of critical components of its solutions.

Palantir was accused to have designed some of its intelligence solutions, copying many of the features present in the i2 software (e.g. User interface).

In February 2011, Federal Judge Liam O'Grady approved a settlement resolving a copyright and breach of contract lawsuit in which i2 Inc., a provider of intelligence software, and accused Palantir Technologies Inc. to have stolen its solution and trade secrets.

Palantir and i2 issued a joint statement revealing they reached a settlement with mutual satisfaction but no further news was provided.

The name Palantir had risen to prominence just after the publication of mail stolen during the hack to Intelligence firm Stratfor. One of the emails published by the group of Anonymous hacktivist, Palantir was expressly indicated as a possible financier of Facebook. The email between the two Stratfor analysts stated:

"I think Palantir is involved in things less clear, including the financing of Facebook."

Palantir was mentioned in the case of the HBGary Federal company hack, when some of stolen documents were detailing the involvement ofPalantir to attack and destroy WikiLeaks. Wikileaks uncovered a presentation from Palantir, which at the behest of the government, was supporting private companies and institutions to contrast the "Wikileaks threat".

Palantir representatives on that occasion refused the accusations interrupting any contact with HBGary and remarked that company mission is to make data analytics software and not offensive solutions for hacking. CEO Alex Karp personally apologized to Glenn Greenwald, a journalist supporter of Wikileaks, who was specifically named in the document as a potential target of the Palantir operations.

"Palantir Technologies provides a software analytic platform for the analysis of data. We do not provide – nor do we have any plans to develop – offensive cyber capabilities. Palantir Technologies does not build software that is designed to allow private sector entities to obtain non-public information, engage in so-called "cyber- attacks" or take other offensive measures. I have made clear in no uncertain terms that Palantir Technologies will not be involved in such activities. Moreover, we as a company, and I as an individual, always have been deeply involved in supporting progressive values and causes. We plan to continue these efforts in the future. The right to free speech and the right to privacy are critical to a flourishing democracy. From its inception, Palantir Technologies has supported these ideals and demonstrated a commitment to building software that protects privacy and civil liberties. Furthermore, personally and on behalf of the entire company, I want to publicly apologize to progressive organizations in general, and Mr. Greenwald in particular, for any involvement that we may have had in these matters." Karp said.

The solutions and field of applications

Palantir Technologies provides solutions across a wide range of industries. Its platforms are adopted with success by governments, pharmaceutical firms, law firms, intelligence companies, banks, hospitals, law enforcement and defense agencies.

The solutions designed by the company result particularly useful for defense and cyber security, both sectors in intelligence analysis offer a huge quantity of information, apparently unrelated and coming from various sources, represents a primary need.

The offer of the Palantir Company is articulated in three principal branches:

• Government: Intelligence / Defense / Law Enforcement /Accountability
  
• Commercial: Cyber / Legal Intelligence / Home Lending / Capital Markets / Anti-Fraud
  
• Health: Pharma / Insurance Analytics / Healthcare Delivery / Disease Response
  

Principal platforms developed by the company are Palantir Government (aka Gotham) and Palantir Finance (aka Metropolis).

Palantir Government integrates structured and unstructured data, provides advanced analysis instrument for knowledge management and facilitates distributed and secure collaboration. It also implements the privacy and civil liberties protections mandated by legal requirements such as those in the 9/11 Commission Implementation Act. Despite the announced care for used privacy, many privacy advocates and organizations for privacy liberties protection are critical with the techniques of date mining implemented by the product. The company has always remarked that thanks to the information labeling and profiling it is able to allow analysts to see only the information they have permission to access. AnalyzeThe.US is a free accessible sub solution of the Palantir Government platform and it makes data analysis possible by elaborating data from data.gov, usaspending.gov, the Center for Responsive Politics' Open Secrets Database, and Community Health Data from HHS.gov.

Palantir Finance is a solution for data integration, information management and quantitative analysis based on commercial; proprietary and public data sets; and discovers trends; relationships and anomalies. The solution: Palantir Finance is typically adopted for the study of the market to elaborate trading strategies, it's public demo is known as JoyRide.

Defense

Palantir provides solutions for defense to access, and analyze real time critical information necessary for data-driven decision making. The solutions are used to acquire and elaborate high-quality data from diverse intelligence disciplines. One of particular importance is adoption of Palantir solutions in war zones, considered hostile environments in which it is essential to collect in real time a huge quantity of information and traffic data to fuel intelligence analysis. Palantir Defense is an open platform that could interact with other intelligent engines adopted in the defense industry through compliance of open standards.

Palantir has designed a specific solution: Palantir Defense, to capture the demand in the defense industry "Transform massive-scale data from multiple intelligence disciplines into actionable insight". The solution allows war fighter's access to critical data, providing for it a usable representation.

"Unstructured message traffic, structured identity data, link charts, spreadsheets, telephony, documents, network data, sensor data—even full motion video can be searched simultaneously and intuitively, without the need for a specialized query language."

Palantir Defense provides a framework for collaborative analysis across geographically dispersed locations, and users supporting tactical operations in the battlefield.

Palantir Defense supports mobile operations enabling "Blue Force Tracking, intelligence gathering, and collaboration between units in the field and commanders at base stations."

Palantir Defense is considered a powerful instrument for Intelligence and OSINT analysis, it is an instrument used by the US Government for Special Operation Forces and Conventional Forces, it is able to overcome problems such as reduced connection bandwidth, high latency networks and satellite communications.

Figure 2 - Palantir Defense

Palantir – A controversial case

It's not clear the real level of adoption of the Palantir Defense solution within the US Army, in September 2012 the U.S. Army investigated how the 3rd Infantry Division obtained intelligence software on a cost-free basis from Palantir prior to its yearlong deployment to Afghanistan.

Figure 3 - Afghan Operations Insight with Palantir

Once discovered the unauthorized installation of Palantir-provided servers at Fort Stewart, Ga., for training and to provide an analytical backbone during its 2012-2013 deployment. The Army ordered the serversddd shut down and has ordered them removed by the end of September 2012.

Essentially the software was used to collect and organize an impressive amount of data stored in different government databases to help deployed units track and pinpoint insurgent leaders, IED (Improvised Explosive Device) strikes and IED territorial distribution, an function that had resulted vital in Afghanistan despite the 82nd Airborne and a Stryker brigade.

It's not a mystery that U.S. Special Forces Command, various Marine units and foreign militaries also use the Palantir solution for data analysis in critical areas such as Afghanistan.

Investigation revealed a strange circumstance, the 3rd ID didn't have the funding to purchase the Palantir solution, but a trusted source revealed that every Army unit that had requested the Palantir software had contacted the company on its own initiative by finding emails and phone numbers on the firm's website.

Palantir is a great tool that allows analysts to correlate events and data coming from the war field to conduct complex analyses in real-time, but various internal currents are pushing for its adoption at different levels and for many purposes. Probably the case of the US Army is not isolated and the deployment of similar solutions request a great effort also on the Army side and a commitment from high officials, how is it possible that Palantir has been deployed without direct approval of the competent authorities.

AnalyzeThe.US

Another solution which is very interestingly provided by Palantir is AnalyzeThe. US that allows its customers to analyze vast amounts of data released into the public domain. It is a mine of information publicly available that includes key datasets from www.data.gov as well as datasets compiled by leading nonprofits and policy centers, covering such areas as federal spending, contracting, lobbying, and campaign finance.

The tool essentially represents the extension of Palantir Government solution for data in the public domain. The information is related to organization and key individuals. The data is integrated in a unique platform and gives to the analysts a powerful instrument to control the flow of resources, money, and the impact they have on the evolution of the country politic and on government functions.

Palantir provides a free demo to evaluate the solution online, a good opportunity for evaluating the possibilities given by the solution and the complexity of the analysis it conducts.

The principal scope declared by the company is to keep citizens well-informed to improve democratizing processes in government, but the information is in reality gathered from countless domains, the solution is in fact able to combine such widely dispersed knowledge in a single analysis platform, some examples are:

• Government spending data from www.usaspending.gov
  
• The GSA's Excluded Parties List System
  
• The Center for Responsive Politics' Open Secrets Database
  

The company incentives is also the sharing of private/public dataset through AnalyzeThe.US, it is enough to send them an email to join in the project.

The application currently is not compliant with Mac OS and Linux due to a bug in Java used by the solution. At first execution, after a free account registration, the system starts to download all the needed resources to run the application, nearly 50MB of material. The Webstarting component for Analyze the U.S. execution allows users to run Palantir without local installation.

Figure 4 - Palantir Analyze the U.S.

Cyber

Another sector in which Palantir solutions is very useful is cyber security, Palantir Cyber solution "provides commercial enterprises, civilian government organizations, and defense and intelligence missions with the ability to rapidly and intuitively interact with massive-scale data to harden defenses against cyber threats."

The principal purpose of Palantir cyber, is the prediction and identification of cyber threats to reduce surface exposure for governments and organizations. The entities that adopt Palantir Cyber could rapidly respond when a penetration does occur, and could identify fraudulent activity in the bud.

The system filters information for homogeneous and elaborate relevant data for cyber security providing a unified view for investigation and analysis.

The system has been designed to cut irrelevant noise and to identify the windows of actionable information providing an efficient solution for investigating incidents.

"While disparate sources and types of data are typically stuck in silos separated according to different analyst functions, Palantir Cyber enables analysts to evaluate data in a unified context and share analysis securely across roles."

Mission … data analysis

Modern society produces and consumes an impressive amount of data. Companies such as Palantir, are assuming a relevant importance, law enforcement intelligence agencies and private businesses consider these platforms an indispensable tool for data mining.

Typically an agency such as the NSA and the CIA have access to thousands of different databases, billions of records that needs to be correlated quickly. Another factor to consider is related to the different nature of information, video, audio tracks, financial records, DNA samples, intelligence reports and much more, in many cases available in different languages and formats. Intelligence agencies have a further need, they in fact monitor and mine big data composed as a result of an intense activity of communication surveillance. The Prism case is just the tip of the iceberg: every conversation, email and phone record are tracked and it's necessary to design tools to query acquired data.

The first question that I have in mind is: Why outsource these intelligence activities? Despite the outsourcing of intelligence activities which is a common practice according to part of the security community, many governments prefer to build and fund independent private companies that could be used for specific analysis, and also to probe the global cyber security market.

These companies are free to operate on a large spectrum, they are usually characterized by a rapid growth, often sustained by government funding. Palantir seems to have the exact profile for a government "satellite" business. It has within its customers practically the principal government agencies such as the CIA, NSA, Army, U.S. Defense Dept., FBI, Marines, Air Force and law enforcement.

The Palantir Company has literally exploded, it has reached nearly $250 million in sales this year and the trend in constant growth. The growth is visible also from company headquarters, Palantir started with Facebook's former headquarters at University Avenue in Palo Alto, during the last years the company needed more space and infrastructures to host its engineers so Palantir has expanded to four other nearby buildings.

The government face of Palantir

According security specialists the core business for Palantir is represented by intelligence analysis conducted for the US Government, the conviction is also associated with common thought that the company in reality is a rib of US Intelligence. The majority of activities conducted by Palantir for the US government are classified, despite in the past there has been a leak of information.

The use of Palantir solutions became famous in Afghanistan where U.S. Special Operations Forces adopted them to plan offensive operations in risky areas. Thanks to the Palantir platform US Forces had at their disposal a wealth of information crucial to the success of military operations. For each location there were archived maps, information on past plots that occurred in the area, evidences collected during past attacks, details on the locations of all reported shooting skirmishes, IED information on the explosive devices used and links to similar incidents that occurred in the area. The artificial intelligence engine allowed the US Forces in the area and prediction for the area in where the likelihood of attacks was highest.

Palantir platform is considered a powerful instrument able to increase the strategic capabilities of militias in a specific area.

Palantir was also used to dismantle a network of subversives and terrorists in Afghanistan, Pakistan and Syria. In 2010 security experts in Canada used company solutions to uncover an espionage operation dubbed "Shadow Network" that had broken into various government offices and Ministries.

Palantir and PRISM revelation

Palantir Technologies is considered the principal company behind the surveillance program known as PRISM and recently revealed by Edward Snowden. Of course program such as PRISM involves various technologies and different solution providers, but many specialists believe that Palantir is the driver company for the program, its services and solutions are able in to integrate heterogeneous datasets composed by other government subcontractors.

Palantir commercialized a product named PRISM, exactly as the US surveillance program, that "that lets you quickly integrate external databases into Palantir. Specifically, it lets you build high-performance Data Engine based providers without writing any code."

[caption id="" align="alignnone" width="625"] Click to Enlarge[/caption]

Figure 5 - Palantir Prism

Legal representatives of the company denied any involvement in the US Government surveillance program sustaining that the platform has totally different scope.

"Palantir's Prism platform is completely unrelated to any US government program of the same name. Prism is Palantir's name for a data integration technology used in the Palantir Metropolis platform (formerly branded as Palantir Finance). This software has been licensed to banks and hedge funds for quantitative analysis and research,"

Y Combinator partner Garry Tan commented Palantir's disclaimers with following tweet:

Despite this, it is still not clear how PRISM works, the slides exposed by Snowden aren't accurate enough to explain how PRISM platform has access to the data of the major IT companies of the country. Some security experts sustain that the companies who collaborated with the US Government provided direct access to their servers, others speculate the companies feed a sort of Dropbox-like system that is accessed by PRISM for surveillance purposes.

In both cases the use of a software such as Palantir PRISM could represent the optimal solution, it is a powerful tool used to mine different datasets composed by an impressive amount of data constantly updated, impossible to do the same task with only human intervention.

In reality the complex machine that in a simplistic way was dubbed PRISM is probably fueled by much more information from various sources, not only IT giants are involved, Digital Net Agency Chief Strategy Officer Skip Graham believes the advertising industry is complicit inducing internet users to provide personal information online.

Personally I believe that the system Palantir PRISM isn't the same solution used by the US Government to analyze real time billions of records obtained by intercepting any kind of information, at the same time I think that Palantir is one of most valid companies in the US able to provide solutions necessary to conduct so complex analysis.

It's not a mystery that when using Palantir technology, US intelligence agencies and law enforcement can instantly profile any US citizen, its platform is able to query a huge quantity of data sources including surveillance video data collected everywhere in the country.

Christopher Soghoian, a popular privacy researcher and activist, is worried by continuous violation of citizen's privacy to fuel data analysis platforms.

"I don't think Palantir the firm is evil," "I think their clients could be using it for evil things." he said.

Soghoian remarked his position evidencing the nearness of Palantir's senior legal adviser, Bryan Cunningham, to the Bush Administration of which he shared the political choice on the need of domestic surveillance for Homeland Security monitoring domestic communication without search warrants. On the other hand, Karp replied sustaining that his company is compliant to a strict internal policy for privacy respect and it has developed sophisticated privacy protection technology on the market.

Palantir solutions includes sophisticated functions to create audit trails on its solutions, specifically the company sustains that it is able to provide detailed information on the use of information gathered, and on the personnel who accessed/requested it for its analysis. Of course I believe that those systems can be manipulated by Palantir to respond to the government agencies needs and requests, the declaration of the existence of the audit module is far from preserving user's privacy.

On the one hand there is the Government need to prevent terrorist attacks like September 11, but on the other hand, there is the urgency to prevent similar systems that could pose a threat to the privacy of citizens because their abuse may be a serious threat to freedom of expression.

Conclusions

The Palantir Technologies will continue its inexorable growth extending its solutions to other industry such as Health and Insurance. Data Analysis solutions are even more requested in every context due to the last amount of information to analyze and integrate from heterogeneous data sources.

The solutions by Palantir are efficient, easy to use and could be easily adapted for various contexts, from cybercrime prevention to Finance analysis. Palantir CEO Karp sustains that the company has largely invested in health care, retail, insurance, and biotech. Palantir's technology for example can be a precious ally to discover health insurance scams, the same solution could be used to develop models to track sequences of procedures and clinical data metrics against diagnoses that classify care.

"Analyze the root causes of readmission and post-surgical complications, and model medical outcomes and associated costs for targeted, preventative interventions. Develop and deploy best practice models across multiple facilities and providers."

Theoretically these solutions have no limits of use neither, their capillary diffusion could provide enormous benefits to the customers, and could also contribute to the growth of a global extended control machine, able to provide for the US Government unlimited access to every sector in which Palantir solutions are deployed … so Big Brother could take a new appearance.

It must be said, however, that the management of Palantir and his staff are an example of quality and vision apart from their possible relationships with government circles.

I would have no doubt in choosing Palantir solution, if I were asked.

References

http://securityaffairs.co/wordpress/15689/intelligence/stellar-wind-prism-evilolive-shelltrumpet-surveillance.html

http://securityaffairs.co/wordpress/15026/intelligence/us-top-secret-massive-surveillance-program-prism-revealed.html

http://www.businessinsider.com/palantir-wikileaks-apology-2011-2

http://www.youtube.com/watch?v=jZug8kV28RU

http://www.palantir.com/solutions/cyber/

https://analyzethe.us

http://securityaffairs.co/wordpress/2946/intelligence/stratfor-on-iran-palantir-and-goldman-sachs.html

http://securityaffairs.co/wordpress/2927/intelligence/wikileaks-anonymous-new-alliance-against-dirty-affairs.html

http://www.businessweek.com/magazine/palantir-the-vanguard-of-cyberterror-security-11222011.html

http://securityaffairs.co/wordpress/15026/intelligence/us-top-secret-massive-surveillance-program-prism-revealed.html

Hands-on threat intel training

Learn how to collect, analyze and act on cyber threat intelligence with expert instruction and hands-on exercises in Infosec Skills.

Get Started

http://www.businessweek.com/magazine/palantir-the-vanguard-of-cyberterror-security-11222011.html#p3