The collection of data goes all the way back to ancient times when the Library of Alexandria in Egypt served as the world’s hub for every papyrus scroll—estimated up to 700,000 or the equivalent of 100,000 printed books.

Some of the greatest scholars, scientists and mathematicians of the age made earth-shattering discoveries after studying and exchanging ideas at that very spot.

  • Aristarchus stated that the earth revolved around the sun, 1,800 years before Copernicus.
  • Euclid wrote the elements of geometry.
  • Herophylus identified the brain as the organ that controlled the body.
  • Callimachus, the poet, became the father of library science after organizing the scrolls by subject and author.

Without those great thinkers making sense of all that information, the heaps of scrolls wouldn’t have served any purpose other than as historical records of the times. And, they came to those conclusions by tediously combing through the rawest of data … word by word.

Today, high-performance computers that process at lightning-fast speeds and software that lends structure and analysis to the zettabyte of data we have amassed are giving businesses that use Big Data a huge advantage.

That’s especially true in the financial industry. According to a Capgemini Consulting report, “Sixty percent of financial institutions in North America believe that big data analytics offers a significant competitive advantage, and 90 percent think that successful big data initiatives will define the winners in the future.”

That’s quite the testimonial … as well it should be.

Big Data can be used by banks and investment firms to identify and study patterns of client behavior to ultimately create just the right products and services for them and boost bottom lines. However, in this day and age, it’s the ability of Big Data to prevent and spot fraud and identify risks that makes it a requirement, not a luxury. After all, fraudulent activity costs businesses a staggering $600 billion per year in the U.S. alone, according to the Commission on the Theft of American Intellectual Property.

So, how can Big Data help? By properly gathering and analyzing the right data, it can uncover what’s hidden and suspicious, reduce the operational costs of fraud investigation, anticipate and prevent fraud, identify and stop rogue traders and protect brands. It can also streamline regulatory reporting and compliance (for instance, for HIPPA), but that’s for another article.

The sheer amount of data available today is both a blessing and a curse. Because traditional customer records only touch the surface, companies must compile and analyze data from a myriad of sources and types—and do so all at once.

It’s no easy task. Think financial transaction data, geo-location data from mobile devices, merchant data, and authorization and submission data. Throw in data from lots of social media channels and your operating system’s mainframe, and it adds up to a significant challenge.

With the right tools, though, this potpourri of data can yield insights and answers never before possible.

Credit Suisse, for instance, launched a compliance joint venture last year to help catch dishonest employees before they did any harm. The bank’s head of compliance told Bloomberg that the company had started working with data analytics firm Palantir after another Swiss bank, UBS, suffered a $2.3 billion loss in 2011 from unauthorized trading by a London-based employee.

Banking behemoth JPMorgan also uses analytics software developed by Palantir to keep track of employee communications to identify possible internal fraud.

Palantir, a Palo Alto, CA-based big data company, is known as one of the best kept secrets by Silicon Valley, fast becoming “the epicenter of big data analytics.”  The company first gained wide popularity for developing cutting-edge intelligence analytics solutions for U.S intelligence community (FBI, CIA and NSA), then extended the analytics solutions to corporate America.

While Palantir is one of the best-kept secrets, there’s nothing secret about New York-based Cloudera. Known for changing the economics of big data analysis, the company provides the popular Hadoop-based big data platform for leveraging analytics. According to Gartner, 60 percent of top user behavior analytic vendors run on Cloudera.

Other companies in this space include:

  • Amazon Elastic MapReduce
  • Hortonworks Data Platform (HDP)
  • MapR Hadoop Distribution
  • IBM Open Platform
  • Microsoft Azure’s HDInsight -Cloud based Hadoop Distribution
  • Pivotal Big Data Suite
  • Datameer Professional
  • Datastax Enterprise Analytics

Judging from spending trends, the industry is taking Big Data very seriously. According to advisory and market intelligence company International Data Corporation (IDC), the global financial services industry spent more than 25 percent of its total IT budget in 2015 on mobility, cloud, and big data & analytics. In other words, that is $114 billion worldwide on these three technologies alone out of a combined IT expenditure of $455 billion.

IDC expects the big data and analytics, mobility and cloud to take up almost 30 percent of the financial industry’s IT budgets globally by 2019.

If your company plans to allocate money to big data analytics, first decide whether to keep the project in-house or to turn to a third party. Not that all firms will require numerous data scientists and engineers on staff; but to give an idea of cost, a team of 20 experts can run more than $4 million a year. That’s compared to $500,000 to $1 million a company offering SaaS (Software as a Service) might charge.

Also consider how quickly an in-house team (or compiling a new one) can complete a given project. Can you wait two to three years to implement a technology that is needed today? You run the risk of a solution becoming outdated before it’s even put in place. Again, by partnering with a third-party, a project can be completed in as little as 20 days.

So, begin the Big Data journey by answering the in-house vs. vendor question; and if the cheaper, more efficient way wins out, any or all of the leading companies referenced here can take it from there.

Karen Riccio

Karen Riccio

Financial Writer at Kurtosys
Karen Riccio has had a 20-year long career in journalism covering the financial and technology industries. She is currently a member of the Financial Braintrust Alliance for Newsmax and writes a weekly blog. Her clients have included Moneynews.com, InvestorPlace.com, ChangeWave, Business Financial Publishers, ETFtrends.com, Global Trends, and Wall Street Daily. Karen graduated from Cal Poly Technical University in San Luis Obispo, Calif., with a BS in Journalism and lives in Orange County, Calif.
Karen Riccio