SEC and Big Data

English: The U.S. Securities and Exchange Comm...

English: The U.S. Securities and Exchange Commission headquarters (Photo credit: Wikipedia)

In a recent speech to SIFMA’s annual Tech Conference, Gregg Berman, the SEC’s Associate Director of Analytics and Research, discussed how new technologies are giving the SEC a better understanding of today’s financial markets.

Berman highlighted the important role that the Market Information Data Analytics System, or MIDAS, plays in giving regulators and market observers greater access to information that undergirds financial transactions in our public markets. Berman said that MIDAS - launched in January 2013 - represents the SEC’s investment in cutting-edge technologies, ensuring that “the world’s largest and most-watched capital market system” remains “robust and reliable.”

MIDAS “is a powerful capability that will allow us to better determine the extent to which markets are resilient to human mistakes, or susceptible to such,” said Berman.

“[I]f we are to better understand the speed of the market, address concerns, and even consider actions, we must start by analyzing data that informs us on how market participants actually behave, and not limit ourselves to simply observing the results of their behavior in the aggregate,” he continued.

Berman said that SEC employees in the Office of Analytics and Research using MIDAS now can analyze and interpret a vast amount of market data stored entirely in the Cloud, allowing regulators to conduct real-time monitoring of market activities, forensic analysis of critical market events (such as the “Flash Crash”), and more substantive research that can better inform public policy decisions.

Every day, the SEC collects nearly “1 billion records from the prop feeds of 13 national equity exchanges.” Berman said these feeds provide SEC officials “with info on every trade and every displayed order, quote, modification, and cancellation on those venues...In fact, our new technology allows us to readily perform analyses of thousands of stocks over periods of 6 months or even a year, building up statistics on 100 billion records.”

Berman stressed that “with the use of technology comes the responsibility to have systems that are working as intended, and market activities, especially those to do with order generation, routing and flow.”