Big Data Is Big

Huge information will turn into a key premise of rivalry, supporting new influxes of efficiency development, advancement, and shopper overflow—as long as the correct strategies and empowering agents are set up.

This has what has happened in the last few years in our world.

Data became Big

As everyone pointed out that with ubiquity of internet and internet connected devices, buttload of data gets generated. This is going to become astronomical in coming future as more and more sensors, people, and devices gets connected.

Now you have data. You can do quite a few things with large data to increase revenue, make service/product better, make forecast more accurately, convince investors/acquirers with facts, make and provide input to critical decision making. But to do all this you need data scientists.

Data became Open

Data is now more available than ever. If you ever wanted to see if your company’s name is referred by people with positive sentiment or negative without actually people filling the Google form or SurveyMonkey forms, you can always stream in Twitter data and do simple Natural Language Processing using Python (programming language)’s Natural Language Toolkit (NLTK). You will need a data scientist for this.

Twitter  is not the only open data source. There are valuable data on AWS & Public Data Sets. If you are a startup focused on Genomics, you’d probably prove that your flagship product works on 1000 Genomes Project.

Right Tools became Accessible

It seems a need to analyse data sets, usually large, leads into high demand of Data Scientists. But there are a couple of factors too. The accommodation of large set used to be hard. MySQL or traditional datastores have their limits. You tune them carefully, and be very careful what not to do to keep the database performant. With availability of robust tools like NoSQL databases and distributed computing, the general approach has become to throw everything in your NoSQL cluster and we may or may not use it to analyze some statistics.

The second half of the story is open sourced, big data processing technologies. They do the hard job of crunching number. They are faithfully used by the big companies and they are free.

Success Stories became Cliché

If you look for Big Data success stories, you will find many many companies used data science (analytics) to increase revenue, improve user base, increased user engagement (YouTube), innovated an existing process or simply raked dollars by providing big data analytics as service.

Hardware became cheap

Imagine 10 years back (2004). You have same amount of large data as today. Same amount of storage technology and processing power from software as today. Could you just bought 42 units of Dell PowerEdge R910 4U rack server on day 0 for some analytics that may or may not help you to improve 1% in revenue? No. But now you can just rent a couple hundred machine instances from any cloud service provider for an hour, do the analysis. Kill the servers. Your job is done in a couple hundred dollars. Compare that with seven thousands dollar for a single Dell machine.

So, enabling technology with cheap hardware availability caused many companies to try out data analytics for maximum gain from their business. So, many people hire data scientists to do that.

Basically, in today’s age, the following has happened:

Data Became Big: That means lot of data sources

Data Became Open: Twitter, government, open data and many more.

Right Tools became Accessible: Open Source reliable tools: Hadoop, NoSQL, Storm

Success Stories became cliche: Success stories, and high paying jobs.

Hardware became cheap: The cloud computing movement.

Future became data driven: With push from pro-data scientists, it seems data is the future.

And that is why Data and everything revolving around it is the next big thing.