IBM announced that it has committed to advance Apache Spark and calls it the most significant open source project of the next decade. The company will embed Spark into its Analytics and Commerce platforms and will offer Spark as a service on IBM Cloud.
There will be more than 3500 researches and developers from the company will work on Spark and projects that are related to Spark. IBM will also donate its machine learning technology SystemML to Spark which will be open-sourced and developed in collaboration with Databricks.

Furthermore, IBM’s Watson Health Cloud will be used with Spark to be able to work faster and to access new heath data analytics that will be used by medical providers and researchers.

Finally, IBM will educate more than one million data scientists and data engineers to be able to work with and on Spark. There have already been made extensive partnerships with AMPLab, DataCamp, MetiStream, Galvanize and Big Data University MOOC.
Spark has grown quickly in popularity among developers and data scientists as an essential platform for helping organizations more easily integrate Big Data into applications, and is quickly gaining momentum with IBM clients looking to transform business decision-making.

“Apache Spark is quickly maturing into a power tool for development of machine-learning analytic applications,” said Darwin Leung, director of informatics at Independence Blue Cross. “It allows our IBC researchers and academic partners to work together more seamlessly, which means we can get new claims and benefits apps up and out to customers much faster.”

The company has set ambitious goals for Spark. It expects analytics and data to become embedded in the fabric of business and society from popular apps, big industrial and research facilities and the Internet of Things (IoT). Since Spark improves performance on data dependent apps and simplifies the process of their development, IBM expects the platform to become even more popular and preferred.

This is why it plans to help train a whopping 1 million engineers and researchers, but even that figure might not be enough if the IoT development is as fast as expected over the next decade. So if you have a passion for the subject you may want to start honing your skills as it shapes up to be a big one in a few years time and given the complexity of it, those who have prepared early will have quite the head start.