logo
MENUMENU
  • Services
        • Salesforce

        • Salesforce OverviewWe help our clients to make the most out of their technology investments.
          • Salesforce Overview
          • Systems Integration
          • Managed Services
          • Field Service Lightning
          • Salesforce CPQ
        • Verint

        • Verint OverviewOur services cover Implementations, Upgrades, Migrations, Reporting Solutions & Application Support.
          • Verint Overview
          • Verint Knowledge Management
          • Verint KM Integrations
          • Verint Omni-Channel
          • Verint Unified Desktop & Case Management
          • Verint Workforce Optimization
          • Verint Intelligent Virtual Assistant
          • Verint Robotic Process Automation
        • Artificial Intelligence

        • Artificial Intelligence
          • AI Overview
          • Data Engineering
          • Analytics/BI
          • Predictive Analytics
          • Computer Vision
          • Natural Language Processing (NLP)
          • ML Ops
        • Cloud & Mobile Apps

        • Cloud & Mobile App Development OverviewOur services cover Implementations, Upgrades, Migrations, Reporting Solutions & Application Support.
          • Cloud & Mobile App Development Services
        • Let’s Get
          Started.

        • Let’s talk about how SPAR Solutions
          can positively impact your business.

        • START TODAY
  • Company
    • About Us
    • Why Work At SPAR
  • Resources
    • Blog
    • Case Studies

GET STARTED 1-855-772-7765

GET IN TOUCH
  • Services
    • Salesforce Overview
      • Salesforce Overview
      • Systems Integration
      • Managed Services
      • Field Service Lightning
      • Salesforce CPQ
    • Verint Overview
      • Verint Overview
      • Verint Knowledge Management
      • Verint KM Integrations
      • Verint Omni-Channel
      • Verint Unified Desktop & Case Management
      • Verint Workforce Optimization
      • Verint Intelligent Virtual Assistant
      • Verint Robotic Process Automation
    • Artificial Intelligence
      • AI Overview
      • Data Engineering
      • Analytics/BI
      • Predictive Analytics
      • Computer Vision
      • Natural Language Processing (NLP)
      • ML Ops
    • Cloud & Mobile App Development Overview
      • Cloud & Mobile App Development Services
  • Company
    • About Us
    • Why Work At SPAR
  • Resources
    • Blog
    • Case Studies

Spark – setting Big Data on Fire!

Home > Big Data > Spark – setting Big Data on Fire!

Spark – setting Big Data on Fire!

October 4, 2015

Over the past decade, internet companies such as Google, Yahoo, Facebook and others have leveraged the Hadoop MapReduce platform effectively to process data that is truly large in scale. In addition, many enterprises with years of information gathered in siloes across multiple systems have also started using this platform to combine and process data at a scale beyond the realm of the imagination of most computer scientists and IT professionals.

In 2013, a new project donated to the Apache foundation started attracting the attention of Big Data practitioners. This was Apache Spark, and it has very quickly set the world of Big Data on fire! In contrast to MapReduce’s storage based two stage MapReduce paradigm, Spark provided developers the ability to access and process the same data in-memory multiple times, thus avoiding the costly expense of disk io. This has allowed Spark based applications to deliver anywhere from 10 to 100 times the performance gains as compared to a similar application using the MapReduce paradigm.

Though it works well on problems that Hadoop and MapReduce were being applied to, Spark was originally built to provide high performance data processing to support applications in the area of machine learning, graph processing and streaming analytics. With the wide adoption of Spark to replace MapReduce, people sometimes incorrectly assume that it was possibly developed to replace MapReduce or even Hadoop as a whole.

Luckily for the development community, the creators of Spark took a complementary approach with the Hadoop stack. Spark is completely built to coexist and work alongside existing investments in Hadoop including Hadoop’s data stores, file formats, data collection and management libraries, as well as it’s scheduling and resource management tools. That’s good news, especially for all the enterprises that have poured millions into applications and infrastructure using these technologies!

There are a few items to keep in mind as you look to implement new Big Data applications using Spark. Spark applications will clearly impose additional memory requirements on your cluster infrastructure to provide the performance improvements at scale. Spark may not always bring additional value for workloads that are not time sensitive, and better fit a batch processing model.

While not perfect, Hadoop MapReduce is definitely a more mature platform at this time. Spark will take some time to mature to the same level. On the flip side, it is highly likely that more resources will move to working on Spark, which will eventually lead to a slower cycle of updates and improvements in Hadoop’s core platform.

With so much promise from Spark, we can all now hope that the weather folks will finally be able to get their predictions right!

Swami Ganapathy
+ posts
  • Swami Ganapathy
    https://sparsolutions.com/author/sparwpadmin/
    SPAR’s Commitment to Diversity, Equality and Inclusion
  • Swami Ganapathy
    https://sparsolutions.com/author/sparwpadmin/
    A Complete Guide to Salesforce Field Service Lightning
  • Swami Ganapathy
    https://sparsolutions.com/author/sparwpadmin/
    Salesforce CPQ - Key features overview
  • Swami Ganapathy
    https://sparsolutions.com/author/sparwpadmin/
    Salesforce.com – Overcoming Apex code limits

Filed Under: Big Data

Search

Categories

  • Apex Testing
  • Big Data
  • Force.com Development
  • General
  • Healthcare
  • Highlighted
  • How to
  • Integration
  • Knowledge Management
  • Salesforce Field Service Lightning
  • Salesforce.com
  • SFDCHighlights
  • Uncategorized

Archives

  • September 2021
  • August 2021
  • June 2021
  • May 2021
  • March 2021
  • January 2021
  • November 2020
  • October 2020
  • September 2020
  • August 2020
  • March 2019
  • June 2017
  • December 2016
  • June 2016
  • December 2015
  • November 2015
  • October 2015
  • July 2015
  • February 2015
  • January 2015
  • October 2013
  • February 2013
  • January 2013
  • December 2012

Ready to Get Started?

Let’s talk about how SPAR Solutions can help you drive greater automation

Get in Touch

Ask us how to get started today.

SPAR Solutions logo

Terms of Service

OFFICE

375 Northridge Rd
Suite #400
Atlanta, GA 30350

CALL US

1-855-772-7765

Copyright © 2020, All Rights Reserved by SPAR Solutions. Digital Marketing by