What does Skipped Stage means in Spark WebUI ?
Skipped Stages in Spark UI You must have come across various scenarios where you see a DAG like below, where you see a few stages shows greyed out with a text (skipped) after the stage...
Tagged: bigdata
Dataframe Operations in Spark using Scala
Dataframe in Apache Spark is a distributed collection of data, organized in the form of columns. Dataframes can be transformed into various forms using DSL operations defined in Dataframes API, and its various functions. In...
Top 15 HDFS Interview Questions
HDFS is the distributed file system used in Hadoop and helps to achieve the purpose of storing very larger files on a commodity Hardware. While working on Hadoop and BigData in general it is very...
What is RDD in Spark ? and Why do we need it ?
Resilient Distributed Datasets -RDDs in Spark Apcahe Spark has already taken over Hadoop (MapReduce) because of plenty of benefits it provides in terms of faster execution in iterative processing algorithms such as Machine learning. In...
What is Apache HCatalog ?
What is HCatalog ? Apache HCatalog is a Storage Management Layer for Hadoop that helps to users of different data processing tools in Hadoop ecosystem like Hive, Pig and MapReduce easily read and write data...
Hive : SORT BY vs ORDER BY vs DISTRIBUTE BY vs CLUSTER BY
In Apache Hive HQL, you can decide to order or sort your data differently based on ordering and distribution requirement. In this post we will look at how SORT BY, ORDER BY, DISTRIBUTE BY and...