Quiz Databricks-Associate-Developer-Apache-Spark

Stages with narrow dependencies can be grouped into one task. Wrong, tasks with narrow dependencies can be grouped into one stage. Tasks with wide dependencies can be grouped into one stage. Wrong, since a wide transformation causes a shuffle which always marks the boundary of a stage. So, you cannot bundle multiple tasks that have wide dependencies into a stage. ache-Spark-3.0.html Tasks are the second-smallest element in the execution hierarchy. No, they are the smallest element in the execution hierarchy. Within one task, the slots are the unit of work done for each partition of the data. No, tasks are the unit of work done per partition. Slots help Spark parallelize work. An executor can have multiple slots which enable it to process multiple tasks in parallel.

The cluster manager allocates resources to Spark applications and maintains the executor processes in client mode. Correct. In cluster mode, the cluster manager is located on a node other than the client machine. From there it starts and ends executor processes on the cluster nodes as required by the Spark application running on the Spark driver. The cluster manager allocates resources to Spark applications and maintains the executor processes in remote mode. Wrong, there is no "remote" execution mode in Spark. Available execution modes are local, client, and cluster. The cluster manager allocates resources to the DataFrame manager Wrong, there is no "DataFrame manager" in Spark. The cluster manager schedules tasks on the cluster in client mode. No, in client mode, the Spark driver schedules tasks on the cluster – not the cluster manager. The cluster manager schedules tasks on the cluster in local mode. Wrong: In local mode, there is no "cluster". The Spark application is running on a single machine, not on a cluster of machines.

Dynamic partition pruning reoptimizes query plans based on runtime statistics collected during query execution. No – this is what adaptive query execution does, but not dynamic partition pruning. Dynamic partition pruning concatenates columns of similar data types to optimize join performance. Wrong, this answer does not make sense, especially related to dynamic partition pruning. Dynamic partition pruning reoptimizes physical plans based on data types and broadcast variables. It is true that dynamic partition pruning works in joins using broadcast variables. This actually happens in both the logical optimization and the physical planning stage. However, data types do not play a role for the reoptimization. Dynamic partition pruning performs wide transformations on disk instead of in memory. This answer does not make sense. Dynamic partition pruning is meant to accelerate Spark – performing any transformation involving disk instead of memory resources would decelerate Spark and certainly achieve the opposite effect of what dynamic partition pruning is intended for.

Spark achieves great performance by storing data in the DAG format, whereas Hadoop can only use ache-Spark-3.0.html parquet files. Wrong, there is no "DAG format". DAG stands for "directed acyclic graph". The DAG is a means of representing computational steps in Spark. However, it is true that Hadoop does not use a DAG. The introduction of the DAG in Spark was a result of the limitation of Hadoop's map reduce framework in which data had to be written to and read from disk continuously. Graph DAG in Apache Spark - DataFlair Spark achieves great performance by storing data in the HDFS format, whereas Hadoop can only use parquet files. No. Spark can certainly store data in HDFS (as well as other formats), but this is not a key performance advantage over Hadoop. Hadoop can use multiple file formats, not only parquet. Spark achieves higher resiliency for queries since, different from Hadoop, it can be deployed on Kubernetes. No, resiliency is not asked for in the question. The Question: is about performance improvements. Both Hadoop and Spark can be deployed on Kubernetes. Spark achieves performance gains for developers by extending Hadoop's DataFrames with a user- friendly API. No. DataFrames are a concept in Spark, but not in Hadoop.

The hierarchy is, from top to bottom: Job, Stage, Task. Executors and slots facilitate the execution of tasks, but they are not directly part of the hierarchy. Executors are launched by the driver on worker nodes for the purpose of running a specific Spark application. Slots help Spark parallelize work. An executor can have multiple slots which enable it to process multiple tasks in parallel.

Manually persisting RDDs in Spark prevents them from being garbage collected. This statement is incorrect, and thus the correct answer to the question. Spark's garbage collector will remove even persisted objects, albeit in an "LRU" fashion. LRU stands for least recently used. So, during a garbage collection run, the objects that were used the longest time ago will be garbage collected first. See the linked StackOverflow post below for more information. Serialized caching is a strategy to increase the performance of garbage collection. This statement is correct. The more Java objects Spark needs to collect during garbage collection, the longer it takes. Storing a collection of many Java objects, such as a DataFrame with a complex schema, through serialization as a single byte array thus increases performance. This means that garbage collection takes less time on a serialized DataFrame than an unserialized DataFrame. Optimizing garbage collection performance in Spark may limit caching ability. This statement is correct. A full garbage collection run slows down a Spark application. When taking about "tuning" garbage collection, we mean reducing the amount or duration of these slowdowns. A full garbage collection run is triggered when the Old generation of the Java heap space is almost full. (If you are unfamiliar with this concept, check out the link to the Garbage Collection Tuning docs below.) Thus, one measure to avoid triggering a garbage collection run is to prevent the Old generation share of the heap space to be almost full. To achieve this, one may decrease its size. Objects with sizes greater than the Old generation space will then be discarded instead of cached (stored) in the space and helping it to be "almost full". This will decrease the number of full garbage collection runs, increasing overall performance. Inevitably, however, objects will need to be recomputed when they are needed. So, this mechanism only works when a Spark application needs to reuse cached data as little as possible. Garbage collection information can be accessed in the Spark UI's stage detail view. This statement is correct. The task table in the Spark UI's stage detail view has a "GC Time" column, indicating the garbage collection time needed per task. In Spark, using the G1 garbage collector is an alternative to using the default Parallel garbage collector. This statement is correct. The G1 garbage collector, also known as garbage first garbage collector, is an alternative to the default Parallel garbage collector. While the default Parallel garbage collector divides the heap into a few static regions, the G1 garbage collector divides the heap into many small regions that are created dynamically. The G1 garbage collector has certain advantages over the Parallel garbage collector which improve performance particularly for Spark workloads that require high throughput and low latency. The G1 garbage collector is not enabled by default, and you need to explicitly pass an argument to ache-Spark-3.0.html Spark to enable it. For more information about the two garbage collectors, check out the Databricks article linked below.

The Dataset API is available in Scala, but it is not available in Python. Correct. The Dataset API uses fixed typing and is typically used for object-oriented programming. It is available when Spark is used with the Scala programming language, but not for Python. In Python, you use the DataFrame API, which is based on the Dataset API. The Dataset API does not provide compile-time type safety. No – in fact, depending on the use case, the type safety that the Dataset API provides is an advantage. The Dataset API does not support unstructured data. Wrong, the Dataset API supports structured and unstructured data. In Python, the Dataset API's schema is constructed via type hints. No, this is not applicable since the Dataset API is not available in Python. In Python, the Dataset API mainly resembles Pandas' DataFrame API. The Dataset API does not exist in Python, only in Scala and Java.

In cluster mode, the driver runs on the master node, while in client mode, the driver runs on a virtual machine in the cloud. This is wrong, since execution modes do not specify whether workloads are run in the cloud or on- premise. In cluster mode, each node will launch its own executor, while in client mode, executors will exclusively run on the client machine. Wrong, since in both cases executors run on worker nodes. In cluster mode, the driver runs on the edge node, while the client mode runs the driver in a worker node. Wrong – in cluster mode, the driver runs on a worker node. In client mode, the driver runs on the client machine. In client mode, the cluster manager runs on the same host as the driver, while in cluster mode, the cluster manager runs on a separate node. No. In both modes, the cluster manager is typically on a separate node – not on the same host as the driver. It only runs on the same host as the driver in local execution mode. More info: Learning Spark, 2nd Edition, Chapter 1, and Spark: The Definitive Guide, Chapter 15. ()

An RDD consists of a single partition. Quite the opposite: Spark partitions RDDs and distributes the partitions across multiple nodes.

There is a place in the Spark UI that shows the property spark.executor.memory. Correct, you can see Spark properties such as spark.executor.memory in the Environment tab. Some of the tabs in the Spark UI are named Jobs, Stages, Storage, DAGs, Executors, and SQL. Wrong – Jobs, Stages, Storage, Executors, and SQL are all tabs in the Spark UI. DAGs can be inspected in the "Jobs" tab in the job details or in the Stages or SQL tab, but are not a separate tab. Via the Spark UI, workloads can be manually distributed across distributors. No, the Spark UI is meant for inspecting the inner workings of Spark which ultimately helps understand, debug, and optimize Spark transactions. Via the Spark UI, stage execution speed can be modified. No, see above. The Scheduler tab shows how jobs that are run in parallel by multiple users are distributed across the cluster. No, there is no Scheduler tab.