You've Migrated a Hadoop Job from an On-Prem Cluster to Dataproc

question 133

Multiple Choice

You've migrated a Hadoop job from an on-prem cluster to dataproc and GCS. Your Spark job is a complicated analytical workload that consists of many shuffing operations and initial data are parquet files (on average 200-400 MB size each) . You see some degradation in performance after the migration to Dataproc, so you'd like to optimize for it. You need to keep in mind that your organization is very cost-sensitive, so you'd like to continue using Dataproc on preemptibles (with 2 non-preemptible workers only) for this workload. What should you do?

• A) Increase the size of your parquet files to ensure them to be 1 GB minimum.
• B) Switch to TFRecords formats (appr. 200MB per file) instead of parquet files.
• C) Switch from HDDs to SSDs, copy initial data from GCS to HDFS, run the Spark job and copy results back to GCS.
• D) Switch from HDDs to SSDs, override the preemptible VMs configuration to increase the boot disk size.

---

Definitions:

Compensatory Damages

Money awarded to a plaintiff as reimbursement for her or his losses; based on the amount of actual damage or harm to property, lost wages or profits, pain and suffering, medical expenses, disability, etc.

Breach of Contract

The failure to perform an obligation or promise specified in a contract, which can lead to legal action by the injured party for damages or enforcement.

Landlord's Consent

Landlord's consent is the approval a landlord must give for certain actions by the tenant, such as alterations to the property or subletting.

Majority Rule

A decision-making process in which the preferences or votes of more than half of the members of a group determine the outcome.