Pyspark Compare VMWare, WSL2 and Native Windows

Pyspark is a python API for Apache Spark. Spark is an execution framework that handles distributed workloads. Written in Scala, it uses in-memory caching and optimized execution, and support batch processing. Spark in turn uses Hadoop. Hadoop is a Java open source storage and processing framework, which has a distributed file system (HDFS), YARN (yet another resource negotiator), map reduce (parallel computing) and a common set of java libraries.

Jupyter notebooks are great for developing and running Python scripts. You can control the notebook from inside a web browser, it allows you to experimenting with a script, save it and share it. They also support visual outputs like graphs. As jupyter notebooks can use the Ipython kernel you can reset the python session and re-run all your code, all from within a web page.

I wanted to use Pyspark for processing some large parquet files, so first I wanted a way to see which method would be fastest to run under windows, either WSL2 with Ubuntu, VM ware with Ubuntu or native Windows.

Rather than ramble on about what I used to test the times, here are the results

All results from same PC, running same version of Windows 10 Pro

PC Specs

AMD Ryzen 5950x

64GB DDR 3200

PCIe Gen 4 NVMe SSD

RTX 3090

Tools and Installation

Spark 3.1.2

Python 3.9.5

OpenJDK 11.0.11

Guest OS

Ubuntu 20.4.02

Windows VMWare Host running Ubuntu guest Firefox 12.7 seconds

Windows WSL2 Host running Ubuntu external Firefox 12.7 seconds

Host OS

Windows 10 Pro Version 10.0.19043 Build 19043

Browsers

Windows 10 Native Firefox 15.79 seconds

Windows 10 Native Microsoft Edge 15.6 seconds

Summary

VMWare running on Windows 10 Pro with a guest Ubuntu (using the Linux version of the tools) seems to be around 25% faster.

Running normal native windows version of the tools, or via WSL2 are both 25% slower.

And so the best thing to use is a virtual machine if you can. The VMWare Player 16 is free, and use an Ubuntu 20.4 LTS to be the guest.

I ran many types of scripts and they all seem to point to VMWare being faster than windows.

Example script and Results

import pyspark.sql.functions as F
from pyspark.sql.functions import col, expr
from pyspark.context import SparkContext
from pyspark.sql import SparkSession #, SQLContext
from pyspark import SparkFiles
import numpy as np
import matplotlib.pyplot as plt
import pyspark

import time
start_time = time.time()

try:
sc
except NameError:
sc = SparkContext(‘local’)
spark = SparkSession(sc)

%matplotlib inline

filename = ‘Q:/Downloads/UchuuData/sim_tmp/halolist_z0p00_0.h5.0.parquet’
print(SparkFiles.get(filename))
halos = spark.read.parquet(SparkFiles.get(filename))
%time hosts = halos.where(col(‘pid’) == -1)

%time n_hosts = hosts.count()
print(‘Total # halos =’, n_hosts)

%time hosts.orderBy(‘Mvir’, ascending = False).select(‘Mvir’, ‘Vmax’, ‘Rvir’, ‘rs’).show(n=10)

%time quantiles = list(1-np.logspace(-6.5, 0, 100))

%time Mvir_quantiles = hosts.approxQuantile(‘Mvir’, quantiles, 0)

%time Mvir_quantiles = np.array(Mvir_quantiles)

%time N_vals = n_hosts + 1 — np.array(quantiles)*n_hosts

plt.plot(Mvir_quantiles, N_vals)
plt.xlabel(‘$M_{vir}$’)
plt.ylabel(‘N(> $M_{vir}$)’)
plt.loglog()
np.savetxt(‘halo_mass_fn.csv’, np.array([Mvir_quantiles, N_vals]), delimiter = ‘,’)

plt.show()

print(“ — — %s seconds — -” % (time.time() — start_time))

Native Windows 10 Pro with Firefox

--- 15.79399824142456 seconds ---

Native Windows 10 Pro with Microsoft Edge

--- 15.594499349594116 seconds ---

WSL 2 with Microsoft Edge running in Native Windows 10 Pro

--- 15.728499412536621 seconds ---

VMWare running in Windows 10 with a Ubuntu Host running Firefox

— — 12.69355845451355 seconds — -