The Photon project provides a high-performance operator framework that integrates with the DBR to enable warehouse-like performance on simple data lakes.
We tested Photon when it was starting out and it was a mixed bag for our use case. I don't recall details but it wasn't as much of a performance hit as we expected - especially when taking the higher cost into account. I wonder if it's because of all the translation that had to happen between native Spark and Photon Spark. And maybe it's much better now.
We tested Photon when it was starting out and it was a mixed bag for our use case. I don't recall details but it wasn't as much of a performance hit as we expected - especially when taking the higher cost into account. I wonder if it's because of all the translation that had to happen between native Spark and Photon Spark. And maybe it's much better now.