Parquet-WASM: Rust-based WebAssembly bindings to read and write Parquet data

by kylebarron on 4/22/2024, 3:10:00 PM with 14 comments

  • by m_d_ on 4/23/2024, 12:55:21 AM

    I'd like to point out that fastparquet has been built for wasm (pydide/pyscript) for some time and works fine, producing pandas dataframes. Unfortunately, the thread/socket/async nature of fsspec means you have to get the files yourself into the "local filesystem" (meaning: the wasm sandbox). (I am the fastparquet author)

  • by jasonjmcghee on 4/22/2024, 5:10:11 PM

    Seeing as the popular alternative here would be DuckDB-WASM, which (last time I checked) is on the order of 50MB, this is comparatively super lightweight.

  • by leeoniya on 4/22/2024, 5:11:08 PM

    in my [albeit outdated] experience ArrowJS is quite a bit slower than using native JS types. i feel like crossing the WASM<>JS boundary is very expensive, especially for anything other than numbers/typed arrays.

    what are people's experiences with this?

  • by FridgeSeal on 4/22/2024, 11:46:25 PM

    @dang we have a mass spam incursion in this comment thread.

  • by rubenvanwyk on 4/22/2024, 7:24:17 PM

    Can this read and write Parquet files to S3-compatible storage?

  • by nickfs on 4/23/2024, 9:58:20 AM

    ;8y aiu;khjbvnvxzg;o9