• by iainmerrick on 5/24/2023, 9:14:30 AM

    Storing the errors alongside other transformed inputs is a really good idea. But manually patching up and/or deleting individual records is a really bad idea! How is that going to help your data integrity?

    Surely you want to fix your importer and re-run it. If there was a smart tool that was able to avoid duplicating work and only re-import the affected records, that'd be great, but that doesn't seem to be what's shown here.

  • by TheAlchemist on 5/24/2023, 11:56:33 AM

    Oh wow ! I was just trying to articulate exactly this kind of approach and looking for it. I would love to see a coherent approach to data errors and this seems like a step in the right direction.

    One question - the blog post covers basically debugging the ingestion of data part. My quite usual issue with older data is that at some point, you discover an issue with it (say it's slightly false, but not too much) - so you want to somehow let users know, or allow to select only the data without the issue (but still let them know how much of it they miss) - is this framework helpful in this situation ?

  • by rpxio on 5/24/2023, 1:41:18 AM

    It took me a minute to realize that this isn’t about the Zed text editor.

  • by ergl on 5/24/2023, 10:10:39 AM

    This article would improve by linking to what Zed is supposed to be: https://zed.brimdata.io/docs

  • by yyyk on 5/24/2023, 10:54:06 AM

    "Wouldn’t it be great if you could see errors in place instead of mysterious NULLs?"

    It's not a good ad when the error message is inadequate even in the supplied example and you need to hack around it.

  • by mkl95 on 5/24/2023, 8:46:51 AM

    Thought it was about Zed Shaw for a moment and got ready for something colorful

  • by jsunderland323 on 5/22/2023, 5:27:10 PM

    This is really cool. I’d imagine this helps with datasets skewed by errors and actually allows incident losses to be quantized in ways that are usually pretty cumbersome. I think SREs would love this

  • by ahefner on 5/24/2023, 2:11:08 AM

    Interesting. Seems like a hassle to store efficiently.

  • by rirze on 5/24/2023, 2:10:27 PM

    Unfortunate product name