by nine_k on 11/5/2019, 4:53:11 AM
by mamcx on 11/5/2019, 4:58:44 AM
You could get interested in:
by adouzzy on 11/5/2019, 4:50:32 AM
When I saw "large", I expected >10GB. And "big data" if it needs to be batch processed or distributed processed.
by paulryanrogers on 11/5/2019, 4:36:51 AM
No mention of vi or Sqlite? While I'm no vi expert it's a great tool for working with big files when you want to browse around without grep. And Sqlite is similarly ubiquitous and capable of crunching large files.
by dbt00 on 11/5/2019, 5:59:51 AM
by throwGuardian on 11/5/2019, 7:33:13 AM
> Excel for Mac performed well but it is a paid solution so I did not consider it viable for many developers
Because developers handling gigabyte size data, and wanting to reliably manipulate it in a GUI, cannot possibly be expected to pay/afford the $7/month to Microsoft.
That said, the recommended solution is probably the best option for developers, not bedside because it's free, but for the ability to run complex SQL statements, and visualize the results.
If I were to edit this article, that'd be my takeaway: use tool X for snappy vitalization of SQL queries, even on multi gigabyte sized CSVs
by lerigner on 11/5/2019, 8:15:37 AM
I wonder how well would Table Tool [1] perform with your large dataset? This is an open source CSV editor for Mac from the developer of Postico, my favorite PostgreSQL client for Mac [2]
by coverman on 11/5/2019, 4:48:18 AM
Python + Pandas
by kjpatel on 11/5/2019, 5:51:23 AM
I use XSV: A fast CSV command line toolkit written in Rust.
by whydoyoucare on 11/5/2019, 5:54:30 AM
I am not clear what "manipulate" means here -- what is the author trying to do with the comma separated values? FWIW, I can accomplish csv manipulation using a handful of Unix utilities: sed, awk, cut, and call it a day.
by 70jS8h5L on 11/5/2019, 6:07:20 AM
A fast viewer a friend of mine created to view large CSVs in a GUI - might be useful to someone.
by GiorgioG on 11/5/2019, 4:33:36 AM
Sublime Text and VSCode work fine with 1GB files in my experience.
by zmmmmm on 11/5/2019, 6:12:27 AM
There seem to be literally dozens of solutions to do read-only operations but very few to enable comfortable editing of the files in-place in a Unix / command line environment.
Seems like a real gap in the software ecosystem atm:
- fast
- no limit on file size
- spreadsheet style layout
- command line
- easily edit and update individual cell => save
by bArray on 11/5/2019, 5:44:33 AM
@alecdibble Could you try LibreOffice's Calc [1]? It's my daily driver in Linux, it mostly works well as an excel replacement but I'm interested to see how well it does on a Mac.
by entee on 11/5/2019, 4:43:31 AM
Was SequelPro trying impute any field types? That could have been causing the crash when faced with a large file.
by banku_brougham on 11/5/2019, 5:15:31 AM
tad is absolutely great for this, i tested it for the exact use case mentioned. I had to file a ticket and wait for him to add an export to csv function. tad is built over sqlite and can filter/sort/pivot/aggregate and export the result, which is all my business team partners ever need. can handle GB files, i didnt test with TB size.
https://www.tadviewer.com/, its a desktop app
edit: one note after seeing other comments based on cli (xsv, sed, awk, etc) — the OP’s use case is something that marketing/pm/business stakeholders can use. my favorite tools are cli-based, however this does not fly with business teams so forget that option.
by jwx48 on 11/5/2019, 9:18:17 PM
As a huge fan of OpenRefine, I want to say thanks for introducing me to Sequel Pro.
by _Codemonkeyism on 11/5/2019, 5:42:25 AM
Used VIM in the past to open several GB of SQL files to edit them without problems.
by thrax on 11/5/2019, 5:53:33 AM
Memory mapped file.. Intermediate binary representation..
by 1996 on 11/5/2019, 4:06:52 AM
Why do you want to do that?
Use cat, pipe, grep, awk. Problem solved.
by thrower123 on 11/5/2019, 5:01:44 AM
I'm always astounded that there doesn't seem to be a decent general purpose CSV editor/viewer application. Excel is atrocious - it's always dog slow, and it mangles any CSV I've ever opened by trying to interpret the data to format it "smartly".
Having to build a table in a database and import the CSV into that feels a bit like hitting a house fly with a sledgehammer, but it's the most effective way I've seen.
The counter-intuitive part is that a 100MB file is considered large on a machine with 8-16GB RAM.
It's definitely not the problem of the resources, but only of the architecture of applications.
(Indeed, the right application is a DBMS, not a spreadsheet.)