As an alternative to (npm -g)'ing here some potentially useful coreutils one-liners I've been using for a similar purpose:

- Dump all .py files into out.txt (for copy/paste into a LLM)

> find . -name "*.py" -exec cat {} + > out.txt

- Sort all .py files by number of lines

> find . -name '*.py' -exec wc -l {} + | sort -n

Love this. I created (half-jokingly, but only half) the concept of a monofile (inspired by our monorepo) in our team. I have not managed to convince my colleagues to switch yet, but maybe this package can help. Unironically, I find that in larger python projects, combining various related sub 100 loc files into one big sub 1000 loc file can do magic to circular import errors and remove 100s of lines of import statements.

I've been dreaming of a tool which resembles this, at least in spirit.

I want to figure out how to structure a codebase such that a failing test can spit out a CID for that failure such that it can be remotely recreated (you'd have to be running ipfs so that the remote party can pull the content from you, or maybe you push it to some kind of hub before you share it).

It would be the files relevant to that failure--both code files and data files, stdin, env vars... a reproducible build of a test result.

It would be handy for reporting bugs or getting LLM help. The remote party could respond with a similar "try this" hash which the tooling would then understand how to apply (fetching the necessary bits from their machine, or the hub). Sort of like how Unison resolves functions by cryptographic hash, except this is a link to a function call, so it's got inputs and outputs too.

Of course that's a long way from vomiting everything into a text file, I need to establish functional dependency at as small a granularity as possible, but this feels like the first step on a path that eventually gets us there.

With fd: https://github.com/sharkdp/fd

    fd [filter options] -X cat

    fd -e js -X cat > combined.js

It'd be nice if something similar were available to traverse, say, directories of writings in Markdown, Word, LibreOffice, etc., and output a single text file so I have all my writings in one place. Plus allow plug-ins to extract from more exotic file types not originally included.

    shopt -s globstar
    tail -n+1 **/*.py | pbcopy

Isn't this a tar file?

I can imagine the token counts to be off the charts. How would an llm handle this input? Llm output quality already drops quite hard at a out 3000 tokens let alone 128k

Similar https://github.com/simonw/files-to-prompt

Similar project: https://github.com/yamadashy/repopack

I have a bash script which is very similar to this, except instead of dumping it all into one file, it opens all the matched files as tabs in Zed. Since Zed's AI features let you dump all, or a subset, of open tabs into context, this works great. It gives me a chance to curate the context a little more. And what I'm working on is probably already in an open tab anyway.

This made me laugh. Thanks!

Can you go 1 more step? Is there a way to not just dump someone's project into a plain text file, but sometime intelligently craft it into a ready to go prompt? I could use that!

Here's my user test: https://www.youtube.com/watch?v=sTPTJ4ladiI

Cool! I'd like to see an indication of the total number of tokens in the output, so I know right away on which LLM I can use this prompt or, if it's too large, I can relaunch the script by excluding other files to reduce the number of tokens in the output

One feature you could add is allowing the user to map changes in the concatenated file back to the original files. For example, if an LLM edits the concatenated file, I would want it to return the corresponding filenames and line numbers of the original files.

Really nice! I made a small cli tool that has an extra step of basically printing out a tree, so you can ask the ai what files you want to output:

https://github.com/markwylde/ai-toolkit

Why do we need modules at all? [1]

[1] https://erlang.org/pipermail/erlang-questions/2011-May/05876...

We use a C compiler for embedded systems that doesn't support link time optimizations (unless you pay for the pro version, that is). I have been thinking about some tool like this that merges all C source files for compilation.

This is really helpful. I immediately thought I’d be useful for sending off to ChatGPT and then saw that’s what it’s actually for. Thank you!

Surely with storage being pretty slow and everything it would be better to compress it into an archive with really basic compression?

Shouldn't this work?

find /path/to/directory -type f -exec cat {} + > output.txt

vim-ai basically supports this use case out of the box. All you need is your a index file listing all the files you want included, starting with

>>> include

This is probably very useful for use with LLM’s.

Love the name :D.

the .sick file extension is a nice touch ^^

Be careful with the name, McDonald’s might sue you for copyright infringement.

find ... | xargs head -n -0

The name links up nicely with AI enshittification. Although if you wanted to be pedantic, for that metaphor to work you'd really want to call it "gorge" or something more related to ingestion rather than vomiting. (I'm aware that a vomitorium was the exit from a Roman stadium, so it's not really about throwing up either).