Quickly creating large files

I’m surprised how many people never think to do this…. but it makes it quite easy.

If you need a large text file, perhaps with 1,000s of lines (or even bigger) – just use doubling to your advantage! For example, create 10 lines. Then use vi (or other editor) to copy the entire file to itself – now 20 lines. If you remember how a geometric progression goes, you’ll have your 1,000s of lines rather fast:

1. 10 lines…
2. 20 lines…
3. 40 lines…
4. 80 lines…
5. 160 lines…
6. 320 lines…
7. 640 lines…
8. 1280 lines…
9. 2560 lines…
10. 10240 lines…

Ten steps and we’re at 10,000+ lines. In the right editor (vi, emacs, etc.) this could be a macro for even faster doubling. This doubling could also be used at the command line:

cat file.txt >> file.txt

Combined with shell history, that should double nicely – though using an editor would be more efficient (fewer disk reads and writes).

When writing code, often programmers will want to set things off with a line of asterisks, hash marks, dashes, or equals signs. Since I use vi, I like to type in five characters, then copy those five into 10, then copy those 10 and place the result three times. There you have 40 characters just like that.

If only a certain number of characters is needed, use dd:

dd if=/dev/random of=myfile.dat bs=1024 count=10

With this command (and bs=1024), the count is in kilobytes. Thus, the example will create a 10K file. Using the Korn shell, one can use this command to get megabytes:

dd if=/dev/random of=myfile.dat bs=$(( 1024 * 1024 )) count=100

This command will create a 100M file (since bs=1048576 and count=100).

If you want files filled with nulls, just substitute /dev/null for /dev/random in the previous commands.

You could use a word dictionary for words, or a Markov chain for pseudo-prose. In any case, if you only want a certain size, do this:

~/bin/datasource | dd if=- of=myfile.dat bs=$(( 1024 * 1024 )) count=100

This will give you a 100M file of whatever it is your datasource is pumping out.