Quickly creating large files
23 November 2007
I’m surprised how many people never think to do this…. but it makes it quite easy.
If you need a large text file, perhaps with 1,000s of lines (or even bigger) - just use doubling to your advantage! For example, create 10 lines. Then use vi (or other editor) to copy the entire file to itself - now 20 lines. If you remember how a geometric progression goes, you’ll have your 1,000s of lines rather fast:
- 10 lines…
- 20 lines…
- 40 lines…
- 80 lines…
- 160 lines…
- 320 lines…
- 640 lines…
- 1280 lines…
- 2560 lines…
- 10240 lines…
Ten steps and we’re at 10,000+ lines. In the right editor (vi, emacs, etc.) this could be a macro for even faster doubling. This doubling could also be used at the command line:
cat file.txt >> file.txt
Combined with shell history, that should double nicely - though using an editor would be more efficient (fewer disk reads and writes).
When writing code, often programmers will want to set things off with a line of asterisks, hash marks, dashes, or equals signs. Since I use vi, I like to type in five characters, then copy those five into 10, then copy those 10 and place the result three times. There you have 40 characters just like that.
If only a certain number of characters is needed, use dd:
dd if=/dev/random of=myfile.dat bs=1024 count=10
With this command (and bs=1024), the count is in kilobytes. Thus, the example will create a 10K file. Using the Korn shell, one can use this command to get megabytes:
dd if=/dev/random of=myfile.dat bs=$(( 1024 * 1024 )) count=100
This command will create a 100M file (since bs=1048576 and count=100).
If you want files filled with nulls, just substitute /dev/null for /dev/random in the previous commands.
You could use a word dictionary for words, or a Markov chain for pseudo-prose. In any case, if you only want a certain size, do this:
~/bin/datasource | dd if=- of=myfile.dat bs=$(( 1024 * 1024 )) count=100
This will give you a 100M file of whatever it is your datasource is pumping out.
Entry Filed under: Linux, Mind Hacks, Programming, Tips, UNIX. Tags: dd, doubling, text.
10 Comments Add your own
Leave a Comment
Some HTML allowed:
<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>
Trackback this post | Subscribe to the comments via RSS Feed
1.
esofthub | 23 December 2007 at 9:05 am
Nice tips.
I’m fairly certain you already know this but I would also suggest using the mkfile command
# mkfile 1m 1MEGABYTE
2.
ddouthitt | 24 December 2007 at 12:14 pm
The mkfile command is a Linux-specific command; it is not on HP-UX 11i (I checked) and it certain isn’t on OpenVMS 8.3…. it’s also not on my FreeBSD 6.2 system either.
These methods will work anywhere.
Even so, it is a good point - just not portable.
3.
esofhub | 25 December 2007 at 5:31 am
It’s part of the Solaris OS but you do make a good point regarding its portability.
Keep posting.
4.
Lou | 22 January 2008 at 10:37 am
Annoyingly
cat file.txt >> file.txt
Doesn’t work on my cygwin setup - it’s clever enough to realise that input and ouput are the same.
The:
dd if=/dev/random of=myfile.dat bs=$(( 1024 * 1024 )) count=100
Method worked great though, I created a 600Gb file in (relatively speaking) no time!
Thanks!
5.
thetechnologyteacher | 28 February 2008 at 2:22 pm
Thanks for this write up.
One small change: “(since bs=104856 and count=100).” should read 1048576
Cheers
6.
ddouthitt | 28 February 2008 at 6:25 pm
Thanks for the comment - and the fix. Done.
7.
natophonic | 12 March 2008 at 9:28 am
Generating the file was creeping along verrry slowly, but I found that switching to the non-blocking /dev/urandom
dd if=/dev/urandom of=myfile.dat bs=$(( 1024 * 1024 )) count=100
got me a perhaps slightly less random file in seconds.
http://en.wikipedia.org/wiki/Urandom
8.
ddouthitt | 12 March 2008 at 4:00 pm
@natophonic:
As you mentioned, /dev/urandom is nonblocking. The article you pointed to was the article on /dev/random. The article makes for very interesting reading.
Under Linux, /dev/random is designed to be free of cryptographic export controls (by not using ciphers to generate randomness), and may block at times to receive enough entropy from the system.
In contrast, /dev/urandom has a feedback loop where it feeds generated entropy back into itself, and it will not block.
Under FreeBSD, two things are notable: 1) the Linux-style randomness generator was replaced by something called the Yarrow algorithm; and 2) /dev/urandom is linked to /dev/random - put another way, neither /dev/random nor /dev/urandom will block.
Both /dev/random and /dev/urandom are available all over: including Solaris, MacOS X, FreeBSD, HP-UX, Tru64, and AIX to name a few.
In summary, if you require the best random number generator possible - use /dev/random. If you require nonblocking I/O or faster number generation, use /dev/urandom. Thirdly, if you’re using FreeBSD it doesn’t matter which you use.
The article is very interesting: if you are interested in mathematics, you simply must read it.
9.
Upayavira | 6 April 2008 at 12:46 pm
I think you mean /dev/zero not /dev/null. At least, the former worked for me and the latter didn’t.
10.
ddouthitt | 14 April 2008 at 9:00 am
@Upayavira: I do believe you are right. Thanks for pointing out the error!