Clever Geek Handbook

Find and replace text in a large 47 GB file

I need to find and replace tasks on a fairly large file of about 47 GB in size.

Does anyone know how to do this? I've tried using services like TextCrawler, EditpadLite and more, but nothing supports this large file.

I assume it can be done via the command line.

Do you have any idea how this can be done?

+27


source to share


7 replies


Sed (stream editor for filtering and transforming text) is your friend.

sed -i 's/old text/new text/g' file


Sed performs transformations of text in one pass.

+25


source


I am using FART - Find and replace the text for Lionello Lunesu.

It works fine on Windows Seven x64.



You can find and replace text using the following command:

fart -c big_filename.txt "find_this_text" "replace_to_this"
+26


source


On Unix or Mac:

sed 's / oldstring / newstring / g' oldfile.txt> newfile.txt

fast and easy...

+5


source


If you are using a Unix-like system, you can use cat | sed do it

cat hosted_domains.txt | sed s/com/net/g

The example replaces com with the net in the list of domain names, and then you can pipe the output to a file.

0


source


For me, none of the tools suggested here work. Textcrawler ate all the computer memory, SED didn't work at all, Editpad complained about memory ...

Solution: Create your own script in python, perl, or even C ++.

Or use the PowerGrep tool, which is the easiest and fastest option.

I haven't tried farting, it's only command line and maybe not very friendly.
Some hex editors, like Ultraedit, also work well.

0


source


With sed (a stream editor for filtering and text trasformation), it does the transformation of the text in one pass.

sed -i 's/old/new/g' bigfile.txt
0


source


I used

sed 's/[nN]//g' oldfile.fasta > newfile.fasta

to replace all n instances in my 7Gb file.

If I omitted the aspect > newfile.fasta

it took an age as it scrolled the screen showing me every line of the file.

With > newfile

it launched it in seconds on ubuntu server

-1


source






More articles:


All Articles