ohjerg
Last Updated: February 25, 2016
·
45.69K
· balaclark

Read large text files in nodejs

I've been playing around with NodeJS recently and got the following error when trying to read a very large text file:

FATAL ERROR: CALL_AND_RETRY_0 Allocation failed - process out of memory

The following solution allows you to stream a file instead of reading it all into memory:

var fs = require('fs');
var readline = require('readline');
var stream = require('stream');

var instream = fs.createReadStream('your/file');
var outstream = new stream;
var rl = readline.createInterface(instream, outstream);

rl.on('line', function(line) {
  // process line here
});

rl.on('close', function() {
  // do something on finish here
});

11 Responses
Add your response

10405

Hi,
I tried this solution, but I still have the same exception on large files (log files ~200Mb)

The only way i found is to use line-by-line plugin : https://npmjs.org/package/line-by-line

over 1 year ago ·
12615

snake51's setup must have been faulty because this code works great. I read a 7.5GB file containing over 800 million lines.

over 1 year ago ·
12682

Nice and clean solution without requiring any third party modules.

over 1 year ago ·
12778

For some reason, it works up until the last line.

over 1 year ago ·
13734

I'm having the same issue as jake_nerdist. Thinking I'll just append a \n to the file before I start processing it.

over 1 year ago ·
14024

how do you limit the number of lines to look at. I'm just trying to debug the workflow and don't want to read the whole file every time.

over 1 year ago ·
14025

also how do you change the encoding to utf-8?

over 1 year ago ·
15482

Is there a way to read with a different separator (',' or '#' instead of '\n')? official documentation do not specify a parameter for that.

over 1 year ago ·
17343

I'm trying to build a Straming for big audio files and serve to the browser,

I'm try with this part of code:

res.setHeader("content-type", "audio/mpeg");
var stream = fs.createReadStream('record.mp3', { bufferSize: 64 * 1024 });
stream.pipe(res);

But the problem with this is that when I put the URL into the audio tag (for example).

audio>
source src="http://127.0.0.1:8080/record.mp3" />
/audio>

Node.js read the entire file and send the complete file to the browser.

I'm working with audio files that they size is biggest that 500 mb.

Thanks.

over 1 year ago ·
18224

Great article. I haven't been able to find something so clear on this type of issue. Great job.

over 1 year ago ·
20880

getting errors in for extended ascii chars °C. Do you have any clue why this happens. fs.createReadStream seems to work fine...but createInterface seems to give problems.

over 1 year ago ·