Last Updated: February 25, 2016
· balaclark

Read large text files in nodejs

I've been playing around with NodeJS recently and got the following error when trying to read a very large text file:

FATAL ERROR: CALL_AND_RETRY_0 Allocation failed - process out of memory

The following solution allows you to stream a file instead of reading it all into memory:

var fs = require('fs');
var readline = require('readline');
var stream = require('stream');

var instream = fs.createReadStream('your/file');
var outstream = new stream;
var rl = readline.createInterface(instream, outstream);

rl.on('line', function(line) {
  // process line here

rl.on('close', function() {
  // do something on finish here

11 Responses
Add your response

I tried this solution, but I still have the same exception on large files (log files ~200Mb)

The only way i found is to use line-by-line plugin :

over 1 year ago ·

snake51's setup must have been faulty because this code works great. I read a 7.5GB file containing over 800 million lines.

over 1 year ago ·

Nice and clean solution without requiring any third party modules.

over 1 year ago ·

For some reason, it works up until the last line.

over 1 year ago ·

I'm having the same issue as jake_nerdist. Thinking I'll just append a \n to the file before I start processing it.

over 1 year ago ·

how do you limit the number of lines to look at. I'm just trying to debug the workflow and don't want to read the whole file every time.

over 1 year ago ·

also how do you change the encoding to utf-8?

over 1 year ago ·

Is there a way to read with a different separator (',' or '#' instead of '\n')? official documentation do not specify a parameter for that.

over 1 year ago ·

I'm trying to build a Straming for big audio files and serve to the browser,

I'm try with this part of code:

res.setHeader("content-type", "audio/mpeg");
var stream = fs.createReadStream('record.mp3', { bufferSize: 64 * 1024 });

But the problem with this is that when I put the URL into the audio tag (for example).

source src="" />

Node.js read the entire file and send the complete file to the browser.

I'm working with audio files that they size is biggest that 500 mb.


over 1 year ago ·

Great article. I haven't been able to find something so clear on this type of issue. Great job.

over 1 year ago ·

getting errors in for extended ascii chars °C. Do you have any clue why this happens. fs.createReadStream seems to work fine...but createInterface seems to give problems.

over 1 year ago ·