What is pETL

Even 70 lines of code is enough to give back to the community. pETL is a tiny gem that brings a better structure, reduce a code duplication and makes your ETL code familiar to the developers.

Features

1. Batching support
2. Automatic validity check
3. Logging of running times

Usage

require 'petl'

module ETL::Example
  extend Petl
  extend self

  def extract
    # Grab all data from source(s) here.
    # Preferably return an array of hashes.
  end

  def transform rows
    # Manipulate the data extracted by the previous extract method.
  end

  def load rows
    # Load the transformed data here into the destination(s).
  end

  def source_count
    # Count the number of records from your source(s).
  end

  def destination_count
    # Same as #source_count but with your destination(s).
  end
end

# Run it!
ETL::Example.perform

Code

https://github.com/zestfinance/petl

Presentation

https://speakerdeck.com/fsproru/petl-pretty-good-etl-framework