Last Updated: February 25, 2016
·
2.321K
· datasaur

Generate Redis protocol from csv using Awk

An example approach for reformatting csv data into Redis protocol, to be bulk loaded with the console client. Data is binned into 5 minute samples for time series analysis.

awk -f redis-proto.awk data.csv | redis-cli --pipe

redis-proto.awk:

BEGIN { FS="," }
    {
        bin=sprintf("%s%04d", strftime("%Y%m%d",$2), $1 - ($1 % 5))
        fld=sprintf("%s:%s", $3, $4)
        rate=int(($5) / 1000)

        print "*4\r\n$7\r\nHINCRBY\r\n$12\r\n" bin "\r\n$" length(fld) "\r\n" fld "\r\n$" length(rate) "\r\n" rate "\r"
    }

2 Responses
Add your response

I think you're missing a \n after rate "\r", but other than that, this is pretty cool!

over 1 year ago ·

IIRC that was because the final \n is added to the line automatically by the print statement. Redis protocol does not like extra carriage returns. I could probably use a printf and be more explicit.

over 1 year ago ·