Porting flat file to Redis database

Download Redis

https://redis.io/download


Install  Redis

make

start Redis Server

~/redis-4.0.9/src$ ./redis-server

Sample Data

Redis has a specific format requirement. So, the flat file will need to be processed. 'sed' can easily altering this.

sed -e 's/^/SET /;s/://g' data_output.txt > data_output_processed.txt 

Using --pipe to import the data into Redis. I was getting errors. 

~/redis-4.0.9/src$ cat ~/datafile/data_output_processed.txt | ./redis-cli --pipe

ERR unknown command 'ET'

ERR unknown command 'ET'

ERR unknown command 'ET'

ERR unknown command 'ET'

ERR unknown command '2'

ERR unknown command '$4'

ERR wrong number of arguments for 'echo' command

ERR unknown command '$20'

ERR unknown command '�Bl���'

No replies for 30 seconds: exiting.

errors: 9498, replies: 9498

Using unix2dos to alter the data_output_processed.txt

unix2dos ~/datafile/data_output_processed.txt

unix2dos: converting file /home/xxxxxx/datafile/data_output_processed.txt to DOS format ...

Attempt to import the data with --pipe again and it works.

~/redis-4.0.9/src$ cat ~/datafile/data_output_processed.txt | ./redis-cli --pipe

All data transferred. Waiting for the last reply...

Last reply received from server.

errors: 0, replies: 9493

Query the data with redis-cli --scan

~/redis-4.0.9/src$ ./redis-cli --scan --pattern '*'

0x010000000154DAEB3E8A6BBC797E4AD2B0339F134B186E46376D0B000000000000

0x0100000001DC2D15A69F6BB33B246AEF40450751C2F6756AD28904000000000000

0x01000000013FE698DDD00CE16BFB0F2B875D29CF6B91EB677DB501000000000000

0x010000000157BDDF078834009C89D88E6282759DC45335B4707E07000000000000

0x020000003504000000000000

0x02000000F908000000000000

0x0100000001EB5F44D72343DF39680AAF77B2CED6586E6918C2090A000000000000

0x0100000003589EA787B46DA08EC8FC081678335C5D0081010EBA03000000000000

0x02000000BB02000000000000

0x010000000132BE343B94F860124DC4FEE278FDCBD38C102D88EC0D000000000000

0x01000000016CF94E97B229262AF23908265C2CF7E810686FA7D409000000000000

0x0100000001E6A7A1D47FF21B6321162AEA7C6CB457D5476BCAEF02000000000000

0x020000009B03000000000000

0x01000000012910543AF39ABA0CD09DBB2D50200B3E800A63D29705000000000000

0x010000000154DAEB3E8A6BBC797E4AD2B0339F134B186E46370D0B000000000000

0x02000000CB07000000000000

Query the data with redis-cli on single key.

~/redis-4.0.9/src$ ./redis-cli

127.0.0.1:6379> get 0x01000000012910543AF39ABA0CD09DBB2D50200B3E800A63D29705000000000000

"0x000000000000000000000000000000000000000000009B2550188E70759173F2"

127.0.0.1:6379>

That's it.