The Administrator Desk: August 2018

Wednesday, August 22, 2018

RocksDB: ldb scan

Make sure the database data is Block type, use hex

If database have sst file type the use RocksDB sst_dump too.

"Failed: Invalid argument: leveldb.BytewiseComparator: does not match existing comparator RocksKeyComparator"

https://github.com/facebook/rocksdb/wiki/Administration-and-Data-Access-Tool#ldb-tool

How to run RocksDB ldb tool

ldb tools reside in rocksdb directory. The location can be anywhere where the user built the rocksdb into. I built mine from home directory.

rocks@ubuntu:~/rocksdb$ pwd; ls -las ldb

/home/rocks/rocksdb

110692 -rwxrwxr-x 1 rocks rocks 113342424 Mar 15 15:44 ldb

./ldb --db=/home/rocks/builds/p2-blockchain/release/Blockchain/EthereumBlockchain/scripts/dbtest2 scan

Step 2b: If the data type is Block data type use '--hex' flag.

./ldb --db=/home/rocks/builds/p2-blockchain/release/Blockchain/EthereumBlockchain/scripts/dbtest2 scan --hex |less

./ldb --db=/home/rocks/builds/p2-blockchain/release/Blockchain/EthereumBlockchain/scripts/dbtest2 scan --hex > output.dat

rocks@ubuntu:~/rocksdb$ ./ldb --db=/home/rocks/builds/p2-blockchain/relwithdebug/Blockchain/EthereumBlockchain/scripts/dbtest1 scan --hex

....

0x02000000F000000000000000 : 0x01000A0021002B0001000422FDF068A5EF85F7C9C532E66D0EB76BF74D8C81CDD9ACE1AEC6C63CB6B5E5BF31

0x02000000F100000000000000 : 0x02001200150027002000470015005C002000014BD5F0EE173C81D42765154865EE69361B6AD189000000000000000000000000000000000000000000000025E46D8ED84BE7C00001FD7C9117BBEB649D9B37B3D3C9917F7C12F526B30000000000000000000000000000000000000000000000004563918244F40000

0x02000000F200000000000000 : 0x01000A0021002B00010004C3440C47368CA813312FDD0763931A0D300DF00DEAA5D9118FEDCB2E945A1C6F31

0x02000000F300000000000000 : 0x02001200150027002000470015005C0020000163B0495E672B1382B1AC7968B02E3B4234FE59DD000000000000000000000000000000000000000000000000001DD7C1681D80000132BE343B94F860124DC4FEE278FDCBD38C102D8800000000000000000000000000000000000000000000042776C17D8292FEC000

0x02000000F400000000000000 : 0x01000A0021002B00010004CE71039FAC556DB796A841D764C6B1E6BD47DF04D2DE82369F9904560019F87A31

0x02000000F500000000000000 : 0x02001200150027002000470015005C00200001A89AC93B23370472DAAC337E9AFDF642543F3E57FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF412E2FD9C261000000192C27672FE65E002159EC2597FCF8897ADBF5B290000000000000000000000000000000000000000000000008AC7230489E80000

0x02000000F600000000000000 : 0x01000A0021002B0001000414574CB3DF1F81C9EB82D4DDF98FE485908D882C597B4055E7C8052FDFC75F4B31

0x02000000F700000000000000 : 0x02001200150027002000470015005C002000010CDA12BF72D461BBC479EB92E6491D057E6B5AD1FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFAEAEE6E67FE2CD0000018DDA5E016E674683241BF671CCED51E7239EA2BC000000000000000000000000000000000000000000000000002386F26FC10000

0x02000000F800000000000000 : 0x01000A0021002B0001000411106C9294A9ACC466053AEF89035726C5F843E5E162AA90DFC3B6C2912884AF31

0x02000000F900000000000000 : 0x02001200150027002000470015005C00200001CCF7110D1BD9A74BFD1D7D7D2D9D55607E7B837DFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFED67EB354AB9040000013DFE98A9CEBAA3C6901E643B50271D3C1A3E4C170000000000000000000000000000000000000000000000129814CAB546FC0000

0x02000000FA00000000000000 : 0x01000A0021002B0001000442BF6EBAAD788AD325B7A9AF4942BA2DC6E7E4EE9C1AEFAE2F38A790BD7E81E431

0x02000000FB00000000000000 : 0x02001200150027002000470015005C00200001F9D5C39F787349C66B088010FD99A29BD5A38989FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFD65DDBE509D40000013F5CD2B711EAF2131237650C2EB03732F1BEA2CD00000000000000000000000000000000000000000000000029A2241AF62C0000

0x02000000FC00000000000000 : 0x01000A0021002B00010004CB33882AC0004112E64C730BBD1C11B9615260B6C2C5E4075D667FB196E8ECF131

0x02000000FD00000000000000 : 0x01000A0021002B000100041B7F313C3AD211C32A43B3285B5A7DCF9E8C4BA45546CDEA19CED8B36AABF56731

0x02000000FE00000000000000 : 0x01000A0021002B00010004E0EE1DA6806A0D11A72063884319BD2FC75C4C0259CE054A8E272EC361C2D69431

0x02000000FF00000000000000 : 0x01000A0021002B000100042668C29191A3789B8E271B8063A122D12DFC845C29E739C958AB212CF43A34D231

....

RocksDB: setting up Python on RocksDB

Python on RocksDB

Looking for potential tool to interact with RocksDB.

Download https://github.com/twmht/python-rocksdb

If you already have the rocksdb installed on earlier project or setup, the build here will not conflicted with the default RocksDB build where it make its own directory and everything is built within it. Something interesting in this part where a lot of Test Cases where they can be potentially be consumed in the future where the RocksDB needs to be tested.

$ apt-get install build-essential libsnappy-dev zlib1g-dev libbz2-dev libgflags-dev
$ git clone https://github.com/facebook/rocksdb.git
$ cd rocksdb
$ mkdir build && cd build
$ cmake ..
$ make
$ export CPLUS_INCLUDE_PATH=${CPLUS_INCLUDE_PATH}:`pwd`/../include
$ export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:`pwd`
$ export LIBRARY_PATH=${LIBRARY_PATH}:`pwd`

Note: The github project has a typo on the instruction where ". bin/active" should be ". bin/activate"

starting from rocksdb directory

$ cd ../
$ sudo apt-get install python-virtualenv python-dev
$ virtualenv pyrocks_test
$ cd pyrocks_test
$ . bin/activate
$ pip install python-rocksdb

Test RocksDB

>>> import rocksdb
>>> db = rocksdb.DB("test.db", rocksdb.Options(create_if_missing=True))
>>> db.put(b'a', b'data')
>>> print db.get(b'a')
data

The test essentially creating a directory with the contents below. So, the test.db is a database/datafile bundle.

ls -las test.db
total 160
4 drwxr-xr-x 2 rocks rocks 4096 Mar 19 13:40 .
4 drwxrwxr-x 8 rocks rocks 4096 Mar 19 13:40 ..
4 -rw-r--r-- 1 rocks rocks 27 Mar 19 13:41 000003.log
4 -rw-r--r-- 1 rocks rocks 16 Mar 19 13:40 CURRENT
4 -rw-r--r-- 1 rocks rocks 37 Mar 19 13:40 IDENTITY
0 -rw-r--r-- 1 rocks rocks 0 Mar 19 13:40 LOCK
128 -rw-rw-r-- 1 rocks rocks 14869 Mar 19 13:41 LOG
4 -rw-r--r-- 1 rocks rocks 13 Mar 19 13:40 MANIFEST-000001
8 -rw-r--r-- 1 rocks rocks 4571 Mar 19 13:40 OPTIONS-000005

If you are getting missing librocksdb.so.5, you will likely need to rebuild the rocksdb again.

(pyrocks_test) rocks @ubuntu:~/rocksdb/pyrocks_test$ python ./backup4.py
Traceback (most recent call last):
File "./backup4.py", line 1, in <module>
import rocksdb
File "/home/rocks /rocksdb/pyrocks_test/local/lib/python2.7/site-packages/rocksdb/init.py", line 1, in <module>
from ._rocksdb import *

ImportError: librocksdb.so.5: cannot open shared object file: No such file or directory

rocks@ubuntu:~/rocksdb$ find . -name librocksdb.so.5
./build/librocksdb.so.5

An example to show what RocksDB sst_dump and how it is used. One of the benefits for my project is to take a quick pick at the contents and dumping the contents to a flat file. This tool is sensitive to how the database is first open and the type of "comparator" used. The tool originally developed by LevelDB, that is an assumed a default comparator.

Using sst_dump to scan through entire 000007.sst file and read only 10 rows out in hex format.

./sst_dump --file=rocksdb --command=scan --read_num=10 --output_hex

/sst_dump --file=/home/ubuntu/dbtest1_bk --command=scan --read_num=10 --output_hex
from [] to []
Process /home/ubuntu/dbtest1_bk/000007.sst
Sst file format: block-based
'01000000010000000000000000000000000000000000000000085A000000000000' seq:57025, type:1 => 0000000000000000000000000000000000000000000000B914F558714F052622
'010000000100000000000000000000000000000000000000004B59000000000000' seq:56553, type:1 => 0000000000000000000000000000000000000000000000B8CF95546DAED7A770
'010000000100000000000000000000000000000000000000005658000000000000' seq:55933, type:1 => 0000000000000000000000000000000000000000000000B88A31C2EB69E3A770
'010000000100000000000000000000000000000000000000005956000000000000' seq:54697, type:1 => 0000000000000000000000000000000000000000000000B88A31BF2C26C0B770
'01000000010000000000000000000000000000000000000000E255000000000000' seq:54410, type:1 => 0000000000000000000000000000000000000000000000B889801C6FF7FBB770
'010000000100000000000000000000000000000000000000005755000000000000' seq:54082, type:1 => 0000000000000000000000000000000000000000000000B7F0D842B7C6AFB770
'010000000100000000000000000000000000000000000000005754000000000000' seq:53462, type:1 => 0000000000000000000000000000000000000000000000B7EF74FD3F6925B770
'01000000010000000000000000000000000000000000000000BA53000000000000' seq:53087, type:1 => 0000000000000000000000000000000000000000000000B7EB4B2CD65087B770
'010000000100000000000000000000000000000000000000001452000000000000' seq:52049, type:1 => 000000000000000000000000000000000000000000000081B5817F1071E7B770
'01000000010000000000000000000000000000000000000000F351000000000000' seq:51968, type:1 => 000000000000000000000000000000000000000000000081A7A0C85CCA83B770

RocksDB: Restore

Restoring RocksDB is fairly straightforward with RocksDB "restore" command.

Prior to restoring the RocksDB database. The database contents look like the following.

ls -las

total 5336

4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 18 12:13 .

4 drwxrwxr-x 11 ubuntu ubuntu 4096 Apr 18 11:10 ..

52 -rw-r--r-- 1 ubuntu ubuntu 50223 Apr 18 12:13 000004.sst

0 -rw-r--r-- 1 ubuntu ubuntu 0 Apr 18 12:13 000006.log

4 -rw-r--r-- 1 ubuntu ubuntu 16 Apr 18 12:13 CURRENT

4 -rw-r--r-- 1 ubuntu ubuntu 37 Apr 18 11:10 IDENTITY

0 -rw-r--r-- 1 ubuntu ubuntu 0 Apr 18 11:10 LOCK

128 -rw-rw-r-- 1 ubuntu ubuntu 16399 Apr 18 12:13 LOG

128 -rw-rw-r-- 1 ubuntu ubuntu 13780 Apr 18 11:10 LOG.old.1524075040138844

128 -rw-rw-r-- 1 ubuntu ubuntu 15445 Apr 18 11:10 LOG.old.1524075040143328

128 -rw-rw-r-- 1 ubuntu ubuntu 15546 Apr 18 11:10 LOG.old.1524075079672369

128 -rw-rw-r-- 1 ubuntu ubuntu 15084 Apr 18 11:11 LOG.old.1524075079683013

128 -rw-rw-r-- 1 ubuntu ubuntu 15546 Apr 18 11:11 LOG.old.1524075087547290

128 -rw-rw-r-- 1 ubuntu ubuntu 15088 Apr 18 11:11 LOG.old.1524075087552152

128 -rw-rw-r-- 1 ubuntu ubuntu 15550 Apr 18 11:11 LOG.old.1524075100714403

128 -rw-rw-r-- 1 ubuntu ubuntu 15088 Apr 18 11:11 LOG.old.1524075100725789

128 -rw-rw-r-- 1 ubuntu ubuntu 15550 Apr 18 11:11 LOG.old.1524075125402198

128 -rw-rw-r-- 1 ubuntu ubuntu 15089 Apr 18 11:12 LOG.old.1524075125404941

128 -rw-rw-r-- 1 ubuntu ubuntu 15551 Apr 18 11:12 LOG.old.1524075171558512

128 -rw-rw-r-- 1 ubuntu ubuntu 15089 Apr 18 11:12 LOG.old.1524075171561161

128 -rw-rw-r-- 1 ubuntu ubuntu 15551 Apr 18 11:12 LOG.old.1524075190238360

128 -rw-rw-r-- 1 ubuntu ubuntu 15089 Apr 18 11:13 LOG.old.1524075190239973

128 -rw-rw-r-- 1 ubuntu ubuntu 15551 Apr 18 11:13 LOG.old.1524075194845338

128 -rw-rw-r-- 1 ubuntu ubuntu 15085 Apr 18 11:13 LOG.old.1524075194846823

128 -rw-rw-r-- 1 ubuntu ubuntu 15547 Apr 18 11:13 LOG.old.1524075325892307

128 -rw-rw-r-- 1 ubuntu ubuntu 15089 Apr 18 11:15 LOG.old.1524075325895101

128 -rw-rw-r-- 1 ubuntu ubuntu 15551 Apr 18 11:15 LOG.old.1524075329179111

128 -rw-rw-r-- 1 ubuntu ubuntu 15089 Apr 18 11:15 LOG.old.1524075329183192

128 -rw-rw-r-- 1 ubuntu ubuntu 15551 Apr 18 11:15 LOG.old.1524075535469131

128 -rw-rw-r-- 1 ubuntu ubuntu 7608 Apr 18 11:18 LOG.old.1524075598421473

128 -rw-rw-r-- 1 ubuntu ubuntu 15089 Apr 18 11:19 LOG.old.1524075598423727

128 -rw-rw-r-- 1 ubuntu ubuntu 15551 Apr 18 11:19 LOG.old.1524075604652854

128 -rw-rw-r-- 1 ubuntu ubuntu 15089 Apr 18 11:20 LOG.old.1524075604655675

128 -rw-rw-r-- 1 ubuntu ubuntu 15551 Apr 18 11:20 LOG.old.1524075695916467

128 -rw-rw-r-- 1 ubuntu ubuntu 15089 Apr 18 11:21 LOG.old.1524075695918368

128 -rw-rw-r-- 1 ubuntu ubuntu 15551 Apr 18 11:21 LOG.old.1524078475647987

128 -rw-rw-r-- 1 ubuntu ubuntu 15089 Apr 18 12:07 LOG.old.1524078475649972

128 -rw-rw-r-- 1 ubuntu ubuntu 15551 Apr 18 12:07 LOG.old.1524078501784982

128 -rw-rw-r-- 1 ubuntu ubuntu 15089 Apr 18 12:08 LOG.old.1524078501787022

128 -rw-rw-r-- 1 ubuntu ubuntu 15551 Apr 18 12:08 LOG.old.1524078508998757

128 -rw-rw-r-- 1 ubuntu ubuntu 15089 Apr 18 12:08 LOG.old.1524078509000417

128 -rw-rw-r-- 1 ubuntu ubuntu 15551 Apr 18 12:08 LOG.old.1524078548279053

128 -rw-rw-r-- 1 ubuntu ubuntu 15089 Apr 18 12:09 LOG.old.1524078548285225

128 -rw-rw-r-- 1 ubuntu ubuntu 15551 Apr 18 12:09 LOG.old.1524078571907249

128 -rw-rw-r-- 1 ubuntu ubuntu 15089 Apr 18 12:09 LOG.old.1524078571908932

128 -rw-rw-r-- 1 ubuntu ubuntu 15551 Apr 18 12:09 LOG.old.1524078654343110

128 -rw-rw-r-- 1 ubuntu ubuntu 7608 Apr 18 12:10 LOG.old.1524078728489411

128 -rw-rw-r-- 1 ubuntu ubuntu 7608 Apr 18 12:12 LOG.old.1524078818755099

4 -rw-r--r-- 1 ubuntu ubuntu 132 Apr 18 12:13 MANIFEST-000005

8 -rw-r--r-- 1 ubuntu ubuntu 4238 Apr 18 11:10 OPTIONS-000005

8 -rw-r--r-- 1 ubuntu ubuntu 4661 Apr 18 12:13 OPTIONS-000008

Restore database

./ldb --db=/home/ubuntu/dbtest1 restore backup_dir=/home/ubuntu/backup_directory/x.bk

open restore engine OK

restore from backup OK

After database restore

ls -las dbtest1
total 76
4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 18 12:37 .
4 drwxrwxr-x 11 ubuntu ubuntu 4096 Apr 18 11:10 ..
52 -rw-r--r-- 1 ubuntu ubuntu 50223 Apr 18 12:37 000004.sst
4 -rw-r--r-- 1 ubuntu ubuntu 16 Apr 18 12:37 CURRENT
4 -rw-r--r-- 1 ubuntu ubuntu 132 Apr 18 12:37 MANIFEST-000005
8 -rw-r--r-- 1 ubuntu ubuntu 4661 Apr 18 12:37 OPTIONS-000008

Summary, make sure the application is able to pick up from where it left off. The transactions will be omitted and all bundled into 3 different directories. That's all, pretty straightforward.