Cloudera Hadoop namenode refused to stay up.

Installed Hadoop from Cloudera into Oracle Virtualbox. I have the same deployment in Mac and other windows boxes and never had an issue. But this one did.

[cloudera@quickstart ~]$ hadoop fs -ls
ls: Call From quickstart.cloudera/10.0.2.15 to quickstart.cloudera:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
[cloudera@quickstart ~]$ service hadoop-hdfs-namenode status
Hadoop namenode is dead and pid file exists                [FAILED]
[cloudera@quickstart ~]$ service  status
status: unrecognized service
[cloudera@quickstart ~]$ uname -a
Linux quickstart.cloudera 2.6.32-358.el6.x86_64 #1 SMP Fri Feb 22 00:31:26 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
[cloudera@quickstart ~]$ service hadoop-hdfs-quickstart.cloudera status
hadoop-hdfs-quickstart.cloudera: unrecognized service
[cloudera@quickstart ~]$ service hadoop-hdfs-namemode status
hadoop-hdfs-namemode: unrecognized service
[cloudera@quickstart ~]$ service hadoop-hdfs-namenode status
Hadoop namenode is dead and pid file exists                [FAILED]
[cloudera@quickstart ~]$ service hadoop-hdfs-quickstart.cloudera status
hadoop-hdfs-quickstart.cloudera: unrecognized service
[cloudera@quickstart ~]$ service hadoop-hdfs-namenode start
Error: root user required

[cloudera@quickstart ~]$ sudo service hadoop-hdfs-namenode start
starting namenode, logging to /var/log/hadoop-hdfs/hadoop-hdfs-namenode-quickstart.cloudera.out
Started Hadoop namenode:                                   [  OK  ]
[cloudera@quickstart ~]$ 



Not really sure what's going on. ..as soon as I started up namenode, it goes back down in less than 30 seconds.  This was fixed when I redeployed. I have the same installation on other more powerful machines and all worked just fine. I suspect this issue has something to with low spec from the machine I was using.

vCenter 6.0 upgrade failed.

In some cases, upgrading vCenter 5.5 to 6.0 could encounter the following errors during pre-upgrade check. That is due to artifacts been left behind since vCenter 2.5 and got carried over from version to version.


[HY000](20000) [Oracle][ODBC][Ora]ORA-20000: ERROR ! Missing constraints:
 VPX_DEVICE.VPX_DEVICE_P1,VPX_DATASTORE.FK_VPX_DS_DC_REF_VPX_ENT;
ORA-06512: at line 260

Create the constraints. Use at your own risk!

alter table VPX_DEVICE add constraint VPX_DEVICE_P1 primary key (DEVICE_ID) using index VPX_DEVICE_P1;
alter table VPX_DATASTORE rename constraint FK_VPX_DS_REF_VPX_ENTITY to FK_VPX_DS_DC_REF_VPX_ENT
alter table VPX_DATASTORE rename constraint FK_VPX_DS_REF_VPX_ENTI to FK_VPX_DS_REF_VPX_ENTITY


Another variant of issue is, the unique index do not even existed. Once can simply try to drop and recreate it.

drop index VPX_DEVICE_P1;
create unique index VPX_DEVICE_P1 ON VPX_DEVICE(DEVICE_ID);
alter table VPX_DEVICE add constraint VPX_DEVICE_P1 primary key (DEVICE_ID) using index VPX_DEVICE_P1;
alter table VPX_DATASTORE rename constraint FK_VPX_DS_REF_VPX_ENTITY to FK_VPX_DS_DC_REF_VPX_ENT
alter table VPX_DATASTORE rename constraint FK_VPX_DS_REF_VPX_ENTI to FK_VPX_DS_REF_VPX_ENTITY

Oracle listener no longer working after migrated to different pool

Common Oracle Listener issues when deploying in vCloud especially when the Vapp moved to another location or gone through IP and Host changes.

Scenario

Vapp - Cloud2 resides in another cloud farm. It was shutdown and migrated to another cloud farm. The IP and Vapp name was forced to change. Obviously, this will impact the /etc/hosts, listener.ora and tnsnames.ora as well. 

After migrated, starting up the lsnrctl will fail. Using netca or changing the listener.ora and /etc/hosts will likely not going to help.

One of the reason is, the listener is still going to be looking a the host_id.

in the listener.xml file 

/home/oracle/app/oracle/diag/tnslsnr/Cloud2-001/listener/alert/log.xml

<msg time='2015-07-06T13:26:52.010-06:00' org_id='oracle' comp_id='tnslsnr'
type='UNKNOWN' level='16' host_id='Cloud2-001'
host_addr='UNKNOWN'>
<txt>TNS-12545: Connect failed because target host or object does not exist
TNS-12560: TNS:protocol adapter error
  TNS-00515: Connect failed because target host or object does not exist
   Linux Error: 99: Cannot assign requested address
</txt>
</msg>

Resolution

Changing the Computer Name to whatever the new /etc/hosts and listener.ora host name is .. will fix the issue. Once the Computer Name is changed, boot up the Vapp and start up the lsnrctl.

In short, the Computer Name is going to impact the Oracle listener.

Windows/Vmware: The disk is offline because of policy by an administrator

I presented the vapp to cloud but the disks never showed up. Upon checking on device manager, it states "The disk is offline because of policy by an administrator". I was looking up and down at the windows policy and nothing stated that the my vapp inherited this policy from the template. When I checked the Disk management, I realized they are all there just never came online on first boot up.

Missing all the disks

Some silly warning.

Bring them all online, initialize, and formatting

After a few reboots, the disks will remain intact. Perhaps, this is a problem for Windows presented in vmdk ? I have plenty of linux vapps that did show this problem.

Oracle Orion on Windows

A IOPS benchmarking tool for Oracle database. It is not only measuring the IOs but simulating as well.

Create a text file call mytest.lun then insert the datafile you would like to test againts.

C:\APP\ADMIN\ORADATA\CS\VCLOUD.DBF


Then run the the command from DOS. Make sure you run it as Administrator that has privide to write CVS file to the folder.

Microsoft Windows [Version 6.1.7601]
Copyright (c) 2009 Microsoft Corporation.  All rights reserved.

C:\Windows\system32>cd C:\vpoc

C:\vpoc>C:\app\ADMIN\product\11.2.0\dbhome_2\BIN\orion -run simple -testname mytest -num_disks 1


ORION: ORacle IO Numbers -- Version 11.2.0.1.0
mytest_20141230_1713
Calibration will take approximately 9 minutes.
Using a large value for -cache_size may take longer.

The Result. All the CVS got created in this folder as well.

Example of mytest


ORION VERSION 11.2.0.1.0

Command line:
-run simple -testname mytest -num_disks 1 

These options enable these settings:
Test: mytest
Small IO size: 8 KB
Large IO size: 1024 KB
IO types: small random IOs, large random IOs
Sequential stream pattern: one LUN per stream 
Writes: 0%
Cache size: not specified
Duration for each data point: 60 seconds
Small Columns:,      0
Large Columns:,      0,      1,      2
Total Data Points: 8

Name: C:\APP\ADMIN\ORADATA\\CS\VCLOUD.DBF Size: 3202228224
1 files found.

Maximum Large MBPS=49.10 @ Small=0 and Large=2
Maximum Small IOPS=156 @ Small=5 and Large=0
Minimum Small Latency=10009.27 usecs @ Small=1 and Large=0

Installing ioping

Sometimes, I am using this tool as a real time IOPS monitoring tool. It is a lightweight version of Orion from the database perspective.

Download ioping. 

https://code.google.com/p/ioping/downloads/list

Logon as root and download the ioping tool

[root@Cloud2 ioping]# wget https://ioping.googlecode.com/files/ioping-0.8.tar.gz
--2014-12-30 16:05:17--  https://ioping.googlecode.com/files/ioping-0.8.tar.gz
Resolving ioping.googlecode.com... 74.125.25.82, 2607:f8b0:400e:c03::52
Connecting to ioping.googlecode.com|74.125.25.82|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 10590 (10K) [application/x-gzip]
Saving to: `ioping-0.8.tar.gz'

100%[================================================================================>] 10,590      62.2K/s   in 0.2s

2014-12-30 16:05:18 (62.2 KB/s) - `ioping-0.8.tar.gz' saved [10590/10590]

Extract the tool

[root@Cloud2 ioping]# tar -zxvf ioping-0.8.tar.gz

ioping-0.8/version

ioping-0.8/ioping.c

ioping-0.8/ioping.1

ioping-0.8/README

ioping-0.8/changelog

ioping-0.8/ioping.spec

ioping-0.8/Makefile

[root@Cloud2 ioping]# ls -lrt

total 24

-rw-r--r-- 1 root root 10590 Dec 30  2013 ioping-0.8.tar.gz

drwxr-xr-x 2 root root  4096 Dec 30 16:06 ioping-0.8

[root@Cloud2 ioping]# cd ioping-0.8

[root@Cloud2 ioping-0.8]# ls -lrt

total 80

-rw-r--r-- 1 1000 1000  1232 Jan 25  2013 ioping.spec

-rw-r--r-- 1 1000 1000   421 Dec 29  2013 README

-rw-r--r-- 1 1000 1000   477 Dec 29  2013 changelog

-rw-r--r-- 1 1000 1000  4331 Dec 29  2013 ioping.1

-rw-r--r-- 1 1000 1000 23033 Dec 29  2013 ioping.c

-rw-r--r-- 1 1000 1000  1493 Dec 29  2013 Makefile

-rw-r--r-- 1 1000 1000     4 Dec 30  2013 version

Build the tool

[root@Cloud2 ioping-0.8]# make ioping
cc -std=gnu99 -g -Wall -Wextra -pedantic -DVERSION=\"0.8\" -c -o ioping.o ioping.c
cc -o ioping ioping.o -std=gnu99 -g -Wall -Wextra -pedantic  -lm
[root@Cloud2 ioping-0.8]# ls -las
total 192
 8 drwxr-xr-x 2 root root  4096 Dec 30 16:09 .
 8 drwxr-xr-x 3 root root  4096 Dec 30 16:06 ..
 8 -rw-r--r-- 1 1000 1000   477 Dec 29  2013 changelog
44 -rwxr-xr-x 1 root root 39311 Dec 30 16:09 ioping
12 -rw-r--r-- 1 1000 1000  4331 Dec 29  2013 ioping.1
28 -rw-r--r-- 1 1000 1000 23033 Dec 29  2013 ioping.c
52 -rw-r--r-- 1 root root 47200 Dec 30 16:09 ioping.o
 8 -rw-r--r-- 1 1000 1000  1232 Jan 25  2013 ioping.spec
 8 -rw-r--r-- 1 1000 1000  1493 Dec 29  2013 Makefile
 8 -rw-r--r-- 1 1000 1000   421 Dec 29  2013 README
 8 -rw-r--r-- 1 1000 1000     4 Dec 30  2013 version
[root@Cloud2 ioping-0.8]#

Run the tool

[root@Cloud2 ioping-0.8]# ./ioping -c 10 -s 1M /tmp

1.0 MiB from /tmp (ext3 /dev/root): request=1 time=53.7 ms

1.0 MiB from /tmp (ext3 /dev/root): request=2 time=68.7 ms

1.0 MiB from /tmp (ext3 /dev/root): request=3 time=53.7 ms

1.0 MiB from /tmp (ext3 /dev/root): request=4 time=61.5 ms

1.0 MiB from /tmp (ext3 /dev/root): request=5 time=53.2 ms

1.0 MiB from /tmp (ext3 /dev/root): request=6 time=57.3 ms

1.0 MiB from /tmp (ext3 /dev/root): request=7 time=54.4 ms

Read the manual and see what's interest you.

https://code.google.com/p/ioping/wiki/man

Starting new PostgreSQL blog

Today, I decided to start a Postgres blog as I started to learn about it. I have had exposure to Postgres since a 3 years back where our company started designing Postgres as backend database for more than half of our products with 8.4 then 9.1 but I have never shown any interests on picking it up and getting serious about it. That's may be due to I am too occupied with Oracle database. In addition, the demand for Postgres job market is always on the weak side. With that, Postgres is not very appealing to me. I hardly seen any companies hiring a Fulltime Postgres DBA. They always comes in the skills that good to have but not the primary skill set that companies are looking for. Market share wise, Postgres ranked 4th after Oracle, MSSQL and MySQL based on this article http://db-engines.com/en/ranking . From my experience, the market share in the link is about right.