I encountered a Cassandra start-up issue after adding a node, specifically after inserting the seed. Restoring the cassandra.yaml couldn't return to its original situation as things have been cached. The VMs had to be restored with snapshots. The node stays up on its own only briefly and then died down with the following error "nodetool: Failed to connect to '127.0.0.1:7199' - ConnectException: 'Connection refused (Connection refused)'"
cassandra@cassandra:~$ nodetool status
Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID Rack
UN 127.0.0.1 228.44 KiB 16 100.0% 37562f38-b9da-4e77-b44d-6572248ec7d1 rack1
As soon as seeds are added to the cassandra.yaml, the nodes died.
cassandra@cassandra02:~$ nodetool status
nodetool: Failed to connect to '127.0.0.1:7199' - ConnectException: 'Connection refused (Connection refused)'
system.log show the following.
Gossiper.java:2142 - No local state, state is in silent shutdown, or node hasn't joined, not announcing shutdown
I did not clear out the node2 Cassandra data I cloned from node1. Once deleted the data and restarted Cassandra, it came back up. Since node2 was cloned from node1 and its data, it caused a synching issue and refused to start up.
cassandra@cassandra02:/var/log/cassandra$ ls -las /var/lib/cassandra/
total 24
4 drwxr-xr-x 6 cassandra cassandra 4096 Aug 15 16:39 .
4 drwxr-xr-x 42 root root 4096 Aug 15 16:39 ..
4 drwxr-xr-x 2 cassandra cassandra 4096 Sep 29 16:36 commitlog
4 drwxr-xr-x 8 cassandra cassandra 4096 Aug 17 21:18 data
4 drwxr-xr-x 2 cassandra cassandra 4096 Aug 15 16:39 hints
4 drwxr-xr-x 2 cassandra cassandra 4096 Sep 28 16:31 saved_caches
cassandra@cassandra02:/var/log/cassandra$ sudo service cassandra stop
cassandra@cassandra02:/var/log/cassandra$ sudo rm -rf /var/lib/cassandra/*
cassandra@cassandra02:/var/log/cassandra$ sudo service cassandra start
cassandra@cassandra02:/var/log/cassandra$ nodetool status
Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID Rack
UN 192.168.1.32 265 KiB 16 100.0% 37562f38-b9da-4e77-b44d-6572248ec7d1 rack1
cassandra@cassandra02:/var/log/cassandra$ nodetool status
Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID Rack
UN 192.168.1.35 79.81 KiB 16 51.2% d9d37072-817a-4095-87da-c4fc7a4686f8 rack1
UN 192.168.1.32 265 KiB 16 48.8% 37562f38-b9da-4e77-b44d-6572248ec7d1 rack1
Had tried the following through google, but I need help with it.
It isn't the JMX issue.
1.$JVM_OPTS -Djava.rmi.server.hostname=127.0.0.1 in cassandra-env.sh
It isn't the Ubuntu firewall UFW issue. It is disabled.
sudo UFW disable
It isn't the 2 gig memory issue.
No comments:
Post a Comment