10.2 Kafka

Addition of another server to the Kafka cluster fails after the only server that was part of the Kafka cluster is decommissioned

Explanation: Consider a scenario, where the replication count is 1 and the only server (broker) that was added to the Kafka cluster is decommissioned. On executing the command zman server-role-kafka-add-broker (zman srkab) to add another server (broker) in the zone, the operation to add the broker to the Kafka cluster fails. The error messages Schema registry started – false and Some of the processes have failed to start. Marking queue as failed are displayed in the zenworks loader-messages.log.
Action: Refer to the following steps to clean up both the ZooKeeper and Kafka data directory:

For ZooKeeper:

  1. Stop all the ZENworks services on the Primary Server on which the ZooKeeper role is enabled. For more information, see Stopping the ZENworks Services.

  2. Remove the zookeeper.properties file from the path %ZENWORKS_HOME%\novell\ZENworks\conf (on Windows) or /etc/opt/novell/ZENworks/conf (on Linux).

  3. Remove the ZooKeeper data directory from the path %ZENWORKS_HOME%\novell\ZENworks\work\zookeeper (on Windows) or /var/opt/novell/ZENworks/zookeeper (on Linux).

  4. Restart all the ZENworks services. For more information, see Starting the ZENworks Services.

For Kafka:

  1. Stop all the ZENworks services on the server on which the zman command to add the Kafka broker was executed but failed. For more information, see Stopping the ZENworks Services.

  2. Remove the Kafka data directory from the path /var/opt/novell/zenworks/kafka-data.

  3. Restart all the ZENworks services. For more information, see Starting the ZENworks Services.

NOTE:If the replication count is 1, then before decommissioning the last server that has a Kafka role enabled in it, run the zman server-role-kafka-add-broker (zman srkab) to add a new broker and then proceed with the decommissioning of the existing server.

However, if you have already decommissioned the server, ensure that you bring up a server with the same hostname by backing up and restoring the Kafka data. For more information, see Replacing an Existing Primary Server with a New Primary Server in the ZENworks Disaster Recovery Reference.

At least one Kafka connector has failed

Explanation: If for any reason any of the Kafka connectors have failed, which is displayed in the Data Sync panel of the Diagnostics page in ZCC, then the failed connectors will not be able to sync data from their associated RDBMS tables to Vertica.
Action: Run the command zman server-role-kafka-reconfigure-connectors -c <name of the connector> to reconfigure the failed Kafka connector.

You can retrieve the names of the all connectors within the command line utility by running the zman server-role-kafka-list-connectors command.

After reconfiguring the Kafka connectors, you need to restart the Kafka-connect service by running the following command:

              systemctl restart microfocus-connect.service

At least one Kafka connector has failed

Explanation: If for any reason any of the Kafka connectors have failed, which is displayed in the Connector Status panel of the Diagnostics page in ZCC, then the failed connectors will not be able to sync data from their associated RDBMS tables to Vertica.
Action: Open the services-messages.log file of the server where you have logged in and search for the ‘Schema being registered is incompatible with an earlier schema’ error. If the error is logged in the server’s log file, then run the command zman server-role-kafka-reconfigure-connectors -c <name of the connector> to reconfigure the failed Kafka connector.

Specify the <name of the connector> in the command. You can obtain the name of the failed connector from the services-messages.log file.

If are unable to find the 'Schema being registered is incompatible with an earlier schema' error message in the services-messages.log file, you can ignore the connector status as the application will try to reconfigure the failed connector during the next recovery attempt.

You can retrieve the names of all connectors within the command line utility by running the zman server-role-kafka-list-connectors command.

After reconfiguring the Kafka connectors, you need to restart the Kafka-connect service by running the following command:

systemctl restart microfocus-zenkafka-connect.service