Although with ZENworks the server settings are tuned during an upgrade or new installation, it is valuable to understand the default settings for your server and how you can further tune these settings if required. The table below shows the default values based on the memory installed in your server:
To utilize the 64-bit JVM (Java Virtual Machine) of ZENworks to the best possible extent, you can increase the following values beyond the defaults:
the maximum number of simultaneous web requests that the server will service. When these are exhausted, the server will begin returning a 503 error to agents indicating that it is busy. This will cause the busy retry logic to be executed. You can monitor the overall threads in use from the Diagnostics tab of the ZENworks Primary Server and, if necessary, adjust appropriately.
These values are found in the server.xml file in the following locations:
To achieve further performance increases from your Primary Servers, you can change the thread values:
Open the server.xml file for the operating system on which the Primary Server is running.
Locate the line with the text <Connector port="7443" and change the value for maxThreads as desired.
Save the file.
Restart the ZENworks services again.
If you increase the number of threads, you will also want to monitor the Java heap usage of the service. If this begins to approach the allocated amount of memory, you will also want to look at increasing the heap size, or reducing the threads.
Open the server.xml file for the operating system on which the Primary Server is running.
Locate the line with the text <Connector port="7491" and change the value for maxThreads as desired. You should consider increasing this number if the server is servicing a large number of configuration or authentication requests.
As the ports in ZENworks can be customized, if you don’t find a connector on port 7491 search instead for the port number which falls in the range 7491 to 7600.
Save the file.
Restart the ZENworksClientMgmt service again.
If you increase the number of threads, you will also want to monitor the Java heap usage of the . If this begins to approach the allocated amount of memory, you will also want to look at increasing the heap size, or reducing the threads.
ZENworks uses the 64-bit JVM, so you can tune the Java memory allocations to allow ZENworks services to utilize more memory if required. The sample settings in the following procedures were tested in the engineering lab on a server with 16 GB of RAM.
Configuring the Maximum Heap Memory Size for ZENworks Loader Service on a Linux Primary Server
Configuring the Maximum Heap Memory Size for ZENworks JoinProxy Service on a Linux Primary Server
Configuring the Maximum Heap Memory Size for ZENworksAdminMgmt Service on a Windows Primary Server
Configuring the Maximum Heap Memory Size for ZENworks Loader Service on a Windows Primary Server
Configuring the Maximum Heap Memory Size for ZENworks JoinProxy Service on a Windows Primary Server
The ZENworks Loader service is responsible for performing background operations such as subscribing to external zones, packaging content, storing collection data in the database, processing audit events, and much more. If you have a server that is doing a lot of these background tasks and you see the heap memory decrease for the ZENworks Loader service, you can increase the heap size by doing the following:
Using a Linux text editor such as vi or gedit, create or modify the file with the following content:
JAVA_MIN_HEAP="-Xms256m" JAVA_MAX_HEAP="-Xmx1024m" JAVA_MIN_PERM_SIZE="-XX:PermSize=128m" JAVA_MAX_PERM_SIZE="-XX:MaxPermSize=128m"
IMPORTANT:Change the JAVA_MAX_HEAP value from -Xmx1024m to -Xmx<desired memory in MB>m. For example, -Xmx4096m to configure 4GB.
You need to reserve RAM for the operating system, so change the values in increments that make sense.
Save the file.
Run the following commands:
Start the ZENworks Loader service by running .
The ZENworks Join Proxy service handles the Remote Management requests from agents in a private network or outside the DMZ. By default, the maximum number of connections that a Join Proxy service can handle is 100 for a Primary Server and 1000 from a Satellite Server. If required, you can modify the number of connections. By increasing the number of connections, if you find that ZENjoinProxy runs out of heap space, you can increase the heap size by performing the following steps.
Perform the following steps on a Linux Primary Server to increase the heap space for the ZENJoinProxy service:
Using a Linux text editor such as vi or gedit, open the systemd-zenjoinproxy file from the following location:
Modify the maximum heap space value from -Xmx512m to -Xmx<desired memory in MB>m.
For example: -Xmx1024m to configure 1GB.
NOTE:Before changing the value, ensure that you reserve enough RAM for the operating system.
Save the file.
Restart .
The ZENworksAdminMgmt service is the Tomcat instance used by the ZENworks Primary Server. It services ZENworks Control Center. If you find that ZENworksAdminMgmt runs out of heap space, you can increase the heap size by doing the following:
Run .
On the Java tab, change the Maximum memory pool from 1024 to a higher value. For example, 4096.
You need to reserve RAM for the operating system, so change the values in increments that make sense.
Click Apply, then click OK.
Restart the ZENworks services.
The ZENworks Loader service is responsible for performing background operations such as subscribing to external zones, packaging content, storing collection data in the database, processing audit events and much more. If you have a server that is doing a lot of these background tasks and if you see the heap memory get low for the service, you can increase the heap size by doing the following:
Run ZENworksLoaderw.
On the Java tab, change the Maximum memory pool from 1024 to a desired value. For example 4096.
You need to reserve RAM for the operating system, so change the values in increments that make sense.
Click Apply, then click OK.
Restart the ZENworksLoader services.
The ZENworks Join Proxy service handles the Remote Management requests from agents in a private network or outside DMZ. By default, the maximum number of connections that a Join Proxy service can handle is 100 for a Primary Server and 1000 from a Satellite Server. If required, you can modify the number of connections. By increasing the number of connections, if you find that ZENjoinProxy runs out of heap space, you can increase the heap size by performing the following steps.
Perform the following steps on a Windows Primary Server to increase the heap space for the service:
Run in the command prompt.
In the ZENworks Join Proxy Properties window, click the Java tab.
Modify the Maximum memory pool from 1024 to a higher value.
NOTE:Before changing the value, ensure that you reserve enough RAM for the operating system.
Click Apply or OK to save the changes.
Restart the ZENworks Join Proxy services.
In large ZENworks deployment zones, if you experience “OutofMemoryExceptions” frequently, you can limit the web service requests coming to the ZENworks primary server by configuring maximum permitted heap limit.
To configure maximum permitted heap limit:
Stop the ZENworks services on the Primary Servers.
Add the following line within the Engine properties of the server.xml file:
<Valve className="com.novell.zenworks.tomcat.ZENRequestValve" debug="false" maxUsedHeapPercent="90"/>
Path for server.xml on Linux:
NOTE:It is not recommended to configure the maxUsedHeapPercent value for the ZENworks Primary Server, which is used for accessing ZENworks Control Center. This could result in blank pages during peak loads that exceed the configured permitted heap limit.
You can enable valve debug logging by configuring the debug value as True.
Start the ZENworks services.
Valve logs can be found at following location:
NOTE:When the heap memory usage by the ZENServer process reaches the maximum permitted heap, web service requests coming from managed devices will be rejected with status 503. You can find the message Request would exceed the maximum permitted heap limit in valve debug logs.
Even after configuring the maxUsedHeapPercent value, if you observe that the ZENworks Primary Server is not responsive or is in a busy state for a long time, revert the configuration and consult the Micro Focus Customer Support team.
Database connections are often expensive to create because of the overhead of establishing a network connection and initializing a database connection session in the back-end database. Hence, ZENworks uses the c3p0 library which helps in maintaining a pool of connections to the database to perform at optimal levels. Like any other pooling mechanism, you will need to tune the connections to your needs. This section will explain the various options available for your tuning needs.
You can tune each of the following connection pools independently:
ZENLoader Connection Pool: This connection pool is used for most of the ZENworks Loader functions that require access to the database, including storing inventory database; recording content repository data during packaging, patch subscriptions, zone-to-zone subscriptions, and all other tasks performed by the ZENworks Loader; and storing the device message and status.
ZENLoader Batch Connection Pool: This connection pool is used by ZENworks Loader for storing device message and status information such as install or launch bundle status. This status is stored in batches and, as such, uses a separate connection pool.
InvUI Connection Pool: This connection pool is used when browsing inventory data on devices or when running canned reports from ZENworks Control Center.
Audit ZENServer Connection Pool: This connection pool is used by the ZENworks Server process to retrieve data from the Audit database. This is used mainly by ZENworks Control Center servers.
Audit ZENLoader Connection Pool: This connection pool is used by the ZENworks Server process to store data in the Audit database. This is mainly used by Collection servers.
ZENworks provides the following parameters for tuning the connection pools used to connect to the ZENworks database.
MinPoolSize: This value specifies the minimum number of connections available for a given connection pool. For instance, if this value is set to 5, then the connection pool will always have at least 5 connections to the database.
Batch.MinPoolSize: This value specifies the minimum number of connections available for a the ZENLoader batch connection pool. If this value is not set then the MinPoolSize parameter value is used to determine the minimum ZENLoader batch connection pool size.
MaxPoolSize: This value specifies the maximum number of connections available for a given connection pool. The pool size will grow dynamically from the minimum size to the maximum size as required by the requests being made to the server. Once the request volume reduces, the number of threads will reduce after the specified period of time.
ZENLoader.MaxPoolSize: This value specifies the maximum number of connections available to the ZENLoader connection pool. If this value is not set, then the maximum number is determined by the MaxPoolSize parameter.
Batch.MaxPoolSize: This value specifies the maximum number of connections available to the ZENLoader batch connection pool. If this value is not set, then the maximum number is determined by the MaxPoolSize parameter.
MaxIdleTimeExcessConnections: This value is set in number of seconds and controls how long before the server should end unused connections. This is useful for situations such as first thing in the morning when a large number of clients are logging in, causing a lot of database activity. After this initial login, the number of requests typically drops dramatically. After the specified number of seconds passes, the database connections that are no longer required for active work will be closed.
ZENLoader.MaxIdleTimeExcessConnections: This value controls the amount of time before the excess threads are used by the ZENLoader connection pool. If this value is not present in the zdm.xml file, then the MaxIdleTimeExcessConnections value is used to determine this value.
Batch.MaxIdleTimeExcessConnections: This value controls the amount of time before the excess threads are used by the ZENLoader batch connection pool. If this value is not present in the zdm.xml file, then the MaxIdleTimeExcessConnections value is used to determine this value.
The columns to the right of the parameter identify the recommended parameter value based on the role(s) the server will be performing in your organization. If a server will be acting in multiple roles, you should plan on setting the parameter to the highest value in the row.
NOTE:The values indicated with n/a are not required to be set for the servers performing the corresponding role. The default values are sufficient.
NOTE:* 900 is the ideal pool size for Micro Focus’s Performance and Scale testing. This is roughly 2/3 of the total number of permitted zenserver threads. If you have additional RAM and the database can support additional connections, you can increase the heaps, threadpools, and database connections accordingly, to further increase scalability.
These parameters can be set in the zdm.xml file in the <conf folder>\datamodel folder where the format of the entry is <entry key="paramater name">parameter value</entry>.
After you have determined the appropriate configuration for each server, you will need to calculate the total potential number of connections that all Primary Servers might require to the database. This will allow you to properly configure the number of connections the database will allow. To do this, simply add up the total number of Max Connections for each Primary Server in your environment. Then, if you are using ZENworks Reporting, add an additional 100 connections. An example is listed below.
Adding these values, you get 985+985+95+95+1020+985 = 4165 connections. If you are using ZENworks Reporting, you will get a maximum of 4165+100 = 4265 connections to the database. You should ensure that your database is configured to permit at least so many connections.
Additionally, a separate set of connection pools is available when connecting to the ZENworks Audit database. Unlike the ZENworks database connection pools, there is only a and a ZENLoader connection pool for Audit. The parameter names are the same as the ZENworks database equivalents. The table below shows the recommended values based on server role:
These parameters can be set in the zenaudit.xml file in the <conf folder>\datamodel folder, where the format of the entry is <entry key="paramater name">parameter value</entry>.
To calculate the required number of audit database connections, follow the same model as that shown at the end of ZENworks Database Connection Pool Parameters (zdm.xml).
You can tune each of the processes responsible to store data (status, audit, patch, inventory) by increasing or decreasing the number of threads allocated to the process. If you have more threads, it means that the server will attempt to store more items in parallel. In most cases, the default values should be acceptable. However, if you find that you are seeing a backlog of files in the collection folder on the server, consider increasing these values.
To tune the threads for these processes:
To adjust the thread pool, open the file for the relevant loader module. The following files can be adjusted:
inventorystorer.xml: Controls the number of simultaneous inventory scans that can be stored in the database.
auditeventsstorer.xml: Controls the number of simultaneous audit events that can be stored in the database.
statusstorer.xml: Controls the number of simultaneous status events that can be stored in the database.
AddPatchlinkAnalyzeReporting.xml: Controls the number of patch DAU results that can be stored simultaneously.
messageprocessor.xml: Controls the number of messages that can be stored simultaneously.
These files can be found in the following locations:
Windows: %zenserver_home%\conf\loader\inventorystorer.xml
Linux: /etc/opt/microfocus/zenworks/loader/inventorystorer.xml
Below the file you should find an entry that sets the ThreadPoolSize. The thread pool for the following files look similar to the following entry:
inventorystorer.xml: <Parameter Name="InventoryStorerThreadPoolSize">10</Parameter>
statusstorer.xml: <Parameter Name="thread-count">10</Parameter>
auditeventsstorer.xml: <Parameter Name="thread-pool-size">10</Parameter>
AddPatchlinkAnalyzeReporting.xml: <Parameter Name="PatchAnalyzeThreadPoolSize">10</Parameter>
messageprocessor.xml: <Parameter Name="MessageFilterProcessorThreadPoolSize">5</Parameter>
Set this value to the desired number of threads.
Save the file and restart the ZENworks Loader service.
NOTE:If you increase the number of threads that you allocate for storing, it might also be necessary to increase the MaxPoolSize parameter for the Connection Pool used by the storer that you are increasing. To do this, see ZENworks Database Connection Pool Parameters (zdm.xml) and ZENworks Audit Database Connection Pool Parameters (zenaudit.xml).
Quick Tasks are administrator-initiated tasks that are sent to devices to perform operations on the devices. They allow an administrator to use ZENworks Control Center to perform remote operations on a device such as refreshing the device’s bundle, policy, and configuration information; initiating an inventory scan; rebooting the device; installing, launching, or repairing a bundle; initiating a patch scan; and much more.
Quick Tasks use the ZENworks push notification service. The push notification service enables a ZENworks server to inform a managed device about work, such as a Quick Task, assigned to the device. Once a device is notified of a pending work, the device contacts a ZENworks Primary Server to retrieve the Quick Task.
In order to support push notifications, the ZENworks Update Service (ZeUS) on a device establishes a long-lived connection to the ZENworks server approximately one minute after device startup. The connection stays open until the server responds that there is work to do or until the connection times out (by default, after 60 minutes). If the server responds with a work notification, the connection is closed, ZeUS fetches the assigned Quick Tasks, processes them, and then establishes a new connection to the server. If the connection times out, ZeUS immediately establishes a new connection.
NOTE:ZeUS also performs agent updates. However, the update notifications are negligible in terms of workload and impact in comparison to the Quick Tasks. Therefore, the Tomcat NIO connector tuning focuses on what is required for Quick Tasks.
A device’s long-lived connection is established using the Apache Tomcat NIO (non-blocking IO) connector listening on the ZENworks server port 443. ZENworks uses the NIO connector to provide asynchronous connections with devices while not requiring a Tomcat thread for each connection. A thread is initially allocated to establish a connection but is quickly freed up and only the connection maintained.
Typically, the default settings should be sufficient and not require modification. If tuning is required, the following settings can be adjusted:
maxThreads: Because the NIO connector releases threads as soon as a connection is established, the default setting for maximum number of Tomcat threads should suffice. This default varies depending on the server’s RAM (see Maximum HTTPS Tomcat Threads).
You can use the ZENworks Control Center Diagnostics dashboard to monitor the NIO connector threads in use and, if necessary, adjust appropriately.
maxConnections: The default maximum number of connections for the NIO connector is 8192. In general, the point at which load/traffic is impacted is when the connections come close to or hit the maximum. To reduce this impact, the ZENworks server tomcat valve limits connections to 95% of maxConnections, which means that when the 95% limit is reached the server returns a Server Busy response to the connecting device. The device can then connect to the next ZENworks Primary Server that is included in its closest server rules.
One ZENworks Primary Server officially supports 10,000 devices. For this reason, you should not need to increase the maxConnections beyond 10,000 because the setting is aligned with the maximum number of supported devices. If the server is supporting more than 10,000 devices with acceptable performance, you can experimentally increase the connections up to 15,000. However, keep in mind that anything beyond 10,000 has not been stress-tested and could cause issues. The best option is to ensure that you have enough ZENworks Primary Servers to support the number of managed devices in your zone (maximum of 10,000 devices per server), that you have configured no more than the optimal number (10,000) of maxConnections, and that you have set up your closest Configuration Server rules to allow a device to fail over to other servers until it finds an available connection.
To monitor the number of connections that are remaining in an ESTABLISHED state, you can use tools such as netstat or iproute2.
The following steps explain how to edit the server.xml file on a Primary Server to change the settings. If the Primary Server is a ZENworks Appliance, you can optionally log into the ZENworks Appliance Console and use the ZENworks Configuration options to change the settings.
Note that the NIO connector default is 8192 maximum connections so the maxConnections setting is not generally listed. You only need to add it if specifying a number other than 8192.
Open the server.xml file for the operating system on which the Primary Server is running.
Windows
ZENAdminMgmt: %ZENSERVER_HOME%\services\zenadmin-mgmt\conf
ZENClientMgmt: %ZENSERVER_HOME%\services\zenclient-mgmt\conf
Linux
ZENAdminMgmt: /etc/opt/microfocus/zenworks/tomcat-conf/zenadmin-mgmt/
ZENClientMgmt: /etc/opt/microfocus/zenworks/tomcat-conf/zenclient-mgmt/
Look for the NIO connector for port 7491 section and modify it as necessary:
<Connector SSLEnabled="true" acceptCount="100" allowHostHeaderMismatch="true" ciphers="TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDH_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDH_RSA_WITH_AES_128_GCM_SHA256,TLS_DHE_RSA_WITH_AES_128_GCM_SHA256,TLS_DHE_DSS_WITH_AES_128_GCM_SHA256,SSL_RSA_WITH_RC4_128_MD5,SSL_RSA_WITH_RC4_128_SHA,TLS_DHE_RSA_WITH_AES_256_CBC_SHA,TLS_DHE_RSA_WITH_AES_256_CBC_SHA256,TLS_DHE_DSS_WITH_AES_128_CBC_SHA,TLS_DHE_DSS_WITH_AES_128_CBC_SHA256,TLS_DHE_DSS_WITH_AES_256_CBC_SHA,TLS_DHE_DSS_WITH_AES_256_CBC_SHA256,TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384,TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA,TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA256,TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA,TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384,TLS_ECDH_RSA_WITH_AES_128_CBC_SHA,TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256,TLS_ECDH_RSA_WITH_AES_256_CBC_SHA,TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384,TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_128_CBC_SHA256,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA256" clientAuth="false" disableUploadTimeout="true" enableLookups="false" keyAlias="tomcat" keystoreFile="C:\Program Files (x86)\Micro Focus\ZENworks\conf\security\server.keystore" keystorePass="74721885bbe85e4ed31504aa80caaa5b" maxHttpHeaderSize="8192" maxPostSize="-1" maxSpareThreads="75" maxThreads="1000" minSpareThreads="25" maxConnections="12000" port="443" protocol="org.apache.coyote.http11.Http11NioProtocol" scheme="https" secure="true" sslEnabledProtocols="TLSv1,TLSv1.1,TLSv1.2,SSLv2Hello" sslProtocol="TLS" />
Save the file.
Start the ZENworks services again.
For more information about the other connector settings, see the Apache Tomcat Configuration Reference.
If the Linux Primary server manages more than 20k devices, you need to increase the ephemeral port range for smoother operation.
To increase the ephemeral port range for non-appliance Linux servers, perform the following steps:
In /etc/sysctl.conf, add the following line:
net.ipv4.ip_local_port_range = 15000 65535
Run the following command to load the newly added configuration:
sysctl -p /etc/sysctl.conf