General Performance/Tuning suggestions for Novell exteNd Director Environment

Does exteNd have some general suggestions/recommendations for Novell exteNd Director?

exteNd General Performance/Tuning Suggestions:

I. We recommend/suggest that all customers have three (3) separate/unique environments: Development, Staging, and Production.

A. The Development Environment

This is where the developer has the exteNd Product(s) installed on their local Windows or Novell Linux desktop machine. They create the application parts (i.e.: Portlets, Style Sheets, Themes, PageFlows, etc.) locally. They should use Director’s Vulture, so that they do not have to deploy every time they make a change. With the Vulture enabled, a Developer is able to see code changes updated on the Portal in "Real Time”.

B. The Staging Environment

The staging environment should be identical to the intended Production Environment. The same software, hardware, memory, disk space, database (i.e.: Oracle, MySQL, MS SQL, Sybase), Application Server (i.e.: exteNd, JBoss, Websphere, WebLogic) should be installed along with the same configuration (i.e.: Security Provider, Cluster/No Cluster). There should be few (if any) differences between the Staging and Production Environments. Make sure to disable all of the exteNd development aids (i.e.: Vulture in Director and Command Shell in the Application Server).

This environment is where you perform Load Testing and Memory Profiling of the application as you develop it.

Test the application as soon as possible and continue to test as you complete stages/segments of the development. It is very important to understand how your application performs initially (without any customization) to provide you with a base line for your environment. Use the Staging environment to learn what your application can and cannot handle before it is deployed to production. Load Testing determines the application server or Director configuration settings you will need in your Production environment. It will also help you determine how many instances of exteNd you need to have for the expected load in your Production environment. Memory Profiling will allow you to learn how much memory the Application needs to run with. Also, it will allow you to see if there are any memory leaks in the code.

Use the Staging Environment to test any patches you receive from Novell, the Database Server you are using, Security Provider, or even your Operating System. Do not roll out any patch or change to Production environment without testing fully in the Staging environment.

C. The Production Environment

The application is deployed for end users in the Production environment. The application in the Production environment has all of the settings and configurations adjusted based on what you learned from your tests in the Staging Environment.

**Note, Novell exteNd Director is a framework and as such it is not set-up Out of The Box in an optimized performance state. You have to Load Test, Memory Profile and tune accordingly. This product should never go into production without having been Load Tested and Memory Profiled based upon your use case (i.e. 500 concurrent users).**

II. Since Novell exteNd Director is a J2EE application/framework, we also strongly recommend that all customers make back-ups of their environments. To accomplish this, Customers should include the following in their set-up:

1. Source Control for the Director Project

2. Routine Database back-ups of the SilverMaster database(if deployed to exteNd)

3. Routine Database back-ups of the Director Database.

4. Portal and CM exports (These are necessary in case the Database server or the backup itself has become corrupt. These exports also help Support should a debug session be necessary. .)

III. How many Portlets should be on any given Portal Page?

This is a very good question and the answer is, "It Depends”. That is because you may have six portlets that display text that is hard-coded (i.e.: the Message Portlet). They do not take a lot of time to render. You might also have portlets that are retrieving data from other applications. In this case, timing becomes very important. You might need to make a trade-off between the amount of time it takes for a page to display and the number of portlets to have a on a page. The amount of time that one back-end application takes to provide data might be longer than another. This amount of time, will affect the amount of time that the Portal takes to display the page.

Security can also impact load speed. If you have portlets on a page that have security applied (for example, they do not display or run if a user does not have permission) this can also add time to the Portal page displaying. It is recommended that you test the amount of time it takes a page to display by adding one (1) portlet at a time. Record the amount of time it takes for both the portal to display and the back-end application to complete its work. Testing in this manner could help determine why some pages perform better than others.

It also allows you to take a closer look at individual portlets. You could discover the back-end application takes a long time, which is why the portlet is slow. If the back-end application is not the issue, then it would help lead towards a Code/Application Review to make sure that there is not another way to accomplish the goal of the portlet. This could lead to an issue/limitation with the Director product.

You can use all of this data to determine the best possible decision about how many and which portlets should be on a page

IV. What are the best types of Groups to use with Director?

Again, this a good question that is answered by "It Depends”.

Static/Normal groups are faster with Director. This is because membership to Static Groups is cached by Director. Therefore, you do not have all of the calls back to the Security Provider (for example, LDAP, SilverMaster database, NT Security) every time a user’s membership has to be check to display a Page or a Portlet. The information is contained within Director’s cache so the call is extremely fast. However, there are a couple of negatives to Static/Normal Groups. First, since Director caches Group Membership if the membership is changed, then you must either flush the Director’s cache or restart the application for the changes to be seen in Director. The second issue is that Static Groups require the Security Provider Administrator to manually maintain group membership.

Dynamic Groups are easier for the Security Provider Administrator to maintain, because they only have to define what user attribute makes a user a member of the Dynamic Group. There is no real manual process that the Administrator has to perform to update the membership of a Dynamic Group. However, Dynamic Groups cannot be cached. Since Dynamic Groups are suppose to be created on demand, this causes an overhead of having to create and query the group to confirm membership as it relates to Director. If you have Pages or Portlets that have security set to Dynamic Group(s), then Director is forced to query the Security Provider (eDirectory at this time) every time a Page or Portlet is rendered to confirm if the logged in user has access rights. As a result, there will be a performance hit with Dynamic Groups compared to using Static Groups, since each time security permission has to be checked a call to the Security Provider is performed. Novell exteNd Director 5.x does not perform well with Dynamic Groups. When deciding which types of Groups to use for Pages and Portlets within Director you will need to understand the nature of the group, will the membership change a lot, as well the performance trade-off between Static and Dynamic Groups.

5. The Session Time-out setting in the exteNd Application Server controls how long exteNdkeeps a User’s session alive when they are not active on the page. When the time-out is reached they are forced to login again. There is a trade-off between resources and end-user satisfaction. This means that some end-users do not like to have their browser session time out if they have not been active for a short period of time (for example, five minutes). However, server resources are allocated for each user session that is active. These resources are returned to the application server or back-end server in two ways.

• When a work requested has been completed and therefore released in your java code

• When the user session has timed-out.

Increasing the Session Time, can affect the overall performance of the Application Server. It is therefore important to understand which resources are controlled by User Sessions and which ones are controlled by your code. Make sure that application resources are returned to the server as soon as possible and having Session Time-out not set so low that your end users continually have to re-login.

VI. How many Client Connections should be set in the Application Server. This should only be set to a number (about 10%) greater than any given Peak Load you have against that Server. If you are running in a Clustered environment you need to take that into account. Be aware of the amount of Client Threads that can be opened on the Operating System the application runs on. There are limits to the number of Threads that can be created on an O/S. Please consult with your O/S administrator or Provider about this. Also please review the Administering the Server Section of the Administrator's Guide for the exteNd Application Server. A key area to review and understand is CHAPTER 11 "Tuning the Server".https://www.novell.com/documentation/extend52/Docs/Start_AppServer_Help.html

VII. SUN's VM:

A. Review the documentation on SUN's site about how to tune the JDK. You will learn about settings that make the JDK perform better under a load. According to SUN's site it is best to set your Min and Max JAVA Heap to the same level, so that the JRE does not have to "think" if it needs to take more memory as the load increases. Note that increasing the Java Heap may not solve the problem, but rather delay it. Based upon the Load Testing and Memory Profiling, you may need to set the Java Heap to a specific value to accommodate the Load of your application.

B. As a rule, the higher you set the Java Heap, the less Garbage Collection ( GC) occurs and when it does occur it takes longer. Therefore, if you are under a load and GC tries to use some resources to perform its job, you can experience a " freeze" of the Application Server until the GC has completed. This is expected, because of the load and the amount of "stuff" that GC is collecting and taking care. When you have lower Java Heap being used, you have more frequent GC. Because they are more frequent you do not notice a "Freeze" since there is less stuff to deal with. You can run with less Java Heap (i.e.: 768m or 1024m). You can set-up a cluster to help distribute the load across x number of servers. Two exteNd servers will perform better than one exteNd server. Each exteNd server needs to be on its own machine as well as the database server having its own machine.

The exteNd Application Server never calls System.gc(). We let the VM do all the work. You should never call to System.gc() in your code because it could cause problems with the VM’s GC algorithm. Also, sometimes you can"out run" the garbage collector by (1) fetching too much data (i.e.: Select * from %table%) and by Not releasing objects in a timely manner.

C. The following link has good information about Tuning GC with the 1.4 JRE:

http://java.sun.com/docs/hotspot/gc1.4.2/index.html#3.2.%20The%20Young%20Generation|outline

D. Just because a machine has 4 Gigs of RAM does not mean the Application Server can use it. The Application Server can only use the amount of JAVA HEAP you set a start-up with the Xmx setting. It can not take more than what is set. Therefore, the amount of RAM that a machine has, is only important in determining the amount of java HEAP you can set. There is a limit to the amount of HEAP that you can set and is set by the JRE. Regarding the Java Heap in the 1.4 JRE. The amount of Java Heap that you can set is dependent upon the amount of RAM of your machine and which operating system (32 or 64 bit) you are running on:

1) On 32-bit x86 architecture, the maximum size of any process is ~2GB. This is a HW limit tied to the address space, not a VM limit, and not related to physical memory (even with 16 GB RAM installed you can only get 2GB per process). It doesn't matter if it's a Java or C program, you just can't get any more than this. That is total process size, including stack and heap segments. As you increase the heap you reduce the max # of threads you can spawn. Extend uses a lot of threads so be careful about setting the heap size too high. If you really need more than 1GB heap it's time to start thinking about clustering the server.

A couple of helpful URLs:

http://www.unixville.com/~moazam/
http://www.unixville.com/~moazam/2004/06/03.html

http://forum.java.sun.com/thread.jspa?threadID=584329&tstart=105

2). On 64-bit SPARC, PPC, Opteron, etc., the address space is huge (like 10 billion GB) so the real limit becomes physical memory and swap space. Other limits (i.e., ulimits) are imposed at the OS level. But the 1.4 VM itself does not impose any limits at all. That's not to say that it scales well up to huge amounts of memory, but again, this is usually an indication that something has gone wrong.

http://java.sun.com/j2se/1.4.2/docs/guide/vm/

http://www.theserverside.com/discussions/thread.tss?thread_id=20157

VIII. In Production you need to disable the following:

A. All unnecessary system logging/debugging (i.e.: system.out.println). Set up your code so that you can enable a flag to "turn on” your debug code. It is important to have your debug when when trying to narrow down a problem. Also, if system.out.println call gets "stuck” your Application Server hangs.

B. If you are running on the exteNd Application Server, disable the commandshell utility in the httpd.props file. This is required when you are running in Production, as a Windows Service or a Daemon on Unix/Linux. Some customers have seen that the Application Server will use extra CPU with just having the CommandShell enabled. This explained in Public KB 10090588, https://support.microfocus.com/kb/doc.php?id=10090588. Even though this was written against Application Server 4.x, it still applies to 5.x of the Application Server.

C. Also related to CommandShell Utility for the Application Server, you need to make sure you have disabled the Vulture in the Director Project. The Vulture is resource intensive, but it has an important role during development. The Vulture scans the folders under /data within your project every couple of seconds and then "hot" load java, xml, xsl, images, ... to the server so that you do not have to continually deploy the project to see your changes. This allows you to develop and see your code in the Portal without having to deploy after each change. However, in Production you do not want this resource running and taking up available resources scanning the file system. Public TID, 10093025 outlines how to disable the Vulture and enable the ResourceSet jar: https://support.microfocus.com/kb/doc.php?id=10093025