Log4j configureAndWatch () spawns thousands of threads
So we have a J2EE application using Log4j like this
public class CustomerController
{
private static Logger logger = Logger.getLogger(CustomerController.class);
public CustomerService customerservice = null;
public CustomerController() throws Exception
{
PropertyConfigurator.configureAndWatch("c:\log4j.property", 50000);
customerservice = ServiceManagerSingleton.getCustomerServiceInstance();
}
}
This way we can change the real time at the log level. Very comfortably. Most of our classes are configured the same as this controller. We use singleton-pattern so that we only have one instance of the eash class; one call to PropertyConfigurator.configureAndWatch () once per class.
Problem: About twice a week our app server dies and creates a heapdump. Using the Heap Analyzer from IBM we can see that there seem to be a lot of threads related to Log4j:
808 (0%) [200] 9 org/apache/log4j/PropertyWatchdog 0x14282ad8
About 30,000 in total. So this is probably the reason for the sudden crash.
- Are we setting it up correctly?
- What happens to all these threads when EAR is reallocated?
source to share
Indeed, you need to use your application server startup routine (they are all different) to initialize the log4j system. Since Log4j relies on static variables, it cannot run independently in its ear (it might, but it really depends on the application server). In most cases, the configuration will indeed be global for the entire application server.
You must make sure the PropertyConfigurator.configureAndWatch method is called only once. One way to do this is to put something in JNDI.
Many of these will depend on what the application server gives you. For example, we are using JBoss and Log4J is configured as part of it and you just modify the log4j.xml file to include what you need. JBoss makes sure this is done dynamically.
EDIT: Here are the instructions for Websphere to create a custom service, and inside it you will create the log4J config and take your control over the file. A few caveats. You will have to add log4j.jar to the classpath of the application server itself to be war or ear accessible (I'm sure this will work anyway) and the custom service will most likely not work inside the ear.
Here is an alternative that will keep everything at war or ear, but at the cost of dynamically loading log changes.
source to share
How often are CustomerController instances created? Once per request? As I believe configureAndWatch () will create a new thread with every call.
Also, if you don't know, the log4j docs cautions against using this feature in a J2EE environment:
Because configureAndWatch starts a separate wathdog thread, and because there is no way to stop that thread in log4j 1.2, the configureAndWatch method is unsafe for use in J2EE environments where applications are being processed.
I know you are not using Spring, but in my opinion the Spring Log4jWebConfigurer class has a more detailed explanation of why this feature is dangerous in J2EE:
WARNING: Log4j watchdog does not exit until VM shutdown; in particular, it does not end when the LogManager is disabled. Therefore, it is recommended not to use the update of the configuration file in a production J2EE environment; watchdog thread does not stop when the application is closed.
Update: Looking at the log4j source, each call to configureAndWatch () will indeed create a new thread .
source to share
Have you looked at logback, the log4j successor? It deals with reloading its configuration in a thread or via JMX . Both approaches avoid the headaches caused by calling PropertyConfigurator.configureandWatch () or DOMConfigurator.configurator. It is also interesting that in the streaming approach it is enabled via a config file (no special code required).
source to share
All registrars use the same configuration file, so if every class that uses the registrar contains this initialization code, each call to configureAndWatch () will most likely spawn a new stream of watchers. (IMHO Log4j should know better and allow at most one Watcher thread in the config file, but apparently it doesn't)
source to share