JAMon (Java Application Monitor) Users Guide

A Monitoring API

JAMon comes with several ways to monitor your application that require no application code changes. Usually all that is required is to add a line to an xml file, and put the jamon jar file in your classpath. You could be monitoring your code in a few minutes.

  1. JAMon is now on Maven. The Maven pom dependency follows (substitute the appropriate version):
  2.     <dependencies>
            <dependency>
                <groupId>com.jamonapi</groupId>
                <artifactId>jamon</artifactId>
                <version>2.78</version>
            </dependency>
        </dependencies>
        
  3. JAMon Videos - Short YouTube videos showing how to use JAMon
    1. Monitoring Distributed Applications with JAMon
    2. JAMon 101 - An intro video to JAMon. Shows how to use start/stop/add and also other highlights of using JAMon.
    3. JAMon GUI - Shows how to access the information in JAMon via the JAMon WAR. Also shows data captured by the JAMon servlet filter and JDBC Proxy driver.
    4. JAMon Operating System monitoring - Monitor your unix/linux/mac scripts with JAMon. Introduced in JAMon 2.72.
  4. Quick Links
    1. Download JAMon
    2. Get support for JAMon
    3. JAMon JavaDocs
    4. JAMon Source Code
    5. JAMon Slides (html)
    6. JAMon 2.2 Slides (powerpoint)
    7. FormattedDataSet HomePage
    8. Donate to JAMon
  5. Monitoring Spring using JAMon and AOP (2.76)
  6. What's new in JAMon 2.72 and 2.73?
  7. What's new in JAMon 2.71?
  8. What's new in JAMon 2.7?
    1. HTTP Monitoring (Tomcat, Jboss, Jetty, Servlet filter)
    2. Monitoring EJB's with the JAMon EJB Interceptor
  9. What's new in JAMon 2.6? -Introducing Log4j JAMon Appenders
  10. What's new in JAMon 2.4 and 2.5 - JAMonListeners
  11. What's new in JAMon 2.2 and 2.3?
    1. The JAMon JDBC Proxy driver
  12. What's new in JAMon 2.1?
    1. Monitoring Interfaces
    2. Monitoring Interfaces - Method Calls (JAMon Summary Statistics)
    3. Monitoring Interfaces - Exceptions (JAMon Summary Statistics)
    4. Monitoring Interfaces - Exceptions (Exception Details)
    5. Monitoring Interfaces - Note on Exception Monitoring
    6. Monitoring Interfaces - JDBC Interfaces
    7. Monitoring SQL Commands
    8. Monitoring SQL Commands (JAMon Summary Statistics)
    9. Monitoring SQL Commands (SQL Details)
    10. JAMon 2.1 Miscellaneous
  13. What's new in JAMon 2.0?
    1. What is JAMon 2.0?
    2. Aggregation is Information/Details are Noise
    3. JAMon 1.0 - Good
    4. JAMon 2.0 - Better
    5. What else is JAMon?
    6. How can I use JAMon?
    7. Specific examples may help you understand the possibilities
    8. How do I use the new MonitorFacatory.add(...) method?
    9. What other improvements were made in JAMon 2.0?
    10. Incompatibilities with JAMon 1.0
    11. JAMon challenge #1! Your Help is needed!
    12. JAMon challenge #2! Your Help is needed!
  14. Advantages of JAMon
  15. The JAMon servlet filter - A no code solution to monitoring your Java web application in 5 minutes!
    1. The JAMon Web Application (WAR) - Viewing JAMon stats via jamonadmin.jsp
  16. JAMon examples
    1. Simple Example
    2. Monitoring a Servlet - Gathering page statistics
  17. Managing JAMon - jamonadmin.jsp
    1. The JAMon Report
    2. Measuring outliers (JAMon Time Ranges)
    3. Controlling JAMon
  18. Ways to use JAMon
  19. Future Directions
  20. Sample Code
  21. Downloading JAMon
  22. Support
  23. JAMon/FormattedDataSet T-Shirts
  24. JAMon License Agreement
  25. Acknowledgements/Contributors
  26. About the Author


JAMon Introduction

The Java Application Monitor (JAMon) is a free, simple, high performance, thread safe, Java API that allows developers to easily monitor production applications.

  • JAMon can be used to determine application performance bottlenecks, user/application interactions, track application scalability, and more.
  • JAMon 1.0 gathers aggregate performance statistics such as hits, execution times (total, average, minimum, maximum, standard deviation), as well as concurrency information such as simultaneous application requests.
  • JAMon 2.7 comes with several ways to monitor your application that require no application code changes. Usually all that is required is to add a line to an xml file, and make the jamon jar file available. Some examples:
    • JDBC/SQL monitoring via the JAMon JDBC proxy driver (works with any other JDBC driver),
    • Log4j monitoring via the JAMon log4j Appender,
    • HTTP Monitoring. Monitor requests/responses to Tomcat, Jetty, JBoss or any other JAVA web container.
    • EJB3 monitoring using the JAMon EJB interceptor.
  • JAMon 2.0 aggregates ANY numeric entity (such as bytes sent, dollars purchased, phone call duration) associated with a key (label and units).
  • JAMon 2.1 adds the ability to track both summary statistics and detils for Interfaces/JDBC/SQL/Exceptions.
  • JAMon 2.2 adds the ability to track JDBC/SQL/Exceptions vi the new JAMonDriver JDBC Driver using a URL (not changing code)
  • JAMon 2.3 minor bug fixes
  • JAMon 2.4/2.5 Introduction of JAMon Buffer Listeners to track details of interest such as extreme values, or most recent values
  • JAMon 2.6 log4j monitoring. Track stats on log4j method calls, as well as have the ability to 'tail' data written to log4j through the JAMon web application.
  • JAMon 2.7 Add the ability to flexibly monitor Tomcat, Jetty, Jboss. More flexible detail tracking was added. Also better exception tracking was added.
  • JAMon data can also be programmatically accessed or queried and your program can take actions on the results.
  • JAMon statistics are displayed in the clickable JAMon Report.

JAMon was developed primarily for monitoring J2EE applications, however JAMon 1.0/2.0 can be used in any JDK 1.2 or higher environment, and JAMon 2.1 can be used in any JDK 1.4 or higher environment. JAMon can be used in Servlets, JSP's, EJB's and Java Beans in various J2EE Application Servers (Sybase's EAServer, and BEA's WebLogic, WebSphere�), and can also be used in other programming environments that can call Java code (ColdFusion, PowerBuilder, BroadVision, ...). JAMon is particularly useful in multithreaded software like a database engine.

Feel free to continue reading the user's guide or download JAMon and read the Java Docs.

Note: The FormattedDataSet API generates dynamic text including HTML and XML. It uses JAMon extensively to monitor performance including all JDBC calls. The FormattedDataSet is a powerful open source API in its own right and is also a great primer in using JAMon. Click this link to learn more about the FormattedDataSet.

Note: Possible WebSphere issue with jamonadmin.jsp.



1. What's new in JAMon 2.7? HTTP and EJB Monitoring

JAMon 2.7 introduces the ability to monitor HTTP requests in web containers such as Tomcat, JBoss, and Jetty. This monitoring is achieved with no code changes required by developers. See HTTP Monitoring with JAMon for more information.

JAMon 2.7 also introduces EJB monitoring. Any EJB3 methods as well as any exceptions they throw will be monitored. The information can be viewed in /jamon/jamonadmin.jsp. EJB requires no coding changes to your EJBs. Simply follow these steps to enable the EJB interceptor on your app server:

  • Make jamon-2.7.jar (or higher) available to your app server
  • Install jamon.war (or higher), so you can view the statistics.
  • Put the following information in your ejb-jar.xml file. This will monitor all EJBs. You may
    • <?xml version="1.0" encoding="UTF-8"?>
      <ejb-jar version="3.0" xmlns="http://java.sun.com/xml/ns/javaee" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://java.sun.com/xml/ns/javaee http://java.sun.com/xml/ns/javaee/ejb-jar_3_0.xsd">
       <assembly-descriptor>
         <!-- Default interceptor that will apply to all methods for all beans in deployment -->
          <interceptor-binding>
           <ejb-name>*</ejb-name>
           <interceptor-class>com.jamonapi.aop.JAMonEJBInterceptor</interceptor-class>
         </interceptor-binding>
       </assembly-descriptor>
      </ejb-jar>


2. What's new in JAMon 2.2 and 2.3?

JAMon 2.3 fixes a few bugs

  • Made a change to MonProxy.equals method so it now works properly with DBCP in JBoss
  • Added MonProxy.getMonitoredObject(obj) static method that returns the original object being monitored
  • Fixed synchronization bug in FactoryEnabled
  • Fixed synchronization bug in FactoryEnabled
  • Added a percent range
  • Fixed bugs that made it difficult for developers to user there on monitor keys

JAMon 2.2 fixes a few bugs and adds the new JAMon JDBC driver which allows developers to monitor JDBC/SQL without changing any code. The JAMonDriver class is com.jamonapi.proxy.JAMonDriver and representative URLS follow. Note in the jamonrealdriver below the actual driver is used.

  • jdbc:jamon:sybase:Tds:localhost:2638?jamonrealdriver=com.sybase.jdbc2.jdbc.SybDriver
  • jdbc:jamon:sybase:Tds:myserver.mycompany.com:2638?jamonrealdriver=com.sybase.jdbc2.jdbc.SybDriver&PROTOCOL_CAPTURE=/opt/sw/eas/eas5/sched.log
  • jdbc:jamon:opena:jamonrealdriver=com.asgardsw.OpenA.OpenADriver&159.234.234.164:5000/USR/DB/ (OPENA)CONFIG/AMWEB/DEV ON SOFTWARE

JDBC Syntax Without JAMon: An example of how to connect to a database using a JDBC driver follows (The example uses Sybase, but any vendors driver would do)

Class.forName("com.sybase.jdbc2.jdbc.SybDriver");
String url = "jdbc:sybase:Tds:MyServerDNS:2638";
Connection conn = DriverManager.getConnection(url, user, passWord);
JDBC Syntax With JAMon: The following code will use JAMon as a proxy to track performance, but uses the underlying driver to perform the queris (The example uses Sybase, but any vendors driver would do). Note no code changes would be required if connection strings were placed in a configuration file. The JAMon jar file must also be placed in the classpath.
Class.forName("com.jamonapi.proxy.JAMonDriver");
String url = "jdbc:jamon:sybase.Tds:MyServerDNS:2638?jamonrealdriver=com.sybase.jdbc2.jdbc.SybDriver";
Connection conn = DriverManager.getConnection(url, user, passWord);

Note JAMon simply removes 'jamon', and 'jamonrealdriver=XXXX' from the url, so the url should work with the underlying driver when they are removed. Note the '?' above is not part of JAMon syntax, but instead part of the underlying Sybase driver syntax. In the Oracle example below, there is no delimiter between 'jamonreaddriver' and the previous part of the oracle driver, which looks odd, but it is appropriate.

Class.forName("com.jamonapi.proxy.JAMonDriver");
String url = "jdbc:jamon:oracle:thin:@dbms:1521:DBjamonrealdriver=oracle.jdbc.OracleDriver";
Connection conn = DriverManager.getConnection(url, user, passWord);

Bug Fixes:

  • Previously 'MonProxyFactory.monitor(object);' could only monitor interfaces that were immediately implemented by the passed in Object. Now any interface in the objects heirarchy can be monitored.
  • The Monitor class inadvertently had 'getLabel()', and 'getUnits()' removed from the last release, and they were returned in this release.


3. What's new in JAMon 2.1?

JAMon 2.1 introduces two easy to use, but powerful new capabilities:

  • The ability to monitor the method calls of ANY Java interface
  • The ability to monitor SQL statments and JDBC calls. SQL/JDBC montoring does everything that interface monitoring does and then some. Note JAMon allows ANY vendors JDBC driver to be monitored (Sybase, SQL Server, Oracle, MySQL, HSQLDB, etc). The current version of JAMon requires that each database conneciton be wrapped by one line of code. The next release will remove this limitation and allow monitoring to be enabled via a JDBC connection URL or DataSource.

JAMon can be enabled/disabled at runtime. If it is disabled then the original object is simply returned, causing no run time monitoring performance. Having said that the monitoring is quite fast and can be enabled in most production systems.

This version of JAMon requires JDK 1.4 or higher. Previous versions were JDK 1.2 or higher.



3.a Monitoring Interfaces

With one line of code you can monitor ANY java interface. Any interface means you can easily monitor standard java interfaces such as JDBC, open source interfaces such as the log4J appender interface or your own custom interfaces. To monitor an interface you simply have to call the following JAMon method:

import com.jamonapi.proxy.*;

// The MyObject() class has to implement MyInterface for monitoring to work.
MyInterface myObject = (MyInterface) MonProxyFactory.monitor(new MyObject());
myObject.myMethod();// method call will be monitored with JAMon.

That's it! Simply make method calls to myObject as you normally would, and the calls will be monitored. (Note it is best to hide the use of JAMon in a monitoring class of your own creation).



3.b Monitoring Interfaces - Method Calls (JAMon Summary Statistics)

JAMon summary statistics such as hits, time statistics (avg,total,min,max), concurrency statistics and more will be tracked for all of methods of the interface. The JAMon label for monitoring of interface method calls always begins with: MonProxy-Interface. JAMon statistics are viewable via the jamonadmin.jsp page in jamon.war (discussed later). The following is an example of the JAMon label associated with the Connection classes close() method. Note the JAMon label consists of the concrete class as well as the interfaces method signature.

MonProxy-Interface (class=org.hsqldb.jdbcConnection): public abstract void java.sql.Connection.close() throws java.sql.SQLException

Interface summary statistics are viewable via jamonadmin.jsp.



3.c Monitoring Interfaces - Exceptions (JAMon Summary Statistics)

JAMon summary statistics are kept whenever a monitored interface throws an Exception.

Any time a monitored interface throws an exception several JAMon summary records will be added. JAMon labels for Exception summary statistics begin with: MonProxy-Exception. These summary statistics allow developers to easily see how many Exceptions the application has thrown. For each exception that the interface throws the the following three types of JAMon summary labels will appear in the report.

  • How many Exceptions were thrown by all monitored interfaces. This allows you to easily see how many Exceptions your applications have thrown as well as when it last threw one - MonProxy-Exception: InvocationTargetException
  • How many Exceptions of each exception type was thrown. For SQL exceptions the ErrorCode and SQLState are also added. This will be discussed further below. Example - MonProxy-Exception: Root cause exception=java.sql.SQLException,ErrorCode=-22,SQLState=S0002
  • How many exceptions were thrown by each method of the monitored interfaces. Example - MonProxy-Exception: (class=org.hsqldb.jdbcStatement) Exception: public abstract java.sql.ResultSet java.sql.Statement.executeQuery(java.lang.String) throws java.sql.SQLException

Exception summary statistics are viewable via jamonadmin.jsp.



3.d Monitoring Interfaces - Exceptions (Exception Details)

The details of the N most recent Exceptions thrown by monitored interfaces (including the stack trace) are kept in a rolling buffer and viewable via exceptions.jsp. The exception buffer size is configurable via the web page, and defaults to the most recent 50 exceptions. Typically such stack trace information is only available in a log which a developer might not have access to and even if they do the log must be parsed to view the stack traces.

Each row in the report represents a recent Exception thrown from a monitored inteface. Each row contains the following columns

  • ID - An increasing integer that indicates the number of the exception since the server was last booted.
  • StartTime - The time the exception was thrown
  • ExceptionStackTrace - The stack trace of the exception.
  • MethodName - The method name of the monitored interfaces method that threw the exception

Exception details statistics are viewable via exceptions.jsp.



3.e Monitoring Interfaces - Note on Exception Monitoring

One nice thing about Exception monitoring is that even when developers gobble/hide exceptions in the following manner they will show up in the JAMonAdmin report and exception detail report. The following example is taken from the JAMon demo.

  // Throw an exception and show that it is also monitored in jamonadmin.jsp and exceptions.jsp
  // Note also even though the catch block is empty it will show up in these pages.
  try { 
   // get a query to throw an Exception.  If enabled will show in jamon report and sql details.
   st.executeQuery("select * from i_do_not_exist");  

  } catch (Exception e) {} 


3.f Monitoring Interfaces - JDBC Interfaces

JDBC interface monitoring has all the capabilities mentioned above, but also a few extras.

  • Monitoring for Connection/Statement/PreparedStatement/CallableStatement/ResultSet JDBC interfaces does not require casting of the returned monitoring proxy.
    	import com.jamonapi.proxy.*;
    
    	// The MyObject() class has to implement MyInterface for monitoring to work.
    	Connection monitoredConnection = MonProxyFactory.monitor(originalConnection);
           
  • If a Connection is monitored then any Statement/PreparedStatement/CallableStatement/ResultSet returned is automatically monitored, and need not be wrapped by a call to monitor(...) itself
    	import com.jamonapi.proxy.*;
    
    	// The MyObject() class has to implement MyInterface for monitoring to work.
    	Connection monitoredConnection = MonProxyFactory.monitor(originalConnection);
    	// The following is not needed to monitor a Statement.  The statement is automatically monitored due to its connection being monitored
    	Statement statement=MonProxyFactory.monitor(monitoredConnection.createStatement());
            
  • Due to the standardized nature of SQLExceptions they have more information in the JAMon summary report than other Exceptions. They include SQLError and SQLState information. Example - MonProxy-Exception: Root cause exception=java.sql.SQLException,ErrorCode=-22,SQLState=S0002


3.g Monitoring SQL Commands

One of the most powerful aspects of JAMon 2.1 is that it can track performance for all queries issued by an application (see 'Monitoring SQL Commands - JAMon Summary Statistics') and allow developers to view all recent queries that have executed in a rolling buffer (see 'Monitoring SQL Commands - Details').



3.h Monitoring SQL Commands (JAMon Summary Statistics)

A number of different JAMon summary statistics will be kept for each query executed from a monitored Statement/PreparedStatement/CallableStatement. JAMon labels for SQL summary statistics begin with: MonProxy-SQL. These summary statistics allow developers to see how many times a query has been executed, what its time statistics are (avg/total/min/max/...), whether or not it is currently executing and if so how many times, and more.

For each SQL command the following JAMon summary labels will appear in the JAMonAdmin report.

  • MonProxy-SQL-Type - Statistics for SQL commands of the following types: select, update, delete, insert, truncate, exec, create, drop, alter, commit, save, grant, revoke, and other. Essentially this is the first word of the SQL command, and other 'types' may be added by developers via the JAMon API. This monitor allows developers to see statistics (hits/avg time/concurrency/...) for select's/update's/delete's etc. Example - MonProxy-SQL-Type: select
  • MonProxy-SQL-PreparedStatement - This monitor will give statistics for every SQL command issued against a PreparedStatement/CallableStatement. Example - MonProxy-SQL-PreparedStatement: select * from SYSTEM_TYPEINFO where LOCAL_TYPE_NAME=?
  • MonProxy-SQL-Statement - This monitor will give statistics for every SQL command issued against a Statement. To allow aggregation all SQL argument values are replaced with '?'. For example, "select * from employee where name='steve'", and "select * from employee where name='jeff'" both become "select * from employee where name=?". Example - MonProxy-SQL-Statement: select * from employee where name=?
  • MonProxy-SQL-PreparedStatement Reuse - Counts how many times PreparedStatements are reused. One of the benefits of PreparedStatements is improved performance over Statements if they are reused. This monitor allows developers to see if they are reusing their PreparedStatements. Note this is an overall number and it is not tracked for individual queries.
  • MonProxy-SQL-Match - The API allows developers to pass in any string and have a monitor show up if the string occurs in a query. The most obvious use of this is to pass in table names and see how many times the table occurs in queries and what the performance characteristics of these queries is. This can lead the way to detecting problem tables. One row will show up in the JAMonAdmin report for each match. Example - MonProxy-SQL-Match: SYSTEM_TYPEINFO

    SQL summary statistics are viewable via jamonadmin.jsp.



3.i Monitoring SQL Commands (SQL Details)

The details of the most recent N SQL commands issued are kept in a rolling buffer and are viewable via sql.jsp. The SQL buffer size is configurable via the web page, and defaults to the most recent 100 queries.

Each row in the report represents a recently executed SQL command. By default the report is sorted with the most recent queries showing at the top. Each row contains the following columns

  • ID - An increasing integer that indicates the number of the query since the server was last booted.
  • StartTime - The time the query started
  • ExecutionTime - The time in milliseconds that the query took to run.
  • StatementReuse - Indicates how many times the PreparedStatement was reused. This value should be high to maximize the benefit of the PreparedStatement. The first time the PreparedStatement is used it will have a value of 0 and will increase with each reuse of the PreparedStatement. Statements will always have a value of 0 here as they aren't reused.
  • SQL - The query that executed. For statements it will include the parameter values. PreparedStatements will have '?' in place of values.
  • ExceptionStackTrace - If an error happened when executing the query this column will contain the stack trace. If no error occurred it will be empty.
  • MethodName - The JDBC method name that executed the query.

SQL details are viewable via sql.jsp.



3.j Live Demo

A live demo is available that demonstrates Interface/JDBC/SQL monitoring. The live demo is also the application that you will use to view JAMon stats in your own environment. Simply install jamon.war and place the jamon jar file in your servers class path (Note it should be put in your servers class path so you can monitor other apps).

3.k JAMon 2.1 Miscellaneous

  • Enabling/Disabling at runtime - JAMon and MonProxy can be enabled/disabled at runtime. In fact when totally disabled there is NO PERFORMANCE IMPACT whatsoever, as the actual object is returned and not a monitored object.
  • MonProxy Performance - Even though you can turn off monitorying at runtime I would suggest that it be kept on at all times. There are various levels of monitoring that you can enable, so at a minimum I would monitor Exceptions and SQL summary statistics. The com.jamonapi.proxy.MonProxyTester class performs various performance tests. The test executes 6000 queries agains HSQLDB for each test case. When monitoring is completely disabled this takes 500 ms., and when it is completely enabled it takes about 1000 ms. Disabling other capabilities falls somewhere in betweent the two. Your mileage will vary so I encourage you to test yourself. The query time for 6000 queries on my computer was faster than I would expect in most applications, and as query time goes up I would expect monitoring time to remain constant and so have a smaller impact on query performance.
  • Web Page Features - All the jsp's in jamon.war have the ability to truncate strings for display. This can be needed as some query and exception strings can be to long for display. This truncation is purely for display and the underlying data is not truncated. In addition every page has the following capabilities: sort, query data, highlight text of interest.
  • Note in this release I changed the name of jamon jar, and the case of the web app, and jamonadmin.jsp file. They are now respectively: jamon-2.1.jar, /jamon/jamonadmin.jsp (from /JAMon/JAMonAdmin.jsp)

4. What's new in JAMon 2.0?

The following subsections describe JAMon 2.0, and how it can be used to improve your programs.



4.a What is JAMon 2.0?

JAMon 2.0 can be thought of as an in-memory aggregate database. Most databases store the details (rows) and these details can be aggregated/summarized by writing SQL (using the �group by� clause). JAMon takes the approach of throwing out the details and keeping the aggregates. This allows JAMon to be very fast and have a small memory footprint.



4.b Aggregation is Information/Details are Noise

Often people make decisions based on summary information (aggregates) and not detailed information. Imagine trying to figure out the performance of a particular query by looking through a multi-gigabyte log file searching for the occurences of the query (buried among other log details). Trying to understand this data would be impossible without aggregation. Using aggregates such as averages, mins and maxes would be much more useful to see if the query needed to be tuned.

This concept of aggregating statistics is the basis of Decision Support Systems (DSS). JAMon 2.0 allows you to easily aggregate ANYTHING that happens in your application.



4.c JAMon 1.0 - Good

JAMon 1.0 only tracked aggregate statistics for time (the units were in milliseconds/ms.). For example, if the following code was called 10 times:
   Monitor mon=MonitorFactory.start(�myPage.jsp�);
   �page code�
   mon.stop();

Representative performance statistics may look like this: 10 hits, average=100 milliseconds (ms.), total=1000 ms., minimum=50 ms., maximum=200 ms. This is very useful, but there are many things it would be useful to aggregate that are not time based (i.e. money, MB, locks, rows, �)



4.d JAMon 2.0 - Better

In JAMon 2.0 the concept of aggregating a time associated with a String has been expanded to aggregate any number associated with any arbitrary key. The key can consist of any number of Objects. Here are some examples of possible JAMon 2.0 keys:

  • Probably the most common key you will use is: key=label, units. In JAMon 1.0 key=label, ms. was a special case of this more general rule. JAMon 2.0 keys could be:
    • key=label, ms. (This was the only possibility in JAMon 1.0)
    • key=label, MB (example: bytessent, MB)
    • key=label, MB (example: freeMemory, MB)
    • key=label, $ (example: purchases, $)
    • key=label, Exception (example: InvalidArgumentExeception, Exception)
    • key=company, department, division (example: Sybase, HR, EEO - possibly summarizing salary)

This is a simple, but powerful concept that allows developers to expose many things that are happening inside their applications. Because JAMon is easy to use and fast, developers can now unleash the power of aggregation to improve their programs in many new, innovative ways. Start aggregating!



4.e What else is JAMon?

Here are ways that I think of JAMon that helps me uncover ways to use it. JAMon is...

  • An in-memory aggregate database - A database retains details, JAMon retains aggregates
  • An in-memory aggregate log - A log retains details, JAMon retains aggregates
  • DSS for the application - JAMon allows developers to unleash the power of statistics on the internals of their applications
  • Similar to the SQL 'group by' clause - Each call to JAMon aggregates information similar to the following select:
    • select key, count(*), sum(values), stddev(values), min(values), max(values), avg(values), min(date), max(date),mostRecentValue,�other stats too� from MyTable group by key
  • A compromise between keeping details or just aggregates (enter JAMon Ranges) - JAMon also has the concept of ranges that allows you to keep track of summary statistics for subsets of data for a particular monitor. (i.e. For example if you were tracking 'points per game' for NBA players you may have the following ranges for 0-10 points, 10-20 points, 20-30 points, 30-40 points, 50-60 points, 70-80 points, 80-90 points, 90-100 points). This can add extra information beyond basic summary data and still doesn�t consume much memory. It is a compromise between keeping all data or just summary data. Developers can define their own ranges.


4.f How can I use JAMon?

Here are a few of the many uses of JAMon.

  • Self tuning/Self managing software - Programs can interrogate JAMon statistics and take action if certain thresholds have been exceeded. For example resources such as locks, memory, and file descriptors that are usually set when a database is booted could now change on the fly based on threshholds being exceeded.
  • Self Monitoring software - Software enabled with JAMon can alert/page when threshholds have been exceeded. For example if an exception is thrown, a user fails to login, , or memory is low.
  • Tracking business data - Business events can be tracked in real-time. For example if every time a purchase is made its value is passed to JAMon we could have real time purchase stats without accessing the database.
  • Monitoring Servers and multi-threaded applications - JAMon is particularly useful in multithreaded applications like database engines, JDBC drivers, frameworks, EJBs, Application Servers, web applications (In fact the JAMon servlet filter lets you monitor web application performance without changing a line of code), and more.
  • Performance Statistics - Execution times for JDBC calls, IO, page hits and any other Java calls can all be tracked.
  • Concurrency/Scalability Statistics - What is currently executing in your application and how it affects performance (i.e. scalability) can also be tracked.
  • Understanding how users interact with your application - What are the peak hours? Is the system used on nights? Weekends? How many simultaneous users are there? Who is logged in now? What are the most popular pages? Are some pages not used?�
  • Tracking phone calls on a Java enabled cell phone - Monthly minutes, number of calls, calls to individual and more could all be tracked.

Note: Querying JAMon statistics - Between JAMon�s API for extracting the aggregate data, and the FormattedDataSet�s ArraySQL class which allows you to query the JAMon statistics in a SQL like manner, your programs can easily take action based on JAMon data.



4.g Specific examples may help you understand the possibilities

The simple syntax below demonstrates how easy it is to aggregate statistics for purchases (the units are in dollars). The number ($123.40) represents the dollar value of the individual purchase: MonitorFactory.add(�purchases�, �$�, 123.4);

If the code above was called every time a purchase was made (with 123.4 replaced by a variable) then the JAMon report would be able to show business users and developers, statistics on total purchases in real time. Using JAMon is that easy!

Here are some other ideas for things to monitor in JAMon 2.0:

  • Timing Code - This is all JAMon 1.0 could do.
    • Web Page performance: MonitorFactory.start(�pageHit�);
    • Web Page performance by user: MonitorFactory.start(�pageHit.ssouza�); (or dynamically getting user name from a variable: MonitorFactory.start(�pageHit.�+userName);
    • Method performance: MonitorFactory.start(�com.mypackage.myMethod()�);

The following measurements/numbers can only be tracked in JAMon 2.0. Strings and numbers in the examples below are hard coded, but variables can just as easily used:

  • Counting Items.
    • Counting Exceptions/Errors/Warnings in your program (a useful log4J appender could be created to do this, or a base class application exception that calls JAMon in the constructor): MonitorFactory.add(�com.jamonapi.MyException�,�Exception�,1)
    • Counting Logged in users: MonitorFactory.add(�users�,�count�,24);
    • Counting Connections opened: MonitorFactory.add(�connection�,�jdbc�, 5);
    • Counting currently opened connections
      • In the open method increment the openConnections counter: MonitorFactory.add(�openConnections�,�jdbc�, 1);
      • In the close method decrement the openConnections counter: MonitorFactory(�openConnections�,�jdbc�,-1);
  • Code metrics: MonitorFactory.add(�com.me.MyBigClass�,�lines of code�,4000);
  • Counting cache hits/misses: MonitorFactory.add(�cachehit�, �count�, 1); MonitorFactory.add(�cachemiss�,�count�,1);
  • Counting binary conditions such as true/false, yes/no, up/down, on/off etc (or more generally anything that has a limited number of outcomes): MonitorFactory.add(�ServerUp�,�boolean�,1); or MonitorFactory.add(�ServerUp,�boolean�, 0)
  • Business/Application Data
    • Money statistics: MonitorFactory.add(�heatingcost�, �$�,70);
    • Song time in minutes: MonitorFactory.add(�songduration�,�minutes�,4.5);
    • Phone call duration in minutes by phone number: MonitorFactory.add(�703-919-9000�,�phonenumber�,16);
    • Scoring average: MonitorFactory.add(�nbaPlayer�,�points�,25);
    • BlackBox type data: MonitorFactory.add(�breakingDistance�,�feet�,200);
  • Internals of a Java program. JAMon 2.0 shines when it comes to tracking data within a Java server program such as an application server, web server or database server. Here are some examples:
    • Buffer size in bytes: MonitorFactory.add(�bufferSize�,�bytes�,list.size())
    • Number of rows in ResultSets: MonitorFactory.add(�ResultSet.numRows�,�rows�,4000);
    • Number of cells in a table: MonitorFactory.add(�table�,�cells�,rows*cols);
    • Free memory: MonitorFactory.add(�Free Memory�, �MB�, Runtime.getRuntime().freeMemory());
    • Free memory With Buffer Detail Info: MonitorFactory.add(new MonKeyImp(�Free Memory�, "max memory="+Runtime.getRuntime().maxMemory()+", totalMemory="+Runtime.getRuntime().totalMemory(),�MB�), Runtime.getRuntime().freeMemory());
    • Count Updates/Deletes/Inserts/Selectcs against a Database table : MonitorFactory.add(�tablename�,�updates�,1);
    • Number of open file descriptors: MonitoFactory.add(�openFileDescriptors�,�count�,500);
    • Number of locks in use: MonitorFactory.add(�locksInUse�,�count�, 250123);
    • Backup time: MonitorFactory.add(�backupTime�,�minutes�,10);

In general you can track any numbers! A list of other ideas follows:

  • Standard Measures that have properties and units
    • property=distance, unit=mile, foot, centimeter, meter, kilometer, light year,�
    • property=weight, unit=pound, ounce, ton, gram, kilogram�
    • property=mass, unit=gram, kilogram,�
    • property=time, unit=ms., second, minute, hour, day, week, month, year, decade, century�
    • property=volume, unit=quart, gallon, litre,�
    • property=temperature, unit=degree fahrenheit, degree celsius
    • property=area, unit=square inch, square foot, square mile, square meter, square kilometer, acre,�
    • property=size, unit=byte,MB,GB,�
    • property=currentcy/money, unit=dollar, franc, peso, euro�

Other numbers you could summarize: quantity, percentage, count, total, min, max, grade, score, rating, rank, price, cost,�



4.h How do I use the new MonitorFacatory.add(...) method?

Here is an example of how you could use the new MonitorFactory.add(...) method to perform the functionality of JAMon 1.0's start/stop timer. In fact this is more or less the way this capability is implemented in JAMon 2.0. The first example does not allow you to tell if the code being monitored is currently running (i.e. active), whereas the second example would. Example 3) is the proper way to do timed monitors, the other examples are merely demonstrative.

import com.jamonapi.*;

//Example 1) Using add to mimic MonitorFactory.start("myCodeTimer").
// This approach DOESN'T allow you to tell that this code is currently
// running (i.e. active)

long startTime=System.currentTimeMillis();
...Code Being Timed...
MonitorFactory.add("myCodeTimer","ms.",System.currentTimeMillis()-startTime);
//Example 2) Using add to mimic MonitorFactory.start("myCodeTimer").
// This approach DOES allow you to tell that this code is currently
// running (i.e. active)

Monitor mon=MonitorFactory.getMonitor("myCodeTimer","ms.").start();
long startTime=System.currentTimeMillis();
...Code Being Timed...
mon.add(System.currentTimeMillis()-startTime);
mon.stop();
//Example 3) Example #2 is equivalent to this

Monitor mon=MonitorFactory.start("myCodeTimer")
...Code Being Timed...
mon.stop();


4.i What other improvements were made in JAMon 2.0?

  • Improved Performance � The following are performance characteristics when executing serially. Tests were done on my rather old IBM ThinkPad T30 Pentium IV NoteBook PC. To execute performance tests execute: "java -cp jamon-2.1.jar com.jamonapi.TestClassPerformance 1000000". All tests call MonitorFactory.start("label"), and mon.stop() the specified number of times per second.
    • JAMon 1.0 using JDK 1.2 - 195,000 calls per second
    • JAMon 2.0 using JDK 1.2 - 225,000 calls per second
    • JAMon 2.0 using JDK 1.4 - 405,000 calls per second
    • JAMon 2.0 using JDK 1.5 - 550,000 calls per second
    • JAMon 2.0 calling MonitorFactory.add(...) - 1,500,000 calls per second!
  • Improved Synchronized Performance - Improved performance when there is contention.
  • More Flexibility - JAMon 2.0 can now track ANY number associated with ANY key (not just time anymore).
  • Simplified Code - Though JAMon 2.0 can do more than JAMon 1.0 the code is simpler.
  • More accessibility of JAMon statistics � Values such as hits, avg, min, max and more are all accessible through the API now. Also the FormattedDataSet�s ArraySQL class allows developers to query JAMon data (see below).
  • Improved Reporting � Data can be queried, and displayed as XML/HTML/Excel
  • ArraySQL - Querying JAMon Data - Though ArraySQL is not part of JAMon it is used in jamonadmin.jsp and can be used to query JAMon stats in a SQL like manner (i.e. select * from array where hits>1000 and total>50000). Being able to so easily get data out of JAMon will open up new ways to leverage the JAMon monitoring data. ArraySQL is part of my FormattedDataSet API. This link goes to the ArraySQL java docs, and from here you can also view the code. If you select the 'view code' link from the java docs and look at the code in the main method, you will see various examples of the ArraySQL syntax.
  • User Defined Ranges



4.j Incompatibilities with JAMon 1.0

Due to the fact that JAMon 2.0 was a significant rewrite, quite a few (the majority) of JAMon 1.0's classes are no longer there, and the methods of some classes have changed. If you just use start()/stop() and jamonadmin.jsp this won't affect you. In general it is a good idea to hide calls to JAMon (or any other 3rd party software) in classes that you own. What follows is a list of some of the major changes:

  • The Monitor.toString() method format is different
  • In general all methods have been deprecated and perform noops in this release
  • MonitorFactory.getComposite(String) previously queried the data and now it takes a range name for an argument
  • MonitorFactory.getReport() previously threw an Exception
  • MonitorFactory.getData() previously returned a String[][], but now returns an Object[][]
  • MonitorFactory.getReport() and MonitorFactory.getData() previously returned range data and now they do not.
  • MonitorFactory debug capabilities have changed.
  • Many classes that probably weren't used, but were not explicitly restricted access to have been removed.
  • Previously key names were case insensitive (i.e. MonitorFactory.start("PAGE") was the same as MonitorFactory.start("page")). A case insensitive map can be supplied to MonitorFactory, however to provide this behaviour.
  • Many methods in MonitorComposite were removed.
  • Being as JAMon 2.0 was a complete rewrite many of the classes that should have been hidden as implementation details (live and learn) were removed. It is unlikely that developers used these classes directly though possible.



4.k JAMon challenge #1! Your Help is needed!

I would like to incorporate JAMon into other open source projects. Often this can be done very quickly. I recommend hiding JAMon behind a class of the specific projects owning. This allows the program to work even if JAMon is not available at runtime. HSQLDB used this approach

JAMon has already been incorporated into HSQLDB, Fulcrum, and the Spring framework. There are many other projects that would benefit from JAMon stats such as JBoss, JDBC drivers, Struts, and TomCat.

I will gladly assist in incorporating JAMon into your project.



4.l JAMon challenge #2! Your Help is needed!

The most important aspect of JAMon is not the current Java implementation, but the concept of aggregating ANY number associated with ANY key using ANY language in ANY program on ANY platform.

The JAMon code is simple enough to implement in many languages. I would like to see JAMon coded in C++, C##, Perl, Ruby and more. I would even love to see JAMon-like aggregate stats at the operating system level!

There could be various levels of compliance. The simplest one could be done in a matter of hours in just about any language (I have sample sample code for this if you would like to see it). Here are the levels of compliance:

  • Basic JAMon - Tracks basic aggregate data such as hits, min, max, total, average and a few more.
  • Basic JAMon with synchronization - Same as above, but with thread safety for multithreaded environments. I will soon post a link here to this minimal implementation in Java, to get you started.
  • Advanced JAMon - Includes the concept of ranges. This implementation becomes substantially more difficult.

I will gladly assist anyone wishing to port JAMon to another language. This would be a great college project!



5. Advantages of JAMon

Using JAMon is easy. Place the jamon jar file (80K) in your classpath, and surround the code that you wish to monitor with the JAMon API's start() and stop() methods. See the code sample below:


import com.jamonapi.*;
...
Monitor mon=MonitorFactory.start("myFirstMonitor");
...Code Being Timed...
mon.stop();

For example, to collect statistics for all JSP pages in a Web Application, a developer would place a call to the start() method at the top of every JSP, and a call to the stop() method at the bottom. With a few minutes work the developer will have statistics for all application pages. The statistics will include number of page hits, average page execution times, and the number of users simultaneously accessing each page. The statistics can be displayed as an HTML table by calling the getReport() method, or using jamonadmin.jsp.

JAMon requires no special administration accounts. Many monitoring tools require administrative accounts to install monitoring software, and to view statistics. Due to the complexity of these tools, developers typically go through specially trained administrators to access monitoring statistics. Many of these monitoring tools also require holes punched in firewalls.

JAMon has none of these problems. Statistics gathered via JAMon are available to all developers via jamonadmin.jsp. No administrative accounts, rights, or set-up are required for gathering and viewing statistics. Because the JAMon API is part of the application code, developers can switch J2EE App Servers and monitoring will continue to work. jamonadmin.jsp is accessed via HTTP and so can be viewed without punching any additional holes in firewalls.

JAMon can be used to monitor production systems. Monitoring production systems is important because it is the only way we can see how our code is being executed by real users. Also, many times the configuration of the production environment is different from the test environment, so the performance numbers between the two environments may be considerably different. For both these reasons monitoring only the test environment is not enough.

Unlike many monitoring tools, JAMon, due to its minimal performance overhead and multi-threaded capabilities, can be used in production systems. JAMon can be disabled at runtime (using jamonadmin.jsp), further reducing performance overhead. On a 2 Ghz Pentium IV, JAMon's start(), and stop() methods execute 550,000 times per second. After disabling, the methods execute 15,000,000 times per second.

To test JAMon's performance download the jamon jar and execute the following command. The number represents how many times start(), and stop() should be called in the various tests (100,000 will be used as the default if a number is not provided).

java -cp jamon-2.1.jar com.jamonapi.TestClassPerformance 500000

The last label ("Full Factory TimingMonitor()") in the output indicates how long it takes to run when monitoring is enabled, and the label starting with "NullMonitor2()" indicates how long it takes to run when monitoring is disabled.

JAMon is flexible. Many monitoring tools specialize in monitoring specific parts of an application. This can force a developer to resort to using multiple monitoring tools to cover all of their monitoring needs. For example, a developer may have to use different tools to monitor each of the following: page hits, EJB invocations, method calls, memory consumption, locks used, database connections consumed,... .

JAMon's start(), and add() methods take a string/label as an argument. This string can be a JSP page name, an EJB name, a method name or any other code or concept that can be represented by a string.

Also, because strings are passed, a developer can decide what to monitor based on runtime data. Using this technique we can monitor trends as well as performance bottlenecks. Some examples of JAMon's flexibility:

  • Keep statistics for a JSP home page
    • Monitor mon=MonitorFactory.start("HomePage");
  • Keep statistics for a DataAccess class's openConnection() method (which opens a database connection)
    • Monitor mon=MonitorFactory.start("DataAccess.openConnection()");
  • Keep statistics for each user that logs into our system
    • Monitor mon=MonitorFactory.start(getUserName());
  • Keep statistics for all page accesses by date. For example, "pageAccesses.05/10/02"
    • Monitor mon=MonitorFactory.start("pageAccesses."+getTodaysDate());


6. The JAMon servlet filter - Begin monitoring in 5 minutes!

A Servlet filters is a simple piece of Java code that is executed whenever a Java Web Application resource is accessed. Resources include Servlets, JSP's, GIFs, JPeg's, and HTML documents. The simple filter example will monitor all of these. By using JAMon the filter becomes a powerful web site hit counter. Statistics for pages will be gathered such as hits, and execution time (avg, min, max). In addition you can see which pages are currently executing.

The JAMon servlet filter does not require ANY code changes. You simply put jamon's jar file in your servers classpath, put the following entry into your web.xml file, and install the JAMon war so you can view your statistics. Note you can choose what types of pages you would like to monitor by altering the 'url-pattern' element below. See Java documentation for more info on how servlet filters work.

Note: The order of entries of your web.xml files is important.

<web-app>
  <display-name>YourApp</display-name>
   <filter>
    <filter-name>JAMonFilter</filter-name>
    <filter-class>com.jamonapi.JAMonFilter</filter-class>
   </filter>

   <filter-mapping>
    <filter-name>JAMonFilter</filter-name>
    <url-pattern>/*</url-pattern>
  </filter-mapping>

<!--
Servlet filter elements come before servlet elements
   <servlet>
    <servlet-name>demo</servlet-name>
    <jsp-file>/demo.jsp</jsp-file>
   </servlet>
-->

</web-app>


7.a The JAMon Web Application (WAR) - Viewing JAMon stats via jsmonadmin.jsp

JAMon comes with the jamon.war web application (Web Application Archive). This application contains jamonadmin.jsp, which enables you to view/query the JAMon statistics via a sortable HTML report, XML or an Excel spreadsheet. To view JAMon statistics you must put the jamon jar file in your server's classpath (In Tomcat you can put the jamon jar file in common/lib), and install jamon.war. If your own application uses the JAMon servlet filter you will be able to view the statistics by running jamonadmin.jsp (http://host:port/jamon/jamonadmin.jsp).



8. JAMon Examples


8.a Simple Example

JAMon gathers statistics for any code that comes between start() and stop() methods. In our first somewhat artificial example we will be timing how long our program sleeps when calling the Java Thread.sleep() method ten times from within a "for" loop.

import com.jamonapi.*;

public class MonitorTest {
    public static void main(String[] args) throws Exception {
        Monitor mon=null;
        for (int i=1; i<=10; i++) {
            mon = MonitorFactory.start("myFirstMonitor");
            Thread.sleep(100+i);
            mon.stop();
        }
        System.out.println(mon);  // toString() method called
    }
}

MonitorFactory.start("myFirstMonitor") both creates a monitor and begins gathering monitoring statistics. The start() method takes a monitor name/label as an argument. Summary statistics are gathered for all monitors that are passed identical labels. In our example the start() method is called 10 times with the same label, so the summary statistics will include data from all 10 calls.

The next line in the example is the sleep() method. The sleep() method takes one argument, which is the number of milliseconds the active thread should sleep. The loop has the effect of sleeping the thread for 101,102,�,and 110 ms. Finally comes the stop() method which will stop gathering statistics.

After the loop we print the statistics to the console, taking advantage of the monitor's toString() method. Let's take a look at what these statistics mean:

110 ms. - The execution time (in milliseconds) of the last monitor that was stopped. (There are 1,000 milliseconds in one second.)

The rest of the line contains summary statistics for the ten monitor calls. Considering our logic timed the sleep method 10 times with values ranging from 101 to 110 ms. the following results seem reasonable.

Hits=10 - A hit occurs whenever the start() method is called with an identical label. In this case "myFirstMonitor" is the monitor label.

Avg =106 ms. - Average is the total execution time divided by hits (i.e., Total/Hits).

Total=1,062 ms. - Total represents the total accumulated time for all 10 monitors that were executed. If we add 101 through 110 we get 1,055 ms., which is close to the total time the monitor calculated. The sleep() method is approximate, so our total is not exactly 1,055 ms.

Min=100 ms. - Min is the minimum execution time for the 10 hits.

Max =111 ms. - Max is the maximum execution time for the 10 hits.

Active=0 - The "Active" statistics have meaning in a multi-threaded environment. It indicates the number of current simultaneously executing monitors with identical monitor labels. This information could be used in a JSP to tell how many users are currently executing a page. In our example all monitors have been stopped prior to calling toString(), so no monitors are active.

Avg Active=1 - Average active indicates the average number of simultaneously executing monitors with identical monitor labels. In our single threaded example we would expect this to be one.

Max Active=1 - Max active indicates the maximum number of simultaneously executing monitors with identical monitor labels.

First Access=5/2/02 10:11:48 AM - Indicates when the monitor with the given monitor label was first executed.

Last Access=5/2/02 10:11:49 AM - Indicates when the monitor with the given monitor label was last executed.


8.b Monitoring a Servlet - Gathering page statistics

In our second example we will use JAMon to gather page statistics for a servlet. To gather page statistics a developer places a start() at the top of the servlets doGet() or doPost() method, and a stop() in the finally clause. If a method that can throw an exception is being monitored, then the stop() method must be called from the method's finally clause. This ensures that the monitor is stopped when an exception is thrown.

The sample servlet doesn't do much. It simply returns a list of musicians from three legendary reggae groups (Sample Output). The servlet sleeps for three seconds which slows down processing enough to get simultaneous servlet instances running. The getMusicians() method sleeps an amount based on the number of servlet instances currently running. This will help in demonstrating how JAMon can be used to measure application scalability. The statistics generated by this example will be discussed in the next section.

The code for the example follows:

import javax.servlet.*;
import javax.servlet.http.*;
import java.io.*;
import java.util.*;
import java.text.*;
import com.jamonapi.*;

public class JAMonDemo extends javax.servlet.http.HttpServlet {
   private static int active;
   protected void doGet(HttpServletRequest request, HttpServletResponse response)
    throws ServletException, IOException     {
        Monitor primary=MonitorFactory.startPrimary("JAMonDemo.pageHits");
        Monitor byDate=MonitorFactory.start("JAMonDemo.pageHits."+getDate());
        Monitor byUsersIP=MonitorFactory.start("JAMonDemo.pageHits."+
        getFormattedIP(request));
        active++;

        response.setContentType( "text/html" );
        PrintWriter out = response.getWriter();

        try {
            Thread.sleep(3000);
            out.println(getMusicians("Culture"));
            out.println(getMusicians("Maytals"));
            out.println(getMusicians("Wailers"));
        }
        catch(Exception e) {
            throw new ServletException(e.getMessage());
        }
        finally {
            byUsersIP.stop();
            byDate.stop();
            primary.stop();
            active--;
        }
    }

    // Returns date as a string in 05/10/02 format.
    private String getDate()     {
        return DateFormat.getDateInstance(DateFormat.SHORT).format(new Date());
    }

    /*
      Replace the standard "." separated ip address with one separated by dashes.
      i.e., 111.222.333.444 becomes 111-222-333-444.  This is done because "." has
      special meaning to the monitor class and will consume more resources.
      However this is not required.
     */
    private String getFormattedIP(HttpServletRequest request)     {
        String ip=request.getRemoteAddr();
        if (ip==null || "".equals(ip))
            return "";
        else
            return ip.replace('.', '-');
    }

    private String getMusicians(String group) throws Exception {
        Monitor mon=MonitorFactory.start("JAMonDemo.getMusicians("+group+")");
        final String culture="Joseph Hill, Albert Walker, Kenneth Dayes";
        final String maytals="Toots Hibbert, Raleigh Gordon, Jerry Matthias,"+
        " Jackie Jackson, Winston Wright, Hux Brown, Rad Bryan, "+
        "Paul Douglas, Winston Grennan";
        final String wailers="Bob Marley, Peter Tosh, Bunny Wailer, "+
        "Junior Braithwaite, Beverly Kelso";

        String musicians=null;
        if ("Culture".equals(group))
            musicians="<b>"+group+"</b>"+" - "+culture;
        else if ("Maytals".equals(group))
            musicians="<b>"+group+"</b>"+" - "+maytals;
        else if ("Wailers".equals(group))
            musicians="<b>"+group+"</b>"+" - "+wailers;

        Thread.sleep(active*50);
        mon.stop();
        return musicians;
    }
}

This example demonstrates the flexibility of the JAMon API, by incorporating runtime data such as date and IP address into our monitor labels. We are monitoring the following items in our servlet:

JAMonDemo.pageHits - Every time the JAMonDemo servlet is executed another hit will be recorded for this monitor label. This monitor is recording page hits. The consequences of starting this monitor with the startPrimary() method will be discussed later.

JAMonDemo.pageHits.Month/Day/Year - This monitor label will monitor page hits by day. There will be one row in the monitor report for each day. Typical entries would be "JAMonDemo.pageHits.5/12/02" and "JAMonDemo.pageHits.5/13/02".

JAMonDemo.pageHits.UsersIPAddress - This monitor label will monitor page hits by users IP address. There will be one row in the monitor report for each IP address. Typical entries would be "JAMonDemo.pageHits.127-0-0-1" and "JAMonDemo.pageHits.205-130-238-40". If the application has accounts then user names could be used instead of IP addresses.

JAMonDemo.getMusicians(group) - This monitor label will monitor calls to the getMusicians() method. The argument that is passed to the method is also incorporated into the monitor label. This adds little value in this example, but in a more real world program this technique can be very powerful. Incorporating class and method names into the monitor label (i.e., MyClass.myMethod) makes it easier to match monitors to the code they are monitoring.



9. Managing JAMon - jamonadmin.jsp

JAMon comes with a JSP that allows developers to view the JAMon report and control JAMon behavior. jamonadmin.jsp can be found in the jamon.war download.


9.a The JAMon Report

The JAMon report has one row for each unique monitor label that was passed to the MonitorFactory.start("�") method. This row is called a "JAMon report entry". The JAMon report displays all monitor information in one HTML table. The following statistics were discussed in enough detail in the previous example and won't be covered here: monitor label, hits, average, total, minimum, maximum, first access and last access.

We will be referring to the JAMon report in the following text (note sorting will only work when viewing the JAMon report from jamonadmin.jsp).

Looking at "JAMonDemo.pageHits" JAMon report entry we can see that the servlet had 30 page hits taking on average 3,441 ms. Some other statistics that require further discussion follow:

Std Dev ms. - 208 ms. - This is the standard deviation for the execution time of the 30 hits. If the statistical distribution is a normal distribution then we would expect 68% of the our hits to be within one standard deviation of the average, and 95% of our hits to be within two standard deviations.

Active - 0 - Indicates the number of monitors with this label that are currently executing (i.e., "Active" indicates concurrency). In the example when the JAMon report was run, no users were waiting for the JAMonDemo servlet to return, so "Active" was 0. However, if five requests were currently being handled by the servlet then the "Active" value would be 5.

Avg Active - 2.8 - Indicates the average number of monitors with this label that were simultaneously executing. In this example, of the 30 page hits, there were on average 2.8 simultaneously executing pages.

Max Active - 5 - Indicates the maximum number of monitors with this label that were simultaneously executing. In this example, of the 30 page hits, the most that were executing simultaneously was 5.

Note the JAMon report in jamonadmin.jsp can be sorted by clicking column headers. Clicking will toggle between ascending and descending sorts, based on column values. An up arrow indicates ascending, and a down arrow indicates descending.

The JAMon report can be displayed in a number of different ways. Two of the ways follow:

MonitorFactory.getReport() - Returns the JAMon report as an HTML table in alphabetical order by monitor label.

MonitorFactory.getRootMonitor().getReport(int sortCol, String sortOrder) - Returns the report formatted as an HTML table. The method is passed the column number to sort by, and whether the sort should be in ascending ("asc") or descending ("desc") order.


9.b Measuring outliers (JAMon Time Ranges)

A problem with the statistics previously mentioned is that they hide extreme data points (i.e., outliers). When application performance degrades we become more interested in outliers. JAMon time ranges contain information about these outliers.

Ideally we would like the average time for a monitor to be close to its minimum. However many factors (such as number of simultaneous users) can cause performance degradation. For example, we would expect our servlet to respond faster with two simultaneous requests than with 5,000.

There are 13 JAMon time ranges. The ranges are measured on an exponential scale. The first range is from 0 to 10 ms., and subsequent ranges double, until the final range, which includes values greater than 20,480 ms. We will use the JAMon report entry "JAMonDemo.getMusicians(Wailers)" and its "81-160ms." time range to discuss the five time range variables. The time range values are "7/114 (1/2.3/7.9)".

Hits - The number of hits within the given time range. All time range hits sum to the total hits for the JAMon report entry. The value for hits in the example is 7. This means that of this report entries 30 hits, 7 of them fell in the "81-160ms." range and averaged 114 ms.

Average - The average time in ms. for all hits within the given range. (see Hits for further info).

(Avg Active/Primary Active/Global Active) - All three of the time range "Active" statistics provide information about application scalability, by correlating application performance to monitors that are currently running. When a monitor is started the various active values are increased by one (based on the rules specified below) and when a monitor is stopped the active values are decremented by one. In general we expect performance to degrade with increasing activity. The time range "Active" statistics are all averages.

Avg Active - Indicates how many monitors with this label were on average "Active" when a hit occurred within the given time range. This allows us to correlate performance of this monitor to itself. This is similar to the "Avg Active" entry that comes earlier in the report. The difference is that this average is only for the hits that occur within this time range. In our example for the 7 hits within this time range on average only one monitor with the label "JAMonDemo.getMusicians(Wailers)" was executing concurrently.

Primary Active - Indicates how many primary monitors were "Active" on average, when a hit occurred within the given time range. A primary monitor is simply a monitor started with the startPrimary() method instead of the start() method (See the second example). This feature allows a developer to correlate the performance of all of their different monitors to a primary monitor. Primary monitors should be chosen to correlate highly with performance.

Page hits is a good choice for a primary monitor. We would expect as the number of simultaneous page hits increases performance will decrease. In the example we used page hits as our primary monitor ( MonitorFactory.startPrimary("JAMonDemo.pageHits") ).

In our example there were an average of 2.3 pages executing when the monitor labeled "JAMonDemo.getMusicians(Wailers)" had an execution time in the "81-160ms." range. The time ranges indicate that this method may have problems scaling. Notice how performance is decreasing with an increased number of active pages.

41-80ms. - When hits were within this time range there was an average of 1 page active.

81-160ms. - When hits were within this time range there was an average of 2.3 pages active.

161-320ms. - When hits were within this time range there was an average of 4.1 pages active.

"Avg Active" and "Primary Active" will be the same for primary monitors. If a primary monitor is not chosen then "Primary Active" will be 0.

Global Active - Indicates how many total monitors were "Active" on average, when a hit occurred within the given time range. The global active counter is incremented whenever a start() or startPrimary() method is executed (regardless of the monitor label), and decremented whenever the stop() method is executed. This feature allows a developer to correlate performance to the number of things being monitored in the application.

Notice how performance is decreasing with an increased number of global active monitors:

41-80ms. - When hits were within this time range there was an average of 4 monitors active.

81-160ms. - When hits were within this time range there was an average of 7.9 monitors active.

161-320ms. - When hits were within this time range there was an average of 13.4 monitors active.

Measuring "Primary Active" has an advantage over "Global Active". If we start monitoring more things then "Global Active" will artificially increase. In the above example if we decided to monitor "JAMonDemo.pageHits.UsersIPAddress.Month/Day/Year" then all of the "Global Active" variables discussed above would increase even though there is really no more activity on the server. This is not true of "Primary Active".

To summarize the concept of "Active" monitors let's look at another example. If the following code were executed there would be six active global monitors, two active primary monitors, two active monitors for "mon1", and one active monitor for all of the other monitors.

MonitorFactory.start("mon1"); // 1 global

MonitorFactory.start("mon2"); // 2 global

MonitorFactory.start("mon3"); // 3 global

MonitorFactory.startPrimary("mon4"); // 4 global, 1 primary

MonitorFactory.start("mon1"); // 5 global, 1 primary, and 2 mon1's are active

MonitorFactory.startPrimary("mon5"); // 6 global, 2 primary


9.c Controlling JAMon

The jamonadmin.jsp page also allows a developer to control JAMon at runtime. A description of the function of each of the buttons follows:

Refresh - Redisplays jamonadmin.jsp with the JAMon report in the default sort order. Refreshing can also be accomplished by clicking a column header.

Reset - Clears all monitors and their statistics. The report will now be empty, and subsequent calls to the start() method will repopulate JAMon statistics.

Enable and Disable - JAMon can be enabled and disabled at runtime. The default is enabled. When the disabled button is pressed JAMon statistics will no longer be gathered, however existing statistics will not be reset. To begin gathering statistics again simply press the enabled button.



10. Ways to use JAMon - More than performance

This section discusses the various ways JAMon can be used. JAMon's most obvious use is in determining application performance bottlenecks by gathering timing statistics. However, it can also be used in other ways such as determining how users interact with an application, and determining application scalability. Some ways that JAMon can be used follow.

To tune performance - Web applications can be difficult to tune due to the fact that there are many possible bottlenecks including: network IO, database IO, application code, poorly tuned queries, or even an increased amount of site traffic. To help pinpoint problems, performance diagnostics must be built into the application.

Without performance measurements, developers will spend their time guessing where performance bottlenecks occur. Performance problems typically occur in a small percentage of overall code. Algorithm guru Donald Knuth estimated that 4% of application code accounts for 50% of the performance. With measurements, developers can quickly locate that 4% of the code and get the biggest bang for their tuning buck. Without measurements developers waste effort tuning code that may have no significant impact on performance.

To improve code design - "More computing sins are committed in the name of efficiency than for any other single reason-including blind stupidity" - W.A. Wulf

"...premature optimization is the root of all evil." - Donald Knuth

Often, when discussing code designs, developers say they don't want to implement certain elegant designs, because of perceived performance problems. JAMon supports the following argument: "Let's code the elegant solution and measure its performance. If it doesn't perform well then we can try the other design."

To determine how users interact with the application - JAMon can be used to answer questions of the following type:

  • What is the application's peak usage time?
  • What is the maximum number of simultaneous users for the application?
  • Who are the application's "power users"?
  • Are certain application pages never used?
  • When did a specific user last login to the application?
  • What are the most commonly searched product categories?

To set coding priorities - Coding priorities can be based on factors such as which features are accessed the most often or what code is the slowest.

To detect programming and user errors - JAMon can be used to track errors, such as when exceptions are thrown, when resources are not released or when a user error occurs such as forgetting a password.

Developers can monitor code that gets and releases a resource, and check to see that the number of hits for both are the same. For example, if we get 1,000 database connections, we should also release 1,000 database connections. MonitorFactory.start("getConnection") would be called when a developer gets a connection and MonitorFactory.start("releaseConnection") would be called when the connection is released. If the hits don't match then we have a resource leak.

Developers can also monitor when Exceptions are thrown or when assertions are triggered. One helpful way to track Exceptions would be by date by user (i.e., "Exception.IllegalArgumentException.05/15/02.jsmith"). This allows developers to determine when exceptions are thrown without depending on users for the information.

To test application scalability - JAMon time ranges correlate performance to application load. Using this information a developer can measure how well an application scales. See the description of JAMon time ranges for more information on how to use JAMon for scalability testing.

To determine what is happening in an application at a snapshot in time - Many monitoring tools monitor an application from the "outside looking in", but because JAMon looks at the application from the inside we can use statistics such as "Active", "First Access" and "Last Access" to determine what activities our application is performing at any given time.

To Assist Testsers in the Test/QA process - This link describes how JAMon can be used in the testing process.



11. Future Directions

Currently JAMon does not persist its statistics. When the JVM goes down the statisistics are lost. A future version will be able to persist monitoring statistics to a database. This will allow developers to track how application usage and performance evolves.

Persisting statistics will also allow developers to run monitoring reports that span all tiers of an n-tier application. For example, if we have a clustered application using three web servers, three application servers and one database, we could persist statistics from all these servers into one database. This would allow us to have one report that shows application monitoring statistics for all tiers.

Porting JAMon to other languages and putting JAMon into other open source programs would also be interesting future initiatives



12. Sample Code

Java code can be called from both Java and non-Java environments. JAMon can be used in any environment that can call JDK 1.2 code. JAMon has been successfully used in ColdFusion script, BroadVision JavaScript and PowerBuilder. JAMon should also work in Active Server Pages, Java in the database, and many other environments. Sample code follows:

Servlet Filters - A Servlet filters is a simple piece of Java code that is executed whenever a Java Web Application resource is accessed. Resources include Servlets, JSP's, GIFs, JPeg's, and HTML documents. The simple filter example will monitor all of these. By using JAMon the filter becomes a powerful web site hit counter. Note the jamon jar file now comes with its own servlet filter class (com.jamonapi.JAMonFilter)

HTTP Sessions - The servlet API provides an interface that allows developers to track when an object is bound to an HttpSession. If the object implements the HttpSessionBindingListener interface then the valueBound(...) method is called when the object is put into the session, and valueUnbound(...) is called when the object is removed from the session. This allows an application to use JAMon to monitor statistics related to HTTP Sessions such as how many sessions are active, and which users are currently using the application.

Monitoring PowerBuilder Code - The sample code contains a Monitor EJB that acts as a thin wrapper for JAMon. The PowerBuilder client takes a monitor label as input and has a start and stop button. The start button passes the monitor label to the EJB and the stop button stops the Monitor. The jamon jar file must be accessible to the EJB at runtime.

ColdFusion sample

BroadVision sample

I will continue to add sample code to this document.



13. Downloading JAMon

Click here to download JAMon. This download contains:

  • The jamon jar file - Contains the JAMon binary
  • The jamon war file
  • fdsapi.war - A sample Java Web Application that uses JAMon. The war uses a servlet filter and JAMon to monitor all accesses to file resources such as HTML, JSPs, GIFs, and JPGs. JAMon also monitors all JDBC accesses in the application. The application uses the an open source product called the FormattedDataSet to generate dynamic text including HTML and XML. Click here to learn more about the FormattedDataSet.
  • JAMon Java Docs
  • JAMon Source Code - The full source code used to build the jamon jar file. There are approximately 1000 lines of code in JAMon, and it has some nice examples of the Gang of 4 design patterns including the creational, iterator, command and decorator patterns.
  • A JAMon Presentation - A PowerPoint presentation saved as an HTML file - Does not included JAMon 2.0 information


14. Support

Please submit any questions about JAMon to the sourceforge support forum, so that all users may benefit from the answers. The following link will take you to the JAMon support forum JAMon Support Forum. If JAMon has been helpful please make a donation to help offset the costs of running the project: Donate to JAMon



15. JAMon/FormattedDataSet T-shirts

Note: I have been happy with the JAMon shirts I own, but buy at your own risk. Should you have any problems with the product please contact Zazzle directly.

In case the flash application does not display properly, you may also purchase T-Shirts via this link: JAMon T-Shirts



16. JAMon License Agreement

JAMon has a very liberal license agreement. The spirit of the license is that there are no restrictions on its use. Should something not be clear in the license feel free to contact me, and I can clarify. In general JAMon binaries can be used free of charge in any software (commercial software too), and JAMon source code may be modified (any modifications must be noted per the license agreement). The JAMon License was adapted from the BSD license. It is requested that any modifications developers make to JAMon be sent to admin@jamonapi.com , so that all JAMon users may benefit.



17. Acknowledgements/Contributors

I am indebted to the following for their support in improving JAMon.



18. About the Author

Steve Souza has 20+ years of software and database development experience specializing in performance tuning.

Document last modified 07/2011.



SourceForge.net Logo