Beware of OEM 11g

Hi Fellow DBA's,

I just wanted to take a moment to let you know that now might not be a good time to upgrade or install the new EM Grid Control 11g.

I've been fighting with it for about a month now and have nothing but bad things to say about it (and i'm the single biggest OEM proponent in my department!).

Here are the main issues I'm seeing with a fresh 11.2 database and a fresh OEM install:
  • It's a major pain to install - go look at MetaLink, er, I mean, Oracle Support note [ID 1067438.1] "Master Note for Grid Control Installation and Upgrade" for an idea of what you're in for.
  • The JDK is hosed - Agent deploys of any kind fail with the dreaded "java.lang.NoClassDefFoundError: Could not initialize class oracle.sysman.oii.oiip.osd.unix.OiipuUnixOps" error (and just for the record: no, upgrading the OMS_HOME jdk to even the newest 1.6_21 does not fix the issue).
  • There are almost no support notes for OEM 11g - you'll never solve any issues without opening an SR.
  • It's unpatchable - there is a bug that prevents you from adding the Software Library path, which of course OPatch patches need once you get to the point of deploying PAR files.
  • Job reporting is broken - the "Jobs" tab always shows no history no matter how you search.
  • Policy Violations are broken - you know those great "All Targets Policy Violations" icons on the home page that you can click on? Yeah, try clicking on one and you'll get a nice "No Search Results found matching the criteria specified." message.
And this is just what I found - what about you guys, what issues have you seen?

PS: I hope to update this list with fixes as they come back from Oracle.

UPDATE: It turns out that some of these issues are due to a blunder on my part! I installed the 32-bit version of WLS 10.3.2, instead of the 64-bit one that you need your own Java install for.

This eliminated the horrible issue with the "OiipuUnixOps" error on Agent deploys and when trying to create a Software Library path.

If you did not install WLS using this syntax on a 64-bit system (eg: running the .bin file), then you did it wrong:
/usr/lib/jdk16/jdk1.6.0_18/bin/java -d64 -jar wls1032_generic.jar

Unfortunately, all the other gripes I have with OEM persist... :)


PhilD said...

Any comeback from oracle - I am having the exact same problems with fresh OEM11g grid install and agent deployment. I assume you have to install the OM Agent locally on all required servers as deploying from the OEM server wont work...

burghman said...

I didn't have any issues installing on 11g Grid Control on RHEL5 64-bit or the 11g Solaris agent onto a 3-node RAC.

I am however having one hell of a time troubleshooting the "Cluster Managed Database Services" page, which fails to log in to either the database or the Solaris server. I'm assuming it is the OS login that is failing, since I am able to log into the target server on the "Performance" tab.

As you suggest, I may have to submit a SR to get past this problem.

Douglas said...

We didn't license the Configuration Management pack, so didn't even try to push the agent from GC to the target.

To do a local agent installation, from the target server, wget the agentDownload. script that resides on your GC server.

wget http://myhost.domain.com:4889/agent_download/

Make it executable and then run it to install the agent.

For a RAC deployment:

./agentDownload.solaris -m myhost.domain.com -r 4889 -b /app/oracle/product -n -c "mynode01,mynode02,mynode03"

burghman said...

Got the Host Preferred Credentials problem resolved. Was a stupid mistake of not remembering to run root.sh after deploying the agents.

Richard Evans said...


UPDATE: It turns out that some of these issues are due to a blunder on my part! I installed the 32-bit version of WLS 10.3.2, instead of the 64-bit one that you need your own Java install for.

-- I did the same thing because that is what they listed under the SPARC download on the software page. Instead, I was supposed to download the GENERIC.... makes no sense at all.

Richard Evans said...

Logfiles are all over the place too! Be ware of gc_inst/WebTierIH1/diagnostics/logs/OHS/ohs1/mod_wl_ohs.log

This log grew to 4.5 GB within a month. Something writes to this file every 1-3 seconds. It says useless things like:

Fri Aug 13 10:24:34 2010 <153191281713074167772> parse_header is done
Fri Aug 13 10:24:34 2010 <153191281713074167772> Method is GET
Fri Aug 13 10:24:34 2010 <153191281713074167772> About to call parseHeaders
Fri Aug 13 10:24:34 2010 <150401281713074167988> parse_header is done
Fri Aug 13 10:24:34 2010 <150401281713074167988> Method is GET
Fri Aug 13 10:24:34 2010 <150401281713074167988> About to call parseHeaders
Fri Aug 13 10:24:34 2010 <150401281713074167988> done with sendRequest
Fri Aug 13 10:24:34 2010 <153191281713074167772> done with sendRequest

It seems like all of the trimming of logging I did on the EMGC 10g was thrown out the window and 11g is back to overly logging EVERYTHING unimportant.

LindaW said...

I'm wondering why you did a fresh install of OEM 11g grid rather than doing an upgrade from the 10g grid to 11g. We are in the beginning of going with a fresh install because of the limitations 10g had with physical host names that are no longer present in 11g (supposedly).

Raymond said...

The correct command to install the Weblogic server in a 64 environment AND using the Sun/Oracle JDK is
java -Xmx1024m -jar wls1032_generic.jar

if you use a mixed 32/64 environment you need the command java -D64 -jar wls1032_generic.jar

It is documented but very obscure.

Bradley S. said...

I'm pretty much at the point of giving up on all Oracle products except the database and RAC. IMHO Gridcontrol and Metalink (and others) have both become bloated and over complicated to astronomical proportions and it causes them to be very unreliable. Oracle Portal was a complete failure for us, Weblogic doesn't look to be much better, and I'm definitely not very impressed with their VM offering.

I wish someone at Oracle with some authority would call a few departments on the carpet and say "wait a minute". Who in there right mind would allow a install procedure like the one required for GridControl 11g. For now, I'm about to reinstall 10g and call it good.

Steve said...

Grid Control 11g is a total clown show, IMHO. The installation process is so bad that it makes me miss the days of Application Server with an infrastructure. An obscure bug seems to hose a Linux 32-bit installation (OMS install/config bombs out when during the install of GC). A 64-bit install actually worked, but then the problem with deploying an agent to another server appears. So for everyone else in the world except for Oracle, you really cannot use the 32-bit bin version of WLS (another disaster acquisition by Oracle) on x86_64. But amazingly, the agent works against the repository database used for GC. Oracle has some of the smartest idiots working for them when it comes to some of their products.

mlboley said...

Have you discovered yet that you have to download and install Cygwin to deploy the OEM agent on a Windows server? Another great Oracle "feature"!!!

Brent, glad to hear that you guys are finally getting to use Grid Control. How did you ever get a server inside the customer's domain so that you could use Grid Control to monitor the databases?