Exadata ovm creation: Second argument is not defined: bondeth_mode

Exadata ovm creation: Second argument is not defined: bondeth_mode

There are some days which start as exatastic days and end up at Oracle Support. This blogpost is (hopefully) one with temporal validity. I know that’s a 12c feature as well, but that’s another story. The error which I’ll discuss here, will tell you about the “Second argument is not defined: bondeth_mode” error in the domu maker logfile.

A while ago, one of my customers asked me to extend one of their exadata’s with 2 extra compute nodes, make sure they could do ovm on it. Oh by the way, make sure that they split the admin network from the existing one and while we’re busy, we’ll put the new vm’s in other public networks as well. Maybe I should put the “how did I manage to get this done” in another blog post, but it went actually surprisingly well. Bit creativity, some escalations at the network/firewall people (this is an installation in a secure environment) later, I found this problem which is quite interesting.

Step 2 of the OEDA is “creating virtual machines”, if you’re not using capacity on demand. So in my case … i had to create vm’s. I took the new configfile and he created one vm. Failed on the other one. This is strange because he managed to create the bridge, so he should be able to use it.
I tried it with the vm’s separately, same result. Basically everything I tried (even older OEDA, mind this sentence) turned out to provide me with this error.

First of all, I want to stress that the output is anonymised in order to keep the customers’ environment secure. So here we go.

First things first. This are the steps I need to execute for install.sh:

[root@demoexa01db01 linux-x64]# ./install.sh -cf example-demoexa01-demoexa01-clu1.xml -l
Initializing
|||||
1. Validate Configuration File
2. Create Virtual Machine
3. Create Users
4. Setup Cell Connectivity
5. Calibrate Cells
6. Create Cell Disks
7. Create Grid Disks
8. Install Cluster Software
9. Initialize Cluster Software
10. Install Database Software
11. Relink Database with RDS
12. Create ASM Diskgroups
13. Create Databases
14. Apply Security Fixes
15. Install Exachk
16. Create Installation Summary
17. Resecure Machine
[root@demoexa01db01 linux-x64]#

The first validation was already ok, so not going to show that one here, but the next output is interesting. So the actual vm creation failed like this. I used a separated xml in order not to disturb the other succeeded vm.

[root@demoexa01db01 linux-x64]# ./install.sh -cf example-demoexa01-demoexa01-clu1.xml -s 2
Initializing
Executing Create Virtual Machine
Creating guests for cluster demoexa01-clu1
Checking status of remote files...
Checking status of existing files on remote nodes...
Getting status of local files...
Creating guest using /EXAVMIMAGES/vanpupi_install/linux-x64/WorkDir/demoexa01adm01vm01.example.com-vm.xml on node demoexa01db01.example.com
Creating guest using /EXAVMIMAGES/vanpupi_install/linux-x64/WorkDir/demoexa01adm02vm01.example.com-vm.xml on node demoexa01db02.example.com
Creating guest using /EXAVMIMAGES/conf/final-demoexa01adm01vm01.example.com-vm.xml on node demoexa01db01.example.com
Creating guest using /EXAVMIMAGES/conf/final-demoexa01adm02vm01.example.com-vm.xml on node demoexa01db02.example.com
====== ERROR from node demoexa01db02.example.com ======
Command = demoexa01db02.example.com | root | /opt/exadata_ovm/exadata.img.domu_maker start-domain /EXAVMIMAGES/conf/final-demoexa01adm02vm01.example.com-vm.xml
Ret code = <1> from node demoexa01db02.example.com
[INFO ] Verify run time values are consistent with the configuration from file: /opt/oracle.cellos/cell.conf
[INFO ] Start network discovery in DOM0 to check and to create bridges for DOMU
## ERROR End from node demoexa01db02.example.com
====== End ERROR Output from node demoexa01db02.example.com ======
Error: Command [/opt/exadata_ovm/exadata.img.domu_maker start-domain /EXAVMIMAGES/conf/final-demoexa01adm02vm01.example.com-vm.xml] run on node demoexa01db02.example.com as user root did not execute successfully...
====== ERROR from node demoexa01db01.example.com ======
Command = demoexa01db01.example.com | root | /opt/exadata_ovm/exadata.img.domu_maker start-domain /EXAVMIMAGES/conf/final-demoexa01adm01vm01.example.com-vm.xml
Ret code = <1> from node demoexa01db01.example.com
[INFO ] Verify run time values are consistent with the configuration from file: /opt/oracle.cellos/cell.conf
[INFO ] Start network discovery in DOM0 to check and to create bridges for DOMU
## ERROR End from node demoexa01db01.example.com
====== End ERROR Output from node demoexa01db01.example.com ======
Error: Command [/opt/exadata_ovm/exadata.img.domu_maker start-domain /EXAVMIMAGES/conf/final-demoexa01adm01vm01.example.com-vm.xml] run on node demoexa01db01.example.com as user root did not execute successfully...
ERROR:
Errors occured.
ERROR:
Errors occured.
Error: Errors occured...
ERROR:
Error running oracle.onecommand.deploy.machines.VmUtils method createVms
Error: Errors occured...
Collecting diagnostics...
Errors occurred. Send /EXAVMIMAGES/vanpupi_install/linux-x64/WorkDir/Diag-170606_132924.zip to Oracle to receive assistance.
 
Errors occured, exiting...
[root@demoexa01db01 linux-x64]#

Pretty cryptic huh? The strangest thing is that by creating the second vm (long story about that one, but I had to create them separately and 2nd one must be done first. Just don’t ask 🙂 ) the bridge was created successfully:

[root@demoexa01db01 linux-x64]# brctl show
bridge name bridge id STP enabled interfaces
vmbondeth0 8000.0010e0c9a365 no bondeth0
 vif18.1
vmeth0 8000.0010e0c9a364 no eth0
 vif18.0
[root@demoexa01db01 linux-x64]#

So investigating the OEDA logfiles will tell you what’s wrong, right? So, no problem, we’ll check what he’s missing, fix it and try again. Sounds like a plan, doesn’t it?

2017-06-06 10:17:23,648 [FINE][ main][ CommonUtils:1842] First message : null
2017-06-06 10:17:23,648 [FINE][ main][ CommonUtils:1859] Exception: Errors occured...
2017-06-06 10:17:23,648 [FINE][ main][ CommonUtils:1860] Stack trace follows:
2017-06-06 10:17:23,649 [FINE][ main][ CommonUtils:1863] oracle.onecommand.commandexec.utils.CommonUtils:CommonUtils.java:checkKommandOutputs:1523
2017-06-06 10:17:23,649 [FINE][ main][ CommonUtils:1863] oracle.onecommand.deploy.machines.VmUtils:VmUtils.java:doDomUerCreateDomain:1376
2017-06-06 10:17:23,649 [FINE][ main][ CommonUtils:1863] oracle.onecommand.deploy.machines.LinuxOvmObject:LinuxOvmObject.java:createVirtualMachines:674
2017-06-06 10:17:23,649 [FINE][ main][ CommonUtils:1863] oracle.onecommand.deploy.machines.VmUtils:VmUtils.java:doCreateClusterVms:464
2017-06-06 10:17:23,649 [FINE][ main][ CommonUtils:1863] oracle.onecommand.deploy.machines.VmUtils:VmUtils.java:createVms:256
2017-06-06 10:17:23,649 [FINE][ main][ CommonUtils:1863] sun.reflect.NativeMethodAccessorImpl:NativeMethodAccessorImpl.java:invoke0:-2
2017-06-06 10:17:23,649 [FINE][ main][ CommonUtils:1863] sun.reflect.NativeMethodAccessorImpl:NativeMethodAccessorImpl.java:invoke:62
2017-06-06 10:17:23,650 [FINE][ main][ CommonUtils:1863] sun.reflect.DelegatingMethodAccessorImpl:DelegatingMethodAccessorImpl.java:invoke:43
2017-06-06 10:17:23,650 [FINE][ main][ CommonUtils:1863] java.lang.reflect.Method:Method.java:invoke:498
2017-06-06 10:17:23,650 [FINE][ main][ CommonUtils:1863] oracle.onecommand.deploy.cliXml.InstallSoftware:InstallSoftware.java:executeStep:660
2017-06-06 10:17:23,650 [FINE][ main][ CommonUtils:1863] oracle.onecommand.deploy.cliXml.InstallSoftware:InstallSoftware.java:executeForwardAction:593
2017-06-06 10:17:23,650 [FINE][ main][ CommonUtils:1863] oracle.onecommand.deploy.cliXml.InstallSoftware:InstallSoftware.java:parseCmdLine:450
2017-06-06 10:17:23,650 [FINE][ main][ CommonUtils:1863] oracle.onecommand.deploy.cliXml.InstallSoftware:InstallSoftware.java:processCli:255
2017-06-06 10:17:23,650 [FINE][ main][ CommonUtils:1863] oracle.onecommand.deploy.cliXml.InstallSoftware:InstallSoftware.java:main:316
2017-06-06 10:17:23,651 [FINE][ main][ InstallSoftware:701] Error running Create Virtual Machine, error message: Error running oracle.onecommand.deploy.machines.VmUtils method createVms
2017-06-06 10:17:23,651 [FINE][ main][ InstallSoftware:736] Error running oracle.onecommand.deploy.machines.VmUtils method createVms
2017-06-06 10:17:23,651 [FINE][ main][ KommandOutput:110] ====== Output from node ======
2017-06-06 10:17:23,651 [FINE][ main][ KommandOutput:112] Command =
2017-06-06 10:17:23,651 [FINE][ main][ KommandOutput:114] Ret code = <2> from node
2017-06-06 10:17:23,651 [FINE][ main][ KommandOutput:119] ## Output Start
2017-06-06 10:17:23,652 [FINE][ main][ EsCommonUtils:1138] Error running oracle.onecommand.deploy.machines.VmUtils method createVms
2017-06-06 10:17:23,652 [FINE][ main][ KommandOutput:122] ====== End Output from node Ret code2 ======
2017-06-06 10:17:23,652 [INFO][ main][ CommonUtils:1502] ERROR:
2017-06-06 10:17:23,652 [FINE][ main][ CommonUtils:1503] Errors printed on screen...
2017-06-06 10:17:23,652 [INFO][ main][ CommonUtils:1506] Error running oracle.onecommand.deploy.machines.VmUtils method createVms
2017-06-06 10:17:23,652 [INFO][ main][ OcmdException:62] Error: Errors occured...
2017-06-06 10:17:23,653 [FINE][ main][ OcmdException:63] Throwing OcmdException... message:Errors occured...
2017-06-06 10:17:23,653 [FINE][ main][ EsCommonUtils:1946] debugProp <null>
2017-06-06 10:17:23,653 [FINE][ main][ EsCommonUtils:1946] debugProp <null>
2017-06-06 10:17:23,653 [FINE][ main][ OcmdException:80] doZip is true and debug = false
2017-06-06 10:17:23,656 [INFO][ main][ ZipUtils:120] Collecting diagnostics...
2017-06-06 10:17:23,657 [FINE][ main][ ZipUtils:260] Getting files in dir log
2017-06-06 10:17:23,657 [FINE][ main][ ZipUtils:280] Added files from dir log
2017-06-06 10:17:23,657 [FINE][ main][ ZipUtils:260] Getting files in dir WorkDir
2017-06-06 10:17:23,697 [FINE][ main][ ZipUtils:280] Added files from dir cellosLogs-demoexa01db01.example.com
2017-06-06 10:17:23,697 [FINE][ main][ ZipUtils:260] Getting files in dir config
2017-06-06 10:17:23,711 [FINE][ main][ ZipUtils:280] Added files from dir config
2017-06-06 10:17:23,711 [FINE][ main][ ZipUtils:143] Creating zip file /EXAVMIMAGES/vanpupi_install/linux-x64/WorkDir/Diag-170606_101723 with 316 files
2017-06-06 10:17:24,871 [INFO][ main][ ZipUtils:147] Errors occurred. Send /EXAVMIMAGES/vanpupi_install/linux-x64/WorkDir/Diag-170606_101723.zip to Oracle to receive assistance.
 
2017-06-06 10:17:24,871 [FINE][ main][ OcmdException:108] Stack trace...
2017-06-06 10:17:24,871 [FINE][ main][ OcmdException:145] OcmdException from node demoexa01db01.example.com return code = 2 output string: Errors occured... stack trace = java.lang.Throwable
 at oracle.onecommand.escommon.common.OcmdException.ocmdException(OcmdException.java:105)
 at oracle.onecommand.escommon.common.OcmdException.<init>(OcmdException.java:65)
 at oracle.onecommand.commandexec.utils.CommonUtils.checkKommandOutputs(CommonUtils.java:1523)
 at oracle.onecommand.deploy.cliXml.InstallSoftware.parseCmdLine(InstallSoftware.java:459)
 at oracle.onecommand.deploy.cliXml.InstallSoftware.processCli(InstallSoftware.java:255)
 at oracle.onecommand.deploy.cliXml.InstallSoftware.main(InstallSoftware.java:316)
 
2017-06-06 10:17:24,871 [INFO][ main][ InstallSoftware:470] Errors occured, exiting...

yeah right … helpful.

Ok, it must be the vm, right? Just create a vm. A plain, simple, stupid vm, without GI or DB. You can find how to do it here: http://docs.oracle.com/cd/E80920_01/DBMMN/maintaining-exadata-database-servers.htm#DBMMN22236

So I did that and that, as this first vm will only be used for monitoring, I actually don’t need the rest, just a plain linux, so this might be a good workaround/solution, right?

[root@demoexa01db01 linux-x64]# /opt/exadata_ovm/exadata.img.domu_maker start-domain /EXAVMIMAGES/conf/final-demoexa01adm01vm01.example.com-vm.xml
[INFO ] Verify run time values are consistent with the configuration from file: /opt/oracle.cellos/cell.conf
[INFO ] Start network discovery in DOM0 to check and to create bridges for DOMU
[root@demoexa01db01 linux-x64]#

Exadata says no … all logfiles show the same. Almost desperate! BUT! There is another place to look. Take a look at the /var/log/cellos/exadata.img.domu_maker.log that is the logfile domu maker uses to store what he is doing. There is a log and a tracefile and that is exactly what we are looking for now. Let’s investigate.

[1496740473][2017-06-06 09:16:30 +0000][INFO][0-0][/opt/exadata_ovm/exadata.img.domu_maker][exadata_img_domu_maker_discover_interfaces_and_bridges][1787] Interface has no IP address: eth0
[1496740473][2017-06-06 09:16:30 +0000][INFO][0-0][/opt/exadata_ovm/exadata.img.domu_maker][exadata_img_domu_maker_discover_interfaces_and_bridges][1788] Obtain any existing bridge for the interface: eth0
[1496740473][2017-06-06 09:16:30 +0000][INFO][0-0][/opt/exadata_ovm/exadata.img.domu_maker][exadata_img_domu_maker_get_bridge_for_int_discovery][1294] Bridge is configured for interface and the bridge is in state UP. Interface; Bridge: eth0; vmeth0
[1496740473][2017-06-06 09:16:30 +0000][INFO][0-0][/opt/exadata_ovm/exadata.img.domu_maker][exadata_img_domu_maker_discover_interfaces_and_bridges][1798] Bridge found for the interface. Bridge; Interface: vmeth0; eth0
[1496740473][2017-06-06 09:16:30 +0000][INFO][0-0][/opt/exadata_ovm/exadata.img.domu_maker][exadata_img_domu_maker_discover_interfaces_and_bridges][1799] Try to obtain IP identity for the bridge: vmeth0
[1496740473][2017-06-06 09:16:31 +0000][INFO][0-0][/opt/exadata_ovm/exadata.img.domu_maker][exadata_img_domu_maker_discover_interfaces_and_bridges][1857] Bridge; IP address; Netmask: vmeth0; 123.123.123.123; 255.255.255.0
[1496740473][2017-06-06 09:16:32 +0000][INFO][0-0][/opt/exadata_ovm/exadata.img.domu_maker][exadata_img_domu_maker_discover_interfaces_and_bridges][2016] Network to be assigned to an interface by discovery has no match to the existing network for the bridge. Interface; Discovery network; Bridge network: eth0; 234.234.234.0/255.255.255.0; 123.123.123.0/255.255.255.0
[1496740473][2017-06-06 09:16:32 +0000][INFO][0-0][/opt/exadata_ovm/exadata.img.domu_maker][exadata_img_domu_maker_discover_interfaces_and_bridges][2017] Stop evaluation for configuration by discovery based on network match for interface: eth0
[1496740473][2017-06-06 09:16:32 +0000][INFO][0-0][/opt/exadata_ovm/exadata.img.domu_maker][exadata_img_domu_maker_discover_interfaces_and_bridges][1719] Evaluate interface for configuration by discovery: eth1
[1496740473][2017-06-06 09:16:32 +0000][INFO][0-0][/opt/exadata_ovm/exadata.img.domu_maker][exadata_img_domu_maker_discover_interfaces_and_bridges][1725] Interface is a slave interface. Discovery expects to find a valid bond master and partner slave for this interface: eth1
[1496740473][2017-06-06 09:16:32 +0000][INFO][0-0][/opt/exadata_ovm/exadata.img.domu_maker][exadata_img_domu_maker_discover_interfaces_and_bridges][1749] Interface is one of the slaves of bond master. Interface; Master; Slave list for the master: eth1; bondeth0; eth1 eth2
[1496740473][2017-06-06 09:16:33 +0000][INFO][0-0][/opt/exadata_ovm/exadata.img.domu_maker][exadata_img_domu_maker_discover_interfaces_and_bridges][1757] Discovered valid bond master and partner slave for the interface. Interface; Master: eth1; bondeth0
[1496740473][2017-06-06 09:16:33 +0000][INFO][0-0][/opt/exadata_ovm/exadata.img.domu_maker][exadata_img_domu_maker_discover_interfaces_and_bridges][1767] If interface is a valid slave, then use its bond master for further discovery. Else use itself. Interface; Effective interface: eth1; bondeth0
[1496740473][2017-06-06 09:16:33 +0000][INFO][0-0][/opt/exadata_ovm/exadata.img.domu_maker][exadata_img_domu_maker_discover_interfaces_and_bridges][1769] Try to obtain IP identification information for the interface: bondeth0
[1496740473][2017-06-06 09:16:33 +0000][INFO][0-0][/opt/exadata_ovm/exadata.img.domu_maker][exadata_img_domu_maker_discover_interfaces_and_bridges][1787] Interface has no IP address: bondeth0
[1496740473][2017-06-06 09:16:33 +0000][INFO][0-0][/opt/exadata_ovm/exadata.img.domu_maker][exadata_img_domu_maker_discover_interfaces_and_bridges][1788] Obtain any existing bridge for the interface: bondeth0
[1496740473][2017-06-06 09:16:33 +0000][INFO][0-0][/opt/exadata_ovm/exadata.img.domu_maker][exadata_img_domu_maker_get_bridge_for_int_discovery][1294] Bridge is configured for interface and the bridge is in state UP. Interface; Bridge: bondeth0; vmbondeth0
[1496740473][2017-06-06 09:16:33 +0000][INFO][0-0][/opt/exadata_ovm/exadata.img.domu_maker][exadata_img_domu_maker_discover_interfaces_and_bridges][1798] Bridge found for the interface. Bridge; Interface: vmbondeth0; bondeth0
[1496740473][2017-06-06 09:16:33 +0000][INFO][0-0][/opt/exadata_ovm/exadata.img.domu_maker][exadata_img_domu_maker_discover_interfaces_and_bridges][1799] Try to obtain IP identity for the bridge: vmbondeth0
[1496740473][2017-06-06 09:16:35 +0000][INFO][0-0][/opt/exadata_ovm/exadata.img.domu_maker][exadata_img_domu_maker_discover_interfaces_and_bridges][1914] Set IP address and Netmask for interface. Interface; IP address; Netmask: vmbondeth0; 234.234.234.234; 255.255.255.0
[1496740473][2017-06-06 09:16:36 +0000][INFO][0-0][/opt/exadata_ovm/exadata.img.domu_maker][exadata_img_domu_maker_discover_interfaces_and_bridges][1965] Gateway is pingable from the interface. Interface; Gateway: vmbondeth0; 234.234.234.250
[1496740473][2017-06-06 09:16:36 +0000][INFO][0-0][/opt/exadata_ovm/exadata.img.domu_maker][exadata_img_domu_maker_discover_interfaces_and_bridges][1971] Bridge found matching the discovery critera for IP address assignment for the interface. Interface; Bridge: vmbondeth0; vmbondeth0
[1496740473][2017-06-06 09:16:36 +0000][EXCEPTION][0-0][/opt/exadata_ovm/exadata.img.domu_maker][exadata_img_domu_maker_check_bondeth_mode][1527] Second argument is not defined: bondeth_mode
[1496740473][2017-06-06 09:16:36 +0000][INFO][0-0][/opt/exadata_ovm/exadata.img.domu_maker][exadata_img_domu_maker_flush_ip_address][1566] Flush IP address from interface: vmbondeth0
[1496740473][2017-06-06 09:16:37 +0000][INFO][0-0][/opt/exadata_ovm/exadata.img.domu_maker][exadata_img_domu_maker_discover_interfaces_and_bridges][2060] Shutdown interface: eth3
[1496740473][2017-06-06 09:16:37 +0000][INFO][0-0][/opt/exadata_ovm/exadata.img.domu_maker][exadata_img_domu_maker_discover_interfaces_and_bridges][2060] Shutdown interface: eth4
[1496740473][2017-06-06 09:16:38 +0000][INFO][0-0][/opt/exadata_ovm/exadata.img.domu_maker][exadata_img_domu_maker_discover_interfaces_and_bridges][2060] Shutdown interface: eth5
[1496740473][2017-06-06 09:16:40 +0000][INFO][/opt/exadata_ovm/exadata.img.domu_makers/imageLogger][imageLogger_destroy][] Closing log and trace files

Spot the problem? One tiny little line says:

[1496740473][2017-06-06 09:16:36 +0000][EXCEPTION][0-0][/opt/exadata_ovm/exadata.img.domu_maker][exadata_img_domu_maker_check_bondeth_mode][1527] Second argument is not defined: bondeth_mode

I checked whatever I could check and eventually ended up opening a SR with oracle. They did a very good job and they confirmed my suspicions. This behaviour matches the description and behaviour in bug# 26184688. Unfortunately it isn’t a public one.

The solution is very simple. We’re at the beginning of june now and this issue should be covered in the OEDA of june 2017. So they provided me a password protected download link for the june2017 OEDA and that should work. So I downloaded the patched version (basically just a release) and tried again. So you’d think all good. Close but no cigar. So I went back to development and 2 days later I got a new Ocmd version via the SR. This one did the job wonderfully.

I’m still puzzled why I received the bondeth message on invoking the domumaker manually. This question is still on the table at Oracle Support, so if I have an answer, I’ll update this blogpost.

Oh one more thing I noticed. It has nothing to do with this problem, but you might want to check Frits Hoogland’s blog about linux auditing. You can find it here: https://fritshoogland.wordpress.com/2017/01/21/auditing-oracle-database-stopping-and-starting-using-the-elk-stack . The reason I mention this is that I noticed that on the ovm based exadata linux auditing isn’t configured with a default audit set by default while it is configured on a bare metal one. So you might want to do this yourself.

Conclusion is: if you are adding vm’s to the ovm on exadata and if you are using the latest oeda. The only way forward is Oracle support at this moment.

As always, questions, remarks? find me on twitter @vanpupi

Comments are closed.
%d bloggers like this: