Browsed by
Tag: Exadata

Memo to Self: Recap cellsrvstat

Memo to Self: Recap cellsrvstat

Sometimes I ask myself “how did that work again”, so I decided to document this every time I have this feeling. With some links to the documentation, easy commands,… you got the picture.

First one today, new customer, new environment, to get some feeling with the cells, I used cellsrvstat.

Documentation reference (here ). Cellsrvstat is also part of the exawatcher on the cells.

A basic overview of the command. If you log on to the cells as root, it is in your $PATH. But in case you’re looking for it, it’s stored in /opt/oracle/cell<version>/cellsrv/bin/

So basics first, what can it do:

# cellsrvstat -h
LRM-00101: Message 101 not found; No message file for product=ORACORE, facility=LRM
Usage:
cellsrvstat [-stat_group=<group name>,<group name>,]
[-offload_group_name=<offload_group_name>,]
[-database_name=<database_name>,]
[-stat=<stat name>,<stat name>,] [-interval=<interval>]
[-count=<count>] [-table] [-short] [-list]

stat A comma separated list of short strings representing
the stats. Default is all. (unless -stat is specified).
The -list option displays all stats.
Example: -stat=io_nbiorr_hdd,io_nbiowr_hdd
stat_group A comma separated list of short strings representing
stat groups. Default: all except database
(unless -stat_group is specified).
The -list option displays all stat groups.
The valid groups are: io, mem, exec, net,
smartio, flashcache, offload, database.
Example: -stat_group=io,mem
offload_group_name
A comma separated list of short strings representing
offload group names.
Default: cellsrvstat -stat_group=offload
(all offload groups unless -offload_group_name is specified).
Example: -offload_group_name=SYS_121111_130502
database_name A comma separated list of short strings representing
database group names.
Default: cellsrvstat -stat_group=database
(all databases unless -database_name is specified).
Example: -database_name=testdb,proddb
interval At what interval the stats should be obtained and
printed (in seconds). Default is 1 second.
count How many times the stats should be printed.
Default is once.
list List all metric abbreviations and their descriptions.
All other options are ignored.
table Use a tabular format for output. This option will be
ignored if all metrics specified are not integer
based metrics.
short Use abbreviated metric name instead of
descriptive ones.
error_out An output file to print error messages to, mostly for
debugging.

In non-tabular mode, The output has three columns. The first column
is the name of the metric, the second one is the difference between the
last and the current value(delta), and the third column is the absolute value.
In Tabular mode absolute values are printed as is without delta.
cellsrvstat -list command points out the statistics that are absolute values


[root@dm06celadm01 ~]#

So it can display all kind of information about your cell status, which can be helpful to see what’s going on. So let’s do the list: (warning: awful lot of info! But i’ll cut out some of the rows, but if you execute it, be prepared for a long list)

[root@dm06celadm01 ~]# cellsrvstat -list
Statistic Groups:
io Input/Output related stats
mem Memory related stats
exec Execution related stats
net Network related stats
smartio SmartIO related stats
flashcache FlashCache related stats
health Cellsrv health/events related stats
offload Offload server related stats
database Database related stats
ffi FFI related stats
lio LinuxBlockIO related stats
mpp Reverse Offload related stats
Sparse Sparse stats

Statistics:
[ * - Absolute values. Indicates no delta computation in tabular format]

io_nbiorr_hdd Number of hard disk block IO read requests
io_nbiowr_hdd Number of hard disk block IO write requests
io_nbiorb_hdd Hard disk block IO reads (KB)
io_nbiowb_hdd Hard disk block IO writes (KB)
io_nbiorr_flash Number of flash disk block IO read requests
io_nbiowr_flash Number of flash disk block IO write requests
io_nbiorb_flash Flash disk block IO reads (KB)
io_nbiowb_flash Flash disk block IO writes (KB)
io_ndioerr Number of disk IO errors
io_ltow Number of latency threshold warnings during job
io_ltcw Number of latency threshold warnings by checker
io_ltsiow Number of latency threshold warnings for smart IO
io_ltrlw Number of latency threshold warnings for redolog writes
...
mpp_nr_blcc Num of reqs not pushed due to low cell cpu (C)
mpp_nr_bhcon Num of reqs not pushed due to high cell outnet (C)
mpp_nr_bhrnin Num of reqs not pushed due to high db node innet (C)
mpp_nincr_mb Num rate increase by reverse offload info from db (C)
mpp_ndecr_mb Num rate decrease by reverse offload info from db (C)
mpp_nincr_rn Num rate increases from db node cpu information (C)
mpp_ndecr_rn Num rate decreases from db node cpu information (C)
mpp_ndecr_ccpu Num rate decreases from low cell cpu utilization (C)
mpp_ndecr_con Num rate decreases from high cell outnet util (C)
mpp_ndecr_rn_in Num rate decreases from high db node innet util (C)
sparse_ncb num buckets compacted by sparse HT background scan
sparse_ios num IOs with sparse regions
sparse_ios_kb Total sparse IOs (KB)
sparse_smartio Total redirected smart ios (KB)
[root@dm06celadm01 ~]#

Let’s say you’re only interested in the io related things you could use a stat_group:

[root@dm06celadm01 ~]# cellsrvstat -stat_group io
===Current Time=== Tue Feb 21 11:29:39 2017

== Input/Output related stats ==
Number of hard disk block IO read requests 0 2226820445
Number of hard disk block IO write requests 0 1033312850
Hard disk block IO reads (KB) 0 1909110664882
Hard disk block IO writes (KB) 0 199121447989
Number of flash disk block IO read requests 0 14301322886
Number of flash disk block IO write requests 0 1008668696
Flash disk block IO reads (KB) 0 789129901568
Flash disk block IO writes (KB) 0 52097067586
Number of disk IO errors 0 0
Number of latency threshold warnings during job 0 1081
Number of latency threshold warnings by checker 0 0
Number of latency threshold warnings for smart IO 0 0
Number of latency threshold warnings for redolog writes 0 0
Current read block IO to be issued (KB) 0 0
Total read block IO to be issued (KB) 0 599867955384
Current write block IO to be issued (KB) 0 0
Total write block IO to be issued (KB) 0 197822797002
Current read blocks in IO (KB) 0 0
Total read block IO issued (KB) 0 599867955384
Current write blocks in IO (KB) 0 0
Total write block IO issued (KB) 0 197822797002
Current read block IO in network send (KB) 0 0
Total read block IO in network send (KB) 0 599867955384
Current write block IO in network send (KB) 0 0
Total write block IO in network send (KB) 0 197822797002
Current block IO being populated in flash (KB) 0 2765920
Total block IO KB populated in flash (KB) 0 32844047616
I/Os queued in IORM for hard disks 0 0
I/Os queued in IORM for flash disks 0 0

[root@dm06celadm01 ~]#

Last 2 lines are also very interesting, it tells you if IORM is kicking in or not. Might be usefull in some cases. Just saying.

The exec group is also nice. Once again I will cut out some rows, but the last lines are very interesting as well:

[root@dm06celadm01 ~]# cellsrvstat -stat_group exec
===Current Time=== Tue Feb 21 11:30:17 2017

== Execution related stats ==
Incarnation number 0 3
Number of module version failures 0 0
Number of threads working 0 2
Number of threads waiting for network 0 23
Number of threads waiting for resource 0 9
Number of threads waiting for a mutex 0 112
Number of Jobs executed for each job type
CacheGet 0 3123536972
CachePut 0 1031998876
CloseDisk 0 15376502
OpenDisk 0 20379160
ProcessIoctl 0 304858117
PredicateDiskRead 0 7462707
PredicateDiskWrite 0 36539
PredicateFilter 0 24054836
PredicateCacheGet 0 140219901
PredicateCachePut 0 16917010
FlashCacheMetadataWrite 0 0
RemoteListenerJob 0 0
CacheBackground 0 0
RemoteCellMgrService 0 0
CopyFromRemote 0 30925
...
sparse_bootstrap 0 0
sparse_free_region 0 0
DelegateIO 0 62678
NetworkPoll 0 0
CopySIFromRemote 0 550
SIGetJob 0 720
NetworkDirectoryGC 0 0

SQL ids consuming the most CPU
INT99 dxpwsgys5za27 3
END SQL ids consuming the most CPU

[root@dm06celadm01 ~]#

This tells me which database is asking the most cpu for which query. Might be usefull in some cases. Remember… in an idle environment and you do something, … then you’re automatically the “top”. But if suspecting things, it’s worth to have a look, it might help.

As always, questions, remarks? find me on twitter @vanpupi

A warm welcome to exadata SL6-2

A warm welcome to exadata SL6-2

Last years Oracle Openworld, uncle Larry announced the sparc based exadata SL6-2, so this means that we have to give the sparc chips a warm welcome to the Exadata family.
During the conference I wrote 2 blogposts. You can find them here and here.

To recap, a little picture of the new one in the family:

Exadata SL6-2

Nowadays, we’re used to the big X for the exadata’s. This is for the x86 infrastructure they are running on. So SL stands for “Sparc Linux”. You should follow the Oracle guys on twitter as well, then you see this product (Linux for sparc) is growing very rapidly. One of the questions which pop into the mind directly, which endianness is this using? Well, linux on sparc is using big endian as the sparc chip itself is big endian.

So in my blog posts I was eagerly looking forward to the spec-sheet and here it is! http://www.oracle.com/technetwork/database/exadata/exadata-sl6-ds-3547824.pdf

A shameless copy out of the datasheet:
“The Exadata SL6 Database Machine uses powerful database servers, each with two 32-core SPARC M7 processors and 256 GB of memory (expandable up to 1TB)”

According Gurmeet Goindi’s blog (@exadatapm) it comes at the same cost as the intel based variant. You can read his blog here: https://blogs.oracle.com/exadata/entry/exadata_sl6_a_new_era

Exadata SL6-2 hardware specifications

Look what’s there! In stead of 2 QDR ports, we now have 4. And also the elastic configs remain. Also remarkable is that the storage cell’s remain on Intel based architecture.
This looks interesting as well (same as the X6-2 trusted partitions):

Exadata SL6-2 mgmt features

 

On this moment (or I have read over it) I can’t see yet how virtualisation will be done, so if someone has info about this, I will be happy to hear this. I heard several rumours about this, but I am eager to find out what it’s going to be!

One question remains … when will I be able to find a customer who buys it and let me explore this to the bottom 🙂

 

As always, questions, remarks? find me on twitter @vanpupi

 

 

 

Documentation bug in ovm for Exadata

Documentation bug in ovm for Exadata

A while ago a customer gave me a heads up about the “bug” concerning the default passwords for root and celladmin. I was thinking a bit further and I wondered if the “documentation bug” I found while adding a new OVM in the virtualised exadata is solved. The official documentation can be found here .

documentation_screenshot
documentation_screenshot

Then “Managing Oracle VM Domains on Oracle Exadata Database Machine” and then “Creating Oracle RAC VM Clusters” brings you to the point I want to warn you for.

All steps are correct, but the last one “Run all steps except for the Configure Cell Alerting step using the XML file for the new cluster. For most installations, the Configure Cell Alerting step is step 7. For example, to execute step 1, run the following command” might be a bit tricky. Why? I will show you.

When deploying the exadata, if you list the steps you will get this output:

$ ./install.sh -cf anonymous_customer.xml -l
 Initializing

1. Validate Configuration File
2. Create Virtual Machine
3. Create Users
4. Setup Cell Connectivity
5. Calibrate Cells
6. Create Cell Disks
7. Create Grid Disks
8. Configure Alerting
9. Install Cluster Software
10. Initialize Cluster Software
11. Install Database Software
12. Relink Database with RDS
13. Create ASM Diskgroups
14. Create Databases
15. Apply Security Fixes
16. Install Exachk
17. Create Installation Summary
18. Resecure Machine
$

But if you take the newly created xml for the new cluster:

$ ./install.sh -cf anonymous_customer_new_clu.xml -l
 Initializing

1. Validate Configuration File
2. Create Virtual Machine
3. Create Users
4. Setup Cell Connectivity
5. Calibrate Cells
6. Create Cell Disks
7. Create Grid Disks
8. Configure Alerting
9. Install Cluster Software
10. Initialize Cluster Software
11. Install Database Software
12. Relink Database with RDS
13. Create ASM Diskgroups
14. Create Databases
15. Apply Security Fixes
16. Install Exachk
17. Create Installation Summary
18. Resecure Machine
$

Do you spot the difference? I don’t.
I just want to say … If you create a new cluster, be careful with “Create Cell Disks”. I should recheck the logfiles, but the time I checked it lately, it was performing a drop of the celldisk before recreating it. So you can imagine what will happen to your other virtual machines. If you have an exadata on which I can try it, please let me know. I’m happy to check it out further 🙂

My trip to Oracle Openworld 2016 #OOW16

My trip to Oracle Openworld 2016 #OOW16

It all started a couple of months ago. I submitted a presentation and my friend Philippe Fierens submitted a panel session together with Adam Bolinski (Ora600.pl) , but unfortunately, I’m joining #teamRejected Lots of cool and nice people had to join as well, so it’s not too bad.
Then great news came and my boss asked if I was interested of visiting Oracle Openworld. I didn’t have to think twice, so my answer was yes very quickly.

The trip starts early. Very early … on saturday at 04:30 in the morning we had the appointment at the company. There a little coach picked us up and after a 2hour drive, we arrived at schiphol.

img_0865

A nice group of friendly and interesting people traveling together. This will be fun!

After a short check-in, bagage drop-off and all other security checks, we had breakfast and after that, off we go!

 

img_0868 Immediately it became very clear what the main topic of this Oracle Openworld would be: CLOUD! 🙂

Personally, I’m very sceptic to this new “hype”, but the only way to know if I’m right or wrong is to go with an open mind and experience it.

 

 

10:30h of flying later, we safely landed in San Francisco. Then the necessary checks, and we took a cab to the hotel. Checking in and then off we went to the Moscone center for the registration.

img_0874

It IS impressive. People told me it’s big and large, but if you stand there for yourself, you know they’re right. This is going to be cool.

After registration we went for a walk. Doing something a little bit active in order not to suffer from the jetlag. Then I received a message from Philippe, he’s already at Jillians, so we went to have some drinks together to catch up. Always nice to see friends again! And friends do have some good ideas, this night’s dinner would be sushi. A very typical sushi bar but so good! If I could find it back, I would recommend it!

The next day, sunday, ok there are already some sessions, but for my first time in San Francisco I really wanted to see the bridge. So with the oracle belgium people we we went biking. After riding the Goldengate bridge and taking the Ferry back, it’s time to follow some sessions.

I tried to create a varying schedule filled with attending some sessions and going to the demo-grounds. My personal feeling was that you really had to search well for technical sessions, whilst when walking a moment in the demo grounds, you could talk to lots of technical people and some of my questions I had were answered very quickly. So it became clear very quickly that this was the place to be.

Sunday evening, the glorious moment of every OOW – virgin. Larry’s keynote. To be honest, he let me down. After hearing so much about the keynotes maybe my expectations were a bit high but, it is an impressive event. We closed the evening with some drinks and then … early to bed!

Monday a day full of sessions/demogrounds/… but most of all, trying to get to know as much as possible about the newly announced exadata, the Exadata SL6-2. A sparc based engineered system which I think Oracle is going to push pretty hard. Engineers I was talking to, said that this machine is to be used in the Oracle cloud. Still trying to figure out on how to check it easily to confirm this.

Tuesday it became very clear that Oracle is focusing on cloud … but … “there will be a coexistence of about 10 a 15 years”. I like this, but in some way it feels like they aren’t very confident in their own product, but time will tell. Larry held his second keynote and this was really what I had expected. If you’re curious, you can see it here. And that evening we had a blast during the benelux party. So we went early to bed, literally 🙂

Wednesday, the last full day. I spent most of the time on the demo grounds this day. Taking a picture of the famous America’s cup  and an interesting session about the exadata technical deep dive and internals. Congratulations also to Oren Nakdimon (@dboriented on twitter), he had a nice presentation about upgrading his pl/sql code without any downtime on standard edition.

Last full day, apparently means also the appreciation event. It was announced to be Billy Joel, but he cancelled and was replaced by Gwen Stefani and Sting. Personally, I’m not really a Gwen Stefani fan … but Sting was really fantastic and he got the atmosphere going!

Thursday was a day of buying some presents for home and rushing to the airport.

A common topic through many presentations and talks at the geek theatre at the demo grounds is encryption. It’s almost named in one sentence with cloud. Which they immediately turn directly to sparc. So my guess is sparc will be pushed more and more. Just because of the encryption on the chip and the software in silicon which can help you with the in-memory option of the oracle database. This screams for some tests.

In summary, my first Oracle Openworld experience was fantastic. I can advise everybody that you need to have experienced it at least once. Such nice friendly people, and lots of knowledge transfer. Thanks Exitas for letting me doing this!

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Welcome Exadata SL6-2!

Welcome Exadata SL6-2!

EDIT: a new entry about this Exadata SL6-2 launch can be found here!

During a session of Juan Loaiza with the lovely title “Oracle exadata: what’s new and what’s coming” suddenly following slide popped up.

img_2209

 

So it looks like this is really going to happen. In the session no exact timings were given, but I’m awaiting this system to play with.

Basically I think this will be useful to migrate current solaris / sparc users to exadata. I’m a bit sceptic about these extreme performance numbers, but hey … if it’s launched, i’ll be happy to test it.

To be continued!

Exadata on sparc: sl6-2

Exadata on sparc: sl6-2

EDIT: a new entry about this Exadata SL6-2 launch can be found here!

Yesterday while walking to Larry’s keynote  on Oracle OpenWorld 2016, all people had to pass the engineered systems.

image

Curious as I am, and fed by lots of rumors I wanted to have a look at them. And indeed, there was one “new” one standing.

the nice thing is that you just can talk to the guys and this extremely friendly guy told me “this one is being announced on Tuesday”. It’s all about the exadata sl6-2. You’d think, ok another exadata, but this one is a bit special. The compute nodes are sparc t7 based and they run Linux on sparc. The nice thing they have 2ib cards and thus potentially 160gb/s available. Sounds something nice to test 🙂

Storage cells are still Intel, but who knows what’s going to come. I had the chance to play with it a very short bit and it’s really running Linux on sparc, ofcourse it’s just exploring and it should be available around December and the virtualized version was to be expected around May 2017. Anyhow, I ‘ll be attending the keynote (so that’s why Larry will do a second one 🙂 ) and then we’ll hopefully know more.

image

What I know now is that it will come in same elastic configurations as the current Intel based exadata’s

Anyhow … To be continued!