Database+Disk+Performance™

New Workshops: SQL Server Performance - Free

2009-03-10T13:47:00.002-04:00

[Reposting as the formatting was very poor on the original post.]

Upcoming Events: Workshops

Workshop Dates

Detroit March 30 & 31

New York City April 2 & 3

Tokyo April 9 & 10

Houston April 23 & 24

Chicago May 12 & 13

Philadelphia May 14 & 15

Workshop Abstract

There are many classes that cover SQL Server. There are also just as many that cover Oracle Applications… but very few cover the intersection of the two.

This two day workshop covers the specifics of how the Oracle application interacts with SQL Server and how to architect a scalable and highly available solution. Its focus starts at the ODBC level and goes down to the disk.

Emphasis is on isolating issues, finding problems before they happen, and performance tuning. The goal of the workshop is to move beyond “my system’s slow” into WHY it’s slow and how to fix it. We’ll cover: what to look at, what it means, and what “good” & “bad” look like on a wide variety of platforms.

While the workshop is in lecture format, real world examples are used exclusively in the form of perfmon and profiler traces in addition to numerous trip reports and health checks that have been done with customers over the last 5 years. We’ll use out of the box tools and basic scripts to gather up information and apply common sense techniques to come up with answers.

Skills learned in this workshop can also be applied to other enterprise applications in your landscape.

Target Audience: DBAs, Siebel/PeopleSoft/JD Edwards Administrators, Consultants

Working knowledge of SQL Server is encouraged.

Registration

Please pull down the registration form here:

http://www.computationpress.com/microsoftoraclecoe/moceevents.html

And e-mail to frankmcb@microsoft.com

Note, NYC is almost completely full. Philly has space. Detroit has space. Houston is filling fast. Chicago has space.

SQL Server Performance Workshop

2008-12-30T17:14:00.001-05:00

SQL Server Performance Workshop: ERP & CRM

This two day Microsoft taught workshop covers the specifics of how the Oracle application (Siebel, PeopleSoft, JD Edwards) interacts with SQL Server and how to architect a scalable and highly available solution. Its focus starts at the ODBC level and goes down to the disk.

Cities Taught In

Zurich, Switzerland

Tallinn, Estonia

Stockholm, Sweden

New York, NY

Los Angeles, CA

San Francisco, CA

Please view the complete logistics at: www.computationpress.com

Abstract

There are many classes that cover SQL Server. There are also just as many that cover Oracle Applications… but very few cover the intersection of the two.

This two day workshop covers the specifics of how the Oracle application interacts with SQL Server and how to architect a scalable and highly available solution. Its focus starts at the ODBC level and goes down to the disk.

Emphasis is on isolating issues, finding problems before they happen, and performance tuning. The goal of the workshop is to move beyond "my system's slow" into WHY it's slow and how to fix it. We'll cover: what to look at, what it means, and what "good" & "bad" look like on a wide variety of platforms.

While the workshop is in lecture format, real world examples are used exclusively in the form of perfmon and profiler traces in addition to numerous trip reports and health checks that have been done with customers over the last 5 years. We'll use out of the box tools and basic scripts to gather up information and apply common sense techniques to come up with answers.

Skills learned in this workshop can also be applied to other enterprise applications in your landscape.

Target Audience: DBAs, Siebel/PeopleSoft/JD Edwards Administrators, Consultants

Working knowledge of SQL Server is encouraged.

Event Registration

Please e-mail Susan Meyer at v-sumeye@microsoft.com to register for the event.

Topics Covered

Top Reasons for SQL Server 2005/2008

Query Repro: Reproducing Problem Queries Outside of the Application and Tools to Fix Them

Performance Tuning Methodology: A Repeatable Process

Hands on Performance Tuning Lab

Databases & Disk Drives: Architecting a Scalable Solution

SQL Server Settings

Case Study Handouts

Architecting for RSCI

SQL Server Compression

Hyper-V and Virtualization

SQL Server Performance Methodology with Oracle Applications

2008-05-12T16:39:00.002-04:00

This deck presents a systematic way of defining and solving performance issues with Oracle Applications. It gives an overview of how disk architecture impacts performance, how to run a trace, and reproduce the queries properly outside of the application.

Please go to www.computationpress.com website to download this deck.

Performance Methodology Flow Chart

2008-05-12T16:32:00.002-04:00

I have uploaded my Performance Methodology Poster (PDF format) to the Computation Press website:

www.computationpress.com

The poster outlines steps and scripts needed to go in and resolve performance issues with enterprise applications (ERP & CRM).

Database & Disk Deck

2008-05-12T16:30:00.003-04:00

I have posted the deck I give at conferences on database and disk drives. It's a "shorter" deck (ie. One hour talk), instead of the larger deck I sometimes use that covers 4 hours.

A short abstract:

It focuses on how to create a scalable database solution on everything from direct attached SCSI to SANs. The deck covers perfmon counters, partition alignment, SQL IO performance testing, proper number of files for space management, etc...

www.computationpress.com

Finding Queries That Kill Your System

2008-02-08T17:05:00.001-05:00

The following is a short article on how to locate poor queries on a JD Edwards/PeopleSoft/Siebel system. I will be following it up with a more comprehensive version taking into account debugging batch jobs and more in depth techniques such as using Excel to drill into issues. But for the moment, I wanted to get this something short and simple posted ASAP.

Introduction

The most difficult issue in finding performance issues usually revolves around just defining what the problem is in the first place. This article will focus on how to use Profiler to capture the necessary data and comb through it to define where the issues are and then using the Database Tuning Advisor (DTA) to solve them.

There are two ways to find long running/resource intensive queries on a system. The first is to use dynamic management views (DMVs) provided in SQL Server 2005, the second is to use Profiler.

In SQL Server 2005, there are a variety of new DMVs that provide insight into how/why your system is running slow. The main drawback to them are the fact that they provide a "macro" view of the system and can lack the ability to pinpoint exactly where issues are at. For example, a DMV cannot be initialized at the beginning of a job to "reset" the counters for a baseline or that the data collected in the DMV may reflect changes since yesterday or a week ago depending on the available memory to store the information.

While DMVs are great tools, they are still not a substitute for SQL Server Profiler. Profiler allows you to capture data over a user defined period of time to nail down a specific job or user. This flexibility lets the DBA get a more granular picture of the situation or, if the trace is run for a day, a macro view of your entire system.

This paper offers a simple step by step method for prioritizing and attacking problems.

Please look for this paper to be posted by February 9, 2008 at www.computationpress.com

Calculating How Often An Index Is Used In SQL Server 2005

2008-01-14T20:21:00.001-05:00

Over the last week, I have been working with several customers trying to manage performance by cutting down the number of non-used indexes in the database. In many canned apps, it's not unheard of to see 50 to 80 indexes on a table where only 10 or 15 are actually populated and used. The problems that occur because of this are numerous: extra disk used, longer backups, and of course—INSERTS, UPDATES and DELETES take much longer.

One of the main pains in using the DMVs are not being able to initialize them and then sorting through the extra data generated in the process. I wrote a few scripts and decided to turn them into a small TechNote on this subject. The core of the paper provides a method for solving this common problem by using the DMV sys.dm_db_index_usage_stats.

Examples and ideas covered work for SAP, PeopleSoft, Siebel, JD Edwards, and many other canned applications that provide out-of-the-box schema.

The full paper is posted at: www.computationpress.com

Best Regards,

Frank

Call for Content

2008-01-02T21:08:00.001-05:00

Finished the last paper and thinking about a new topic. I have probably a dozen or so possibilities, but I'd like to open the floor to the readers and ask the question: What is important to you? What topic would you like written about next that centers around databases + disk + performance?

Please send me an e-mail @ frankmcb@gmail.com and let me know.

Look forward to hearing from you.

Best-

Frank

Paper Published & Posted: Poor Disk Performance, Time Outs, Database and the SQL Server Errorlog

2007-12-29T21:27:00.001-05:00

Sorry for the delay of a few days. Was caught up in an escalation and the paper went longer than I thought it would—21 pages with references and end notes. Please review this draft and feel free to comment on it. The graphics need to be cleaned up some, etc…

Please find the paper posted on: www.computationpress.com

Best Regards,

Frank

Poor Disk Performance, Time Outs, Database and the Errorlog: SQL Server has encountered % occurrence(s) of IO requests taking longer than 15 seconds to complete on file

2007-12-25T09:54:00.001-05:00

The SQL Server errorlog provides a wealth of information and is often overlooked, even though it should be the first place we all turn to. The problem though is that most people look at the errorlog only after a problem has occurred instead of proactively looking at it as part of the daily list of tasks.

I've written a TechNote on a common error that affects the disk subsystem and the database and shows up in the errorlog as:

SQL Server has encountered 4507 occurrence(s) of IO requests taking longer than 15 seconds to complete on file [D:\Program Files\Microsoft SQL Server\MSSQL\Data\BIGDB_Data.MDF] in database [BIGDB] (7)

While many people have blogged on WHAT this issue is and WHAT to look at. Microsoft has published KB Article 897284 on the topic—which gives a good DESCRIPTION. I want to explore more of the areas around WHY it happens and suggest some actual suggestions on HOW to fix it.

Please visit the Computation Press website on December 26, 2007 and I'll have the note posted.

www.computationpress.com

2007-12-19T11:13:00.000-05:00

After a long absence, I'm going to kick start this blog again. Over the last year, I've been refining more performance techniques and thoughts as I've been working. More scripts and tools. In addition to the blog, I'll also start creating and posting videos to accompany the postings.

Look for a new posting in the next week.

Best-
Frank

Index Covering: Database, Disk IO, and Enterprise Apps

2006-07-06T19:57:00.000-04:00

The Series

This is the first in a series of articles on Index Covering which is a performance technique to cut down disk IO and improve query performance. The first article here will define the basic elements in a query execution plan and how to find the bad parts. The next article will show how to fix them. Finally I'll conclude with some larger examples from PeopleSoft, JD Edwards, Siebel and SAP.

Introduction

Databases have two basic structures for storing information. Index pages and data pages. Both reside on the disk. To achieve optimal performance, DBA’s create indexes which are really just organized subsets of data.

Since an index is a subset of data, extra reads are often needed to find all the additional information. Typically, this requires a bookmark lookup.

These extra reads on a disk can add up quickly and cause poor performance. The extra IOs also cause the database to use cache less efficiently.

What is a bookmark lookup?

Databases have a concept called “book mark lookups”. This means that the database has to find what it needs in the index page, then find the corresponding data page for the rest of the information required.

Think of it like this: A person needs to find George Washington’s birthday. They look in the index of an encyclopedia to find “Washington, George” which tells them to turn to page 98 to find an entry on him. On page 98, they find the birthday. This is an example of a “book mark lookup”. The researcher looks in the index, keeps their finger there, turns to a page in the book, finds the birthday for the rest of the information.

A bookmark lookup occurs when the database cannot find all the needed information for a query in the index page and so it goes to the data page to complete the request.

For example:

create table A (col_one int, col_two int, col_three int)

create index A_P1 on a(col_one, col_two)

[Data added to table…]

Select col_three

From A

Where col_one = ‘22’

In this example, the query would used index A_P1 to find the row pointer to the data page in the table, but since A_P1 index does not contain column col_three, it would have to do a book mark lookup from the index to the data page.

Why are bookmark lookups bad for performance?

Not all bookmark lookups are horrible. It just means that the database had to do some extra IO to find the data it needed. If you are only hitting a few rows, then it’s not an issue. If you are having to sort through hundreds or thousands of rows to find the one row you needed, or are in a massive nested loop, then book mark lookups can be a problem.

There are several examples below which will illustrate the pain in enterprise apps that can be large and small depending on the query.

How do you know you have a bookmark lookup?

A bookmark lookup shows up in the execution plan (ex. set statistics profile on) looking like this:

|--Bookmark Lookup(BOOKMARK:([Bmk1001]),OBJECT:([JDE_CRP].[CRPDTA].[F4311] AS [T1]))

Ok, so I got bookmark lookups, so what?

First, figure out if it’s causing a drag on the system. Look at the number of rows and executions that the bookmark lookup is using. Is it 5 or 50,000? This can be seen on the left hand side of the execution plan using set statistics profile on command.

In this example, zero rows were brought back and the loop was executed only once. This bookmark lookup was not a problem.

Rows Executes StmtText

------ --------- ---------------------------------------------------

0 1 |--Nested Loops(Inner Join,

0 1 |--Sort(ORDER BY:([T0].[ILUKID] DESC))

0 1 | |--Bookmark Lookup(BOOKMARK:([Bmk1000]),

0 1 | |--Index Seek(OBJECT:([JDE_CRP].[CRPDTA].[F4111].[x] AS

0 0 |--Clustered Index Seek(OBJECT:([JDE_CRP].[CRPDTA].[F4311].

Here is an example where a bookmark lookup caused severe problems. It is bringing back 85 million rows and is being executed 4,017 times.

Rows Executes StmtText

----------- ----------- ------------------------------------------------------------

1 1 |--Nested Loops(Inner Join, WHERE:([T1].[PR_POSTN_ID]=

364442 1 | |--Index Scan(OBJECT:([Siebel_Prd].[dbo].

85205956 4017 | |--Bookmark Lookup(BOOKMARK:([Bmk1001])

85205956 4017 | |--Nested Loops(Inner Join, OUTER

4017 4017 | |--Nested Loops(Inner Join, OUTER

4017 4017 | | |--Bookmark Lookup(BOOKMARK:([Bmk1002

4017 4017 | | | |--Nested Loops(Inner Join,

4017 4017 | | | | |--Bookmark Lookup(BOOKMARK:([Bmk

4017 4017 | | | | | |--Index Seek(OBJECT:([Siebel_

4017 4017 | | | | |--Index Seek(OBJECT:(

4017 4017 | | |--Clustered Index Seek(OBJECT:([Siebel

85205956 4017 | |--Index Seek(OBJECT:([Siebel_Prd].[dbo].[S_

Disk Performance Problem Isolation Techniques: Simple Tools

2006-06-26T15:00:00.000-04:00

Introduction: “I told you SQL Server couldn’t scale…”

The usual conversation goes like this… “I told you SQL Server couldn’t scale…” and so the finger pointing begins. As a student of philosophy in college, I learned about rhetoric and baiting techniques. As a DBA, I had to learn to fight back and not be on the defensive from such attacks. From personal experience, I have managed many databases with tens of thousands of users that were multi-terabyte and mission critical… so I knew that these statements were spurious.

To me, the issue wasn’t whether the database could scale, but whether the customer architected a scalable architecture that could handle the load and meet the business SLA.

In a database server running an enterprise application, there are many moving main moving parts. Server, network, OS, app servers, web servers, batch servers, software, HBAs, fabric switches, application customization, etc… All can lead to issues. Regrettably, it’s usually the database that gets the blame.

This note will outline tools and techniques to eliminate the database software as the bottleneck and help the team start looking for the real issue, which for database scalability is usually the disk sub system.

Tools & Usage

The first objective is to test outside the SQL Server environment. The reason for this is to eliminate the database as an issue. If you get poor performance outside of SQL Server, then it’s not the database, it’s something else… usually the disk configuration.

Disk Copy

The easiest test to run is a file copy. Simply find a large file, for example, 2G in size and then copy it from one disk to another. You can create this test file by making a test database with a data file of 2G. Detach the database files using sp_detach_db. Then xcopy the files from your drive F: to drive G:.

CREATE DATABASE test

( NAME = test_dat,

FILENAME = 'c:\tmp\test_dat.mdf',

SIZE = 2000MB)

LOG ON

( NAME = test_log,

FILENAME = 'c:\tmp\test_log.ldf',

SIZE = 1MB )

sp_detach_db test

Another way to create a large file is to use the Windows NT Resource utility "creatfil.exe" which can create a large fixed size file.

Then go to a DOS command prompt and run the following batch file:

time /t

xcopy F:\temp\test_dat.mdf G:\temp\test_dat.mdf

time /t

At this point there are few things to remember:

Make sure you are copying data to physical different drives. Please remember that different LUNs can still share the same physical drives and controller.
Run a PERFMON trace and look at the logical transfer rates. This is outlined in the previous note on latency. If you are seeing times greater than 10 ms, you probably have a configuration issue of your SAN. You may need more spindles.
Look at how many MB/sec you are pushing through. Take the size of the data file (ex. 100MB) and divide it by how many seconds it took to copy it. To get an idea of throughput, the following table gives a broad idea to compare your results with:

Laptop	34 MB/sec
Poor SAN	35 MB/sec
Good SAN	180 MB/sec
Great SAN	412 MB/sec

Lastly, run the same test on another server that is running “well”. Compare the results.

SQLIOSTRESS

SQLIOSTRESS is a free tool that Microsoft offers to create a disk load that simulates SQL Server activity. Please note, you do not have to have SQL Server installed on the server to run it. Hence, it makes a good tool to run prior to creating your database and installing a lot of software that you may have to redo in case you have to flatten and rebuild your SAN to get adequate numbers.

SQLIOSTRESS can be obtained at this location:

http://support.microsoft.com/kb/q231619/

The tool will give the DBA plenty of information. What I like to do is run the tool and then monitor the IO patterns with PERFMON.

IOMETER

IOMETER is another tool that creates a large workload on the disk drive sub system. The tool is not database specific and allows the user to configure the type of workload and IO patterns that need to be run against the disk This tool is also free. It can be downloaded from:

http://www.iometer.org/doc/downloads.html

Again, use PERMON to watch while the tool is running. Look at the output reports from IOMETER.

SQLIO

Lastly, another Microsoft tool that can be downloaded is SQLIO. The tool is located at:

http://download.microsoft.com/download/f/3/f/f3f92 f8b-b24e-4c2e-9e86-d66df1f6f83b/SQLIO.msi

This tool will generate data and give an output report. A sample run on a small file from my laptop looks like this:

C:\Program Files\SQLIO>sqlio -kR -s360 -fsequential -o8 -b64 -LS -dE -Fparameter.txt
sqlio v1.5.SG
using system counter for latency timings, -1302407296 counts per second
parameter file used: parameter.txt
file C:\TestFile\TestFile.dat with 16 threads (0-15) using mask 0x0 (0)
16 threads reading for 360 secs from file C:\testdir\test.dat
using 64KB sequential IOs
enabling multiple I/Os per thread with 8 outstanding
using specified size: 100 MB for file: C:\testdir\test.dat
initialization done
CUMULATIVE DATA:
throughput metrics:
IOs/sec: 6755.07
MBs/sec: 422.19
latency metrics:
Min_Latency(ms): 2
Avg_Latency(ms): 18
Max_Latency(ms): 47

Summary

Secure a good baseline by running some of the above tools and processes. Then compare the numbers to other “good” systems you have running and to other published data. Work with your drive vendor if the throughput numbers are not like you would suspect. Keep in mind, you get what you pay for. If you spend little, don’t expect 500 MB/sec throughput!

After running the above types of test, I’ll often run a BACKUP to a NUL: device and then compare a database operation to a non-database run. For example, what is the MB/sec throughput of a BACKUP to NUL: versus an XCOPY. If they are the same, then you know the database is not the issue and it’s the disk that is probably the bottleneck.

The key idea in problem isolation and resolution is figure out which part of the equation is causing the problem To do that, you need to break up the variables and test them independently. The above tools give you mechanisms in which to do this.

Addendum:

Microsoft TechNet Article: Ruling Out Disk-Bound Problems

Focuses on perfmon counters. Covers Exchange and some database.

http://www.microsoft.com/technet/prodtechnol/exchange/guides/TrblshtE2k3Perf/5bcdd349-dcc6-43eb-9dc3-54175f7061ad.mspx?mfr=true

IMPORTANT NOTE: When running these tests, make sure that the data used is greater than the size of the SAN cache. The reason for this is the SAN cache will buffer the file and distort the results. For example, if the SAN cache is 1G in size, then make the file used in XCOPY or SQLIO to be 2G in size.

Powered By Qumana

Finding the Top 10 Long Running Queries

2006-06-26T14:56:00.000-04:00

Introduction

When having a performance issue, the first thing the DBA needs is to define what the problem is. The first thing I ask when someone says, “it’s running slow…” is to respond, “can you please give me a list of the top 10 worst queries.” Usually, the response is, “I don’t know exactly what they are…”

This note will explain how to isolate the queries by letting the computer tell you where the problems are.

The process is simple, it encompasses the following methodology:

Turn on SQL Server Profiler
Run it for a few hours filtering on long duration or high reads
Save the profiler trace into a temporary table
Run a few queries against the data
Prioritize them as a working list to attack

The key concept: Long running queries hammer the disk and cause poor cache hit ratios. If too many users run them, the disk subsystem can suffer because a few users are monopolizing all the resources.

Collecting the Data

Typically, I’ll start up profiler and run it for 2 or 3 hours to capture a representative sample of data. Then I’ll use this information to make my decisions. This data collected will serve as a baseline for whether I got better or worse as I tune.

Start up SQL Server Profiler. Collect on these two events:

RPC:Completed
SQL:BatchComplete
These two will show queries that have completed.

Filter on columns:

Duration and/or,

The criteria should start off with 30,000

The unit of measure is milliseconds, hence, 30,000 = 30 seconds.

Reads

The criteria should start with 10,000

The unit of measure is 8K. 10,000 reads = 81,920,000 bytes of IO. If you are doing 81M of IO, you probably have a query that needs investigating!

Let the trace run for a while. Then stop is and “Save As” a profiler trace file. Once it’s in a file, the DBA can start analyzing the data.

Rolling Up Queries

Usually, the easiest way to analyze the information is from within SQL Server. Import the trace file and then run queries against it to find the problems.

The trace file itself has the issues in it. We’ve already filtered for long running queries. Now, we just need to organize the data a bit.

First import the trace file using the following SQL Server function call:

use tempdb

SELECT IDENTITY(int, 1, 1) AS RowNumber, * INTO profiler_analysis

FROM ::fn_trace_gettable('c:\tmp\profilerdata.trc', default)

Next, get an idea of what you are looking at. For example, how much IO occurred for the monitoring run? What was the overall duration for all the long running queries?

select sum(Reads)*8192. 'Bytes Read' from profiler where Reads is not NULL;

Bytes Read

---------------------------------------

277,818,179,584

(1 row(s) affected)

select sum(Duration)/1000. 'Number of Seconds' from profiler_analysis where Duration is not NULL;

Number of Seconds

---------------------------------------

8914.941000

(1 row(s) affected)

select sum(Duration)/1000./3600.'Number of Hours' from profiler_analysis where Duration is not NULL;

Number of Hours

---------------------------------------

2.47637250000

(1 row(s) affected)

The following query shows the total amount of Reads by user:

select convert(char(20),LoginName)'User Name', sum(Reads)*8192. 'Total Bytes Read'

from profiler_analysis

where Reads is not NULL

group by LoginName

order by sum(Reads) desc

User Name Total Bytes Read

-------------------- ---------------------------------------

jde 178276974592

sa 53321981952

usera 20445822976

userb 10917101568

userc 5227069440

userd 2638151680

usere 2081947648

userf 2063392768

userg 1147445248

userh 670384128

useri 406921216

userj 316260352

userk 169639936

userl 55287808

userm 43941888

usern 19152896

usero 9584640

userp 4866048

userq 2252800

(19 row(s) affected)

The following query shows the total amount of seconds by user:

select convert(char(20),LoginName)'User Name', sum(Duration)/1000. 'Seconds Run'

from profiler_analysis

where Duration is not NULL

group by LoginName

order by sum(Duration) desc

User Name Seconds Run

-------------------- ---------------------------------------

jde 5456.860000

JDEService 1999.540000

sa 313.579000

usera 240.462000

userb 176.452000

userc 135.483000

userd 115.636000

usere 100.881000

userf 90.918000

userg 76.247000

userh 52.656000

useri 40.941000

userj 37.466000

userk 28.084000

userl 19.438000

userm 11.656000

usern 11.329000

usero 4.673000

userp 2.640000

(19 row(s) affected)

Finally, these two queries show the DBA the top 10 queries for Reads and Duration:

select top 10 RowNumber, Duration, Reads, LoginName

from profiler_analysis

order by Reads desc

RowNumber Duration Reads LoginName

----------- -------------------- -------------------- -----------

485 257230 3886609 sa

239 87690 1370174 usera

853 101810 1264835 userb

629 142370 1264577 jde

7 118890 1264197 JDE

747 8596 801035 sa

289 13970 740066 sa

264 7063 661617 sa

665 8576 356531 jde

193 3483 313031 userb

(10 row(s) affected)

select top 10 RowNumber, Duration, Reads, LoginName

from profiler_analysis

order by Duration desc

RowNumber Duration Reads LoginName

----------- -------------------- -------------------- -----------

503 335213 23 JDEService

502 333026 631 JDEService

485 257230 3886609 sa

528 224200 108896 jde

831 203590 2 JDEService

347 184183 103651 jde

532 181400 14 JDEService

627 175056 77320 jde

411 153933 307751 JDE

823 152746 23 JDEService

(10 row(s) affected)

To find the actual query for RowNumber 485 run a select statement and get the TextData column which will have the statement. The following analysis shows that the high IO and duration were due to an index being built:

select RowNumber, Duration, Reads,TextData

from profiler_analysis

where RowNumber = 485

RowNumber Duration Reads TextData

----------- -------------------- -------------------- ---------------

485 257230 3886609

create index x on CRPDTA.F4111( ILITM, ILLOCN, ILLOTN, ILKCOO, ILDOCO, ILDCTO, ILLNID )

The query for RowNumber 155 shows that it did 162,386 Reads for 1,330,266,112 bytes probably because the user put a wild card on the front and back of the criteria: F4220 WHERE (F4220.SWLOTN LIKE @P1) '%208547%' And forced a table scan.

select RowNumber, Duration, Reads,TextData

from profiler_analysis

where RowNumber =155

RowNumber Duration Reads

----------- -------------------- --------------------

155 10186 162386

(1 row(s) affected)

declare @P1 int set @P1=180151263 declare @P2 int set @P2=1 declare @P3 int set @P3=1 declare @P4 int set @P4=5 exec sp_cursoropen @P1 output, N'SELECT F4220.SWSHPJ, F4220.SWLITM, F4220.SWDCTO, F4220.SWSRL2, F4220.SWDSC2, F4220.SWSRL1, F4220.SWLOTN, F4220.SWLOCN, F4220.SWAITM, F4220.SWSFXO, F4220.SWDOCO, F4220.SWAN8, F4220.SWITM, F4220.SWMCU, F4220.SWDSC1, F4220.SWLNID, F4220.SWORDJ, F4220.SWKCOO, F4220.SWVEND, F4220.SWSHAN FROM PRODDTA.F4220 F4220 WHERE (F4220.SWLOTN LIKE @P1) ORDER BY F4220.SWLITM ASC, F4220.SWLOTN ASC, F4220.SWSRL1 ASC ', @P2 output, @P3 output, @P4 output, N'@P1 varchar(8000) ', '%208547%' select @P1, @P2, @P3, @P4

Summary

The above queries collect data and roll it up into a digestable format instead of a pile of random bits. Once the DBA has the performance data they can prioritize the issues and figure out a plan of attack. For example, change the configuration of the application so users cannot put in leading wildcards that force a table scan.

Knowing your system and where it’s issues are forms the backbone of being able to isolate and resolve performance problems. The good news is it’s to do with a small tool box.

The key concept is to let SQL Server collect the information, and using a few simple queries shown above, show you where the exact problems exist.

Powered By Qumana

Disk Queue Length vs. Disk Latency Times: Which is Best for Measuring Database Performance

2006-06-22T20:09:00.000-04:00

Introduction

There are several common metrics that people use to look for performance bottlenecks. They are usually:

- Disk queue length

- Disk latency/transfer times

- MB/sec

- IO/sec

For the rest of this note, I’ll focus on the top two.

Disk Queue Length

The most common performance monitoring metric people quote is usually disk queue length. While this is important, now days it’s almost impossible to use as an accurate metric because the SAN used in an enterprise database application abstracts the DBA from a lot of the physical characteristics needed to make a reasonable judgment about what is good or bad.

The usual “rule of thumb” for “bad performance” is a queue length greater than 2 for a disk drive. But when you have a SAN with 100 drives in the cabinet, you have no idea how many are being used for your “drive F:” for your data file.

For example, the DBA watches perfmon and sees the disk queue length is 10 on drive F:. While this is a big number, it’s relative. If the F: is one physical disk, yes, you have a problem. If the F: is a logical disk composed of 8 physical disks, then it’s not an issue. Why? Because 8 disks x 2 queue length = 16… and the DBA sees 10… so 10 is less than 16 (our threshold for pain)… hence there is not an issue.

The main problem with queue length is this: very few people can ever provide me with a comprehensive disk map of their SAN showing all the logical volumes. For example, how many disks are participating in an EMC Sym hyper volume.

The following is a very small example of what a mapping would look like. Each of the “73” are 73G disk drives. You can see multiple drives participating in the LUN (Note, this is RAID 0+1 so you see 2 73G drives making up a 59.39G formatted capacity):

RG0, R1, 60.39 GB		RG1, R1, 60.39 GB		RG2, R1, 60.39 GB		RG3, R1, 66.61 GB
73	73	73	73	73	73	73	73
00_00	00_01	00_02	00_03	00_04	00_05	00_06	00_07
Private System Partition						LUN 30 (1 GB), Unallocated (65.61 GB)
LUN 0 (1 GB), Unallocated (59.39 GB)		LUN 10 (1 GB), Unallocated (59.39 GB)		LUN 20 (1 GB), Unallocated (59.39 GB)
LUN 0 (1 GB), Unallocated (59.39 GB)		LUN 10 (1 GB), Unallocated (59.39 GB)		LUN 20 (1 GB), Unallocated (59.39 GB)

Without a map like this, disk queue length is meaningless to me. I have nothing to calculate what the queue length should be less than.

With regards to SQL Server, it will not let the transaction log volume have a queue length of more than 3. Thus, the DBA may never see a “problem”. The reason for this is SQL Server is trying not to swarm the disk drives and will throttle back the work to the log. Since the database follows the Write Ahead Log (WAL) protocol, if it can’t write to the log fast enough, it will not be able to write to the data volumes either… and it will slow down the whole system’s throughput.

Per the previous posting on throughput testing, if you don’t have enough spindles to support the volume of data being pushed through the system, the whole box will slow down. This would be especially important in an enterprise app such as SAP or PeopleSoft that run large batch jobs and are writing to the transaction log constantly.

Disk Transfer Time

As you can read from above, disk queue length can be deceptive. You need a lot of information to make a good judgment.

Disk transfer time is what the disk sub system is reporting to the operating system. It abstracts the DBA from having to know the absolute physical characteristics of the underlying disk.

The key point is that it keeps the DBA from rat holing on a lot of very specific information that most people don’t have… namely, a comprehensive disk mapping of their SAN. It also distills a lot of other complicated measurements down to a few simple ones that mean the most.

For example, if I have a disk queue length of 20 on my F: drive, but my response time is less than 5ms… then everything is working great. What this probably means is that the F: drive has 20 (or more) spindles behind it or that there is massive caching taking place.

In the context of disk latency, the following is an example of modern performance characteristics of a SCSI disk drive:

• Maxtor Atlas 15K II

– Ultra320 SCSI

– Average 3ms, Max 8ms

– Max Sustained 98 MB/sec

– Burst 320 MB/sec

– 8M Cache

– 15K RPM

Source Maxtor: http://www.maxtor.com/_files/maxtor/en_us/documentation/data_sheets/atlas_15kii_data_sheet.pdf

To monitor the times:

Start Windows perfmon.
Use counter “Logical Disk”
Select “Avg Disk/Sec” and choose Read, Write or Transfer depending on what you want to monitor. I typically choose Transfer.

Note, when you read the perfmon data and see a number like “.010” this means 10 milliseconds.

Additionally, when monitoring a SAN, they often have their own perfmon counter you need to use. For example, EMC and Veritas have their own.

There is plenty of room for intelligent people to argue on the following data, but I use the following table to determine the meaning of the data:

10 ms		Good
10 ms	20 ms	Reasonable
20 ms	50 ms	Busy
50 ms +		Bad

I have a good friend that will credibly argue that anything more than 1 ms response times is a poorly configured SAN. He has customers that get in the 250 to 500 microsecond responds times.

Note, many times the performance problems are tied to firmware revisions, HBA configuration, and/or BIOS issues.

Summary

When you start focusing on response time, which is what really matters most, the queue length starts to become irrelevant. If you can get some kind of super drive that can handle everything, terrific! (Think: Solid State RAM Drives.)

What the latency metric does is cut through a lot of the red tape that architects usually have. This metric gives the DBA the ability to just tell the SAN architect when they lay out the LUNs is, “Slice the disk in a way where I get 5 ms disk access times. The rest is up to you.”

If the database can get sustained 5ms disk access times with a heavy stress test (ex. SQLIOSTRESS) then it can probably handle the load of the user base with the app running… which is what you ultimately want for success.

Powered By Qumana

Disk Performance and BACKUP to NUL: Command

2006-06-20T18:26:00.000-04:00

Introduction

Disk drive performance and databases are directly tied to each other with backups. A full backup, by definition, will start at the beginning of a database and read all the way through to the end. In the process, it can generate a large amount of disk IO. If you have a slow disk subsystem, it can show up here.

As you will see in the following examples, slow backups can be an indication of overall system health and poor database architecture.

The following table illustrates the different IO patterns that SQL Server uses when going after data. It is possible that you can get good performance in an OLTP environment because it’s doing many small 8K IOs, and then have poor performance when running backups because it’s doing 1M IOs.

OLTP Log	Sequential	Write	512 bytes - 64KB
OLTP Data	Random	R/W	8K
Bulk Insert	Sequential	Write	8K multiples to 128K
Read Ahead	Sequential	Read	8K multiples to 256K
Backup	Sequential	R/W	1MB
Restore	Sequential	R/W	64K
CHECKDB	Sequential	Read	8K - 64K

Table: SQL Server Disk IO Access Patterns

Source: http://blogs.msdn.com/sqlcat/

One of the best reasons to use the backup command is its uniformity and simplicity. Every SQL Server has it which enables comparison between different platforms, and at the end of the backup, it summarizes for you the actual throughput so you don’t have to even calculate anything.

BACKUP to a Null Device

The main purpose of a backup is an offline copy of the database for possible recovery purposes. Typically backups dump to disk or tape. In some cases, it’s beneficial to make a backup, but not store the results anywhere. For example, if you want to run through an operational process for practice… or in our case, stress the disk subsystem.

When a backup is run, it reads from the database, and then writes to the backup media. Hence it will read then pause then read then pause when accessing the database itself. The pause to the database occurs as it’s writing to the backup media. This does not help with stress testing by putting a constant load on the system.

The only way to put a full load using the BACKUP command is running it with the NUL: device syntax.

Syntax:

backup database <db_name> to disk = ‘NUL:’ with stats = <perc compl>

Example of Usage:

backup database siebelprd to disk = ‘NUL:’ with stats = 1

Sample Output:

Processed 1416 pages for database 'newmsdb', file 'MSDBData' on file 1.

100 percent backed up.

Processed 1 pages for database 'newmsdb', file 'MSDBLog' on file 1.

BACKUP DATABASE successfully processed 1417 pages in 0.071 seconds (163.392MB/sec).

Backing up to the NUL: device will cause a full time read to the database without pausing to write to the backup media.

Another issue when testing databases is caching. If the same test is run often, the results will always be in RAM hence distorting the results. By running a backup, it usually exceeds the amount of cache on the server thus helps eliminate this distortion.

Judging the Results: Bad, Good, Best

Once the command is run, the DBA needs to know what the results mean. The following examples were taken from customers I have worked with over the last 2 years and help illustrate what the throughput numbers mean.

Laptop Disk Drive With 36MB Database

99 percent backed up.

100 percent backed up.

BACKUP DATABASE successfully processed 4473 pages in 1.172 seconds (31.259 MB/sec).

Desktop internal disk drive 3.4GB database:

100 percent backed up.
BACKUP DATABASE successfully processed 425841 pages in 135.945 seconds (25.660 MB/sec).

Bad

Customer’s 1.6TB database with 12 CPU and High End SAN RAID 1:

100 percent backed up.

BACKUP DATABASE successfully processed 202986177 pages in 47070.773 seconds (35.326 MB/sec).

This is almost the same IO as my laptop (31.259 MB/Sec)?!

Concern

Customer’s 87GB database with 8 CPU 32G and High End RAID 5:

ファイル 1 で、データベース 'siebeldb' 、ファイル 'siebeldb_prm1' の 2670712 ページが処理されました。

ファイル 1 で、データベース 'siebeldb' 、ファイル 'siebeldb_prm2' の 2665816 ページが処理されました。

ファイル 1 で、データベース 'siebeldb' 、ファイル 'siebeldb_prm3' の 2662176 ページが処理されました。

ファイル 1 で、データベース 'siebeldb' 、ファイル 'siebeldb_prm4' の 2662064 ページが処理されました。

ファイル 1 で、データベース 'siebeldb' 、ファイル 'siebeldb_log1' の 47 ページが処理されました。

BACKUP DATABASE は 10660815 ページを 789.876 秒で正常に処理しました (110.565 MB/ 秒 ) 。

Good

Customer 575GB SAN:

100 percent backed up.

BACKUP DATABASE successfully processed 70246627 pages in 3540.611 seconds (162.531 MB/sec).

Microsoft Dev Database 101G with RAID10 on 146Gb 10K drive on SAN:

100 percent processed.

BACKUP DATABASE successfully processed 12376579 pages in 707.413 seconds (143.323 MB/sec).

Microsoft DSS database 28GB database High End SAN 72G 15K:

100 percent backed up.

BACKUP DATABASE successfully processed 3468866 pages in 151.831 seconds (187.161 MB/sec).

Microsoft OLTP Database 2TB with 72Gb 15K drives Raid10 on SAN:

100 percent backed up.

BACKUP DATABASE successfully processed 244138625 pages in 5561.102 seconds (359.638 MB/sec).

Great

Processed 25342792 pages for database 'TUC', file 'data1' on file 1.

Processed 25215088 pages for database 'TUC', file 'data3' on file 1.

Processed 25215168 pages for database 'TUC', file 'data5' on file 1.

Processed 25214320 pages for database 'TUC', file 'data2' on file 1.

Processed 25214344 pages for database 'TUC', file 'data4' on file 1.

Processed 25215168 pages for database 'TUC', file 'data6' on file 1.

Processed 25213704 pages for database 'TUC', file 'data7' on file 1.

Processed 25214552 pages for database 'TUC', file 'data8' on file 1.

Processed 25215200 pages for database 'TUC', file 'data9' on file 1.

Processed 25214280 pages for database 'TUC', file 'data10' on file 1.

Processed 25214920 pages for database 'TUC', file 'data11' on file 1.

Processed 25321384 pages for database 'TUC', file 'data12' on file 1.

Processed 3 pages for database 'TUC', file 'log1' on file 1.

BACKUP DATABASE successfully processed 302810923 pages in 6010.430 seconds (412.720 MB/sec).

DB Size= 2,365,619Mb

Backup time to Null: 1 hr. 40 min.

What to Do?

If you see that you are having slow throughput times (low MB/sec), the fixes are usually to add more spindles to the database and spread the load out more and/or increase bandwidth to the disk by adding faster/more controllers.

Warning!

This command runs great accept under the following conditions:

If you run a differential backup, don’t run this command! The reason is that the differential backup picks up pages that have changed. Running the backup command resets the flag on the page. Since the backup to a NUL: device reads the page, but does not back it up to media, the flag is reset as if it HAD been backed up… so the database does NOT back it up during the next differential. If you have run a backup to NUL: and do differential backups, run a FULL backup immediately and everything will be fine.
If you have heavy disk IO now, a backup to a NUL: device will only add more stress slowing things down more.
For reason #2, run this command in off hours.

Summary

Backup to a NUL: device is a handy command. It allows you to compare disk performance throughput of a laptop to a high end database server with a SAN. It’s a poor man’s stress test and doesn’t take much to setup or run… or interpret the results. It will not fix any of the problems, but hopefully show whether the database as a whole can sustain massive IO within its architecture.

BOE (Back of Envelope) Method – Quick and Dirty Method for Finding High CPU Jobs

2006-06-18T17:08:00.000-04:00

Situation: Your database server is sitting at 100%. There are not that many users logged into SAP doing work. Somehow, a bad plan is running and using all your resources on the computer. Problem is, you have no idea which spid(s) have the bad plan!

Within SQL Server it is extremely difficult to find which spid is using the most CPU. There are two methods for doing this:

- Using PERFMON, looking at the Thread Performance Object: % Processor Time, ID Process, ID Thread. Select ALL SQL threads (sqlservr/*). Then flipping it to “tabular” report format (ie. NOT the graph) and scrolling over 100 entries to find the thread that is using the most CPU time. Since a query can move from thread to thread, it’s a real tough time. Once you have the thread with the CPU, then get the ID Thread. The ID Thread is the KPID in sysprocesses. Join on this to get the SPID. Then DBCC INPUTBUFFER(spid) to see what the query is running. You can also KILL the spid, too. The ID Process is the NT PID.

select spid

from master..sysprocesses

where kpid =

DBCC INPUTBUFFER()

- As you can see, this is a difficult process. A quick and dirty method that I use that isn’t 100% accurate, but is about 75% correct is to run the following query. The long running jobs that this query reports tend to be the ones using the most CPU time.

CREATE proc boe_proc as

select datediff(mi, last_batch,getdate())'minutes',spid, waittype, cpu, physical_io, convert(char(15),hostname), convert(char(15),program_name), convert(char(20),getdate()),spid, last_batch, cmd

from master..sysprocesses

where spid > 50 and

cmd not like '%WAIT%' and

datediff(mi, last_batch,getdate()) > 1

order by last_batch

Here’s sample output, it shows the backup running and a batch job that has been going for 5 minutes app server MSSAP32 Batch Process (BTC) 20.

exec boe_proc

minutes spid waittype cpu physical_io spid last_batch cmd

----------- ------ -------- ----------- -------------------- --------------- --------------- -------------------- ------ ------------------------------------------------------ ----------------

389 150 0x0422 2424729 6929618 MSSAP01 R3D03(3) unc rd May 9 2002 6:05PM 150 2002-05-09 11:36:02.040 SELECT

65 1687 0x0000 2875 5756332 MSSAP99 SQLAgent - TSQL May 9 2002 6:05PM 1687 2002-05-09 17:00:03.267 BACKUP DATABASE

65 1687 0x0000 63828 5756332 MSSAP99 SQLAgent - TSQL May 9 2002 6:05PM 1687 2002-05-09 17:00:03.267 BACKUP DATABASE

65 1687 0x0000 62391 5756332 MSSAP99 SQLAgent - TSQL May 9 2002 6:05PM 1687 2002-05-09 17:00:03.267 BACKUP DATABASE

65 1687 0x0000 59281 5756332 MSSAP99 SQLAgent - TSQL May 9 2002 6:05PM 1687 2002-05-09 17:00:03.267 BACKUP DATABASE

65 1687 0x0000 29703 5756333 MSSAP99 SQLAgent - TSQL May 9 2002 6:05PM 1687 2002-05-09 17:00:03.267 BACKUP DATABASE

65 1687 0x0000 27984 5756333 MSSAP99 SQLAgent - TSQL May 9 2002 6:05PM 1687 2002-05-09 17:00:03.267 BACKUP DATABASE

65 1687 0x0000 29671 5756333 MSSAP99 SQLAgent - TSQL May 9 2002 6:05PM 1687 2002-05-09 17:00:03.267 BACKUP DATABASE

65 1687 0x0000 33641 5756333 MSSAP99 SQLAgent - TSQL May 9 2002 6:05PM 1687 2002-05-09 17:00:03.267 BACKUP DATABASE

65 1687 0x0000 30922 5756333 MSSAP99 SQLAgent - TSQL May 9 2002 6:05PM 1687 2002-05-09 17:00:03.267 BACKUP DATABASE

65 1687 0x0000 30859 5756333 MSSAP99 SQLAgent - TSQL May 9 2002 6:05PM 1687 2002-05-09 17:00:03.267 BACKUP DATABASE

65 1687 0x0000 28438 5756333 MSSAP99 SQLAgent - TSQL May 9 2002 6:05PM 1687 2002-05-09 17:00:03.267 BACKUP DATABASE

65 1687 0x0000 30703 5756333 MSSAP99 SQLAgent - TSQL May 9 2002 6:05PM 1687 2002-05-09 17:00:03.267 BACKUP DATABASE

65 1687 0x0000 28781 5756333 MSSAP99 SQLAgent - TSQL May 9 2002 6:05PM 1687 2002-05-09 17:00:03.267 BACKUP DATABASE

65 1687 0x0000 28719 5756333 MSSAP99 SQLAgent - TSQL May 9 2002 6:05PM 1687 2002-05-09 17:00:03.267 BACKUP DATABASE

65 1687 0x0000 29359 5756334 MSSAP99 SQLAgent - TSQL May 9 2002 6:05PM 1687 2002-05-09 17:00:03.267 BACKUP DATABASE

65 1687 0x0000 30844 5756334 MSSAP99 SQLAgent - TSQL May 9 2002 6:05PM 1687 2002-05-09 17:00:03.267 BACKUP DATABASE

65 1467 0x0800 7467502 12792823 MSSAP32 R3B14(3) unc rd May 9 2002 6:05PM 1467 2002-05-09 17:00:28.953 SELECT

35 1333 0x0800 6006302 13123578 MSSAP31 R3B17(3) unc rd May 9 2002 6:05PM 1333 2002-05-09 17:30:26.043 SELECT

5 1503 0x0208 643300 6526323 MSSAP32 R3B20(3) unc rd May 9 2002 6:05PM 1503 2002-05-09 18:00:30.493 SELECT

5 1503 0x0404 8078 6526323 MSSAP32 R3B20(3) unc rd May 9 2002 6:05PM 1503 2002-05-09 18:00:30.493 SELECT

5 1503 0x0422 8031 6526323 MSSAP32 R3B20(3) unc rd May 9 2002 6:05PM 1503 2002-05-09 18:00:30.493 SELECT

5 1503 0x0404 8000 6526323 MSSAP32 R3B20(3) unc rd May 9 2002 6:05PM 1503 2002-05-09 18:00:30.493 SELECT

5 1503 0x0404 7453 6526323 MSSAP32 R3B20(3) unc rd May 9 2002 6:05PM 1503 2002-05-09 18:00:30.493 SELECT

5 1503 0x0404 7906 6526323 MSSAP32 R3B20(3) unc rd May 9 2002 6:05PM 1503 2002-05-09 18:00:30.493 SELECT

5 1503 0x0404 7750 6526323 MSSAP32 R3B20(3) unc rd May 9 2002 6:05PM 1503 2002-05-09 18:00:30.493 SELECT

(26 row(s) affected)

McBath's Oil Argument

2006-06-18T16:22:00.000-04:00

McBath’s Oil Argument

The argument postulates that when you change the oil in your car, do you notice it going any faster? Probably not… but does your car know the difference? Definitely. If you don’t change the oil, the car’s engine will seize up.

Databases are the same. Any system can run with bad queries and a few users with brute force, but it will fail to scale. By tuning your system, you are changing your database’s “oil” and will get more users and more throughput from the same hardware.

For example, by repacking a database page from 80% full to 100% full you will go from 5 reads to 4 reads for the same amount of data. While you can’t even blink your eyes that fast, the aggregate effect is that a long running batch job that iterates on this small query will run 20% faster—and on a long payroll job, this can be significant and measured in hours saved.

Another example… It’s easy to find the few long running queries. I’m usually not scared of those. They are easy to locate with Profiler. What I am scared of is the query being run 10,000 times a day that is going very fast (sub-second), but doing more IO than it needs to. The aggregate of this common query can consume a lot of resources over time. If I can find those queries, then I can free up more disk and CPU resources for other needs.

I’ll use McBath’s Oil Argument fairly often when writing in the future.

Oracle: RAC Content on Windows 2003 64 bit

2006-06-14T12:49:00.001-04:00

A few weeks ago, Microsoft gave a free three day workshop on Oracle RAC running on Windows 2003 64 bit. The workshop was recorded along with the slide decks and posted at the following site:

http://www.microsoft-oracle.com/technical_content/oracle/workshops.aspx

The bulk of Oracle's materials cover Linux and RAC. This was really the first time all of it has been comprehensively put together in one spot covering Oracle 10g & Windows 2003.

There's also some side stories in the presentation that cover Windows 2003 & iSCSI. The course materials cover how to install Oracle RAC on Windows 2003 iSCSI with Microsoft's new aquisition of String Bean.

Additionally, Jim Olsen, Sr. Director of Oracle Enterprise Technology Center, gives an excellent introduction to Oracle's vision on grid computing and database server technology. He discusses Real Application Clustering concepts in depth (how they provide scalability and high availability). Oracle's commitment to Windows Server, and Oracle and Microsoft interoperability and partnership are also discussed.

Inaugural Post: The Mission

2006-06-13T17:26:00.000-04:00

The purpose of this blog is to facilitate the exchange of ideas between people regarding databases and how disk drives interact with them in a high performance, available, and scalable way.

While this mandate sounds very dry and technical... which it is... there really seems to be no central clearing house specifically around this area. There's plenty of information about databases and plenty around disk perf and SANs... very little intersects with practical daily application, and when you can find it... it's buried in a white paper on a vendor's website.

My basic approach is that everyone has a database and everyone has performance problems... and you can only go as fast as your slowest part... usually your disk. I'd like to cover the technical issues, tools, balanced system design and methodologies around isolating and resolving these issues.

In particular, my area of interest is around database/disk performance with large enterprise applications. For example, SAP, PeopleSoft, Siebel, and JD Edwards.

This blog will hopefully help make matters clear as to how these all these parts work together.

Please let me know how I can make this blog work for you better.

Best-
Frank