Accelerating ETL Data Retrieval

As I’ve mentioned previously, SQL Server 2012 Integration Services Design Patterns is now available on Amazon. PASS has invited the ENTIRE author team to give a pre-conference training session at Summit 2012. Precons are all-day training events and are held on the days before Summit. Our precon will be held on Monday, November 5th. I’ve attended several precons at Summit, and in my opinion, they provide unparalleled depth and access to the presenters. I expect our precon will be no different.

So what will I be presenting on? Good question. I’m going to talk about Accelerating ETL Data Retrieval. Specifically, I’ll discuss strategies for retrieving data for full loads vs. incremental loads, and I’ll explore some strategies for retrieving data from large tables. To clarify “large,” since everyone has a different definition, I’ve successfully employed these strategies on tables with 10 billion rows.

Now you might be reading this and thinking, “Hey, that doesn’t sound like SSIS!” Well… yes and no. While I will be discussing some strategies within SSIS, most of what I discuss will take place at the database layer. In fact, most of my content could be useful for any ETL tool. is ivermectin recommended to treat the coronavirus I still think it’s a good topic for this precon for three reasons. First, most ETL developers I talk with — that is, people who specialize in and are primarily tasked with ETL — overlook the data retrieval step. I frequently hear, “I’m pulling all of the data, so it doesn’t matter, I can’t get the data any faster than SELECT * FROM. ivermectin hookworms ” That’s not always true, and I’ll explain why in the precon. Secondly, having a thorough understanding of data retrieval techniques is important. Data retrieval is one of the most — if not the single most — expensive components of ETL. Lastly, I think there’s a huge knowledge gap in this area. While there is a lot of discussion about query optimization, it rarely has to do with the type of query patterns that ETL requires.

So that’s what I’ll be talking about and why. What do you think? Worthwhile content?

Also, this is just one of the things that will be covered in our SSIS Design Patterns precon. The rest of the author team — Andy Leonard, Matt Masson, Tim Mitchell, and Jessica Moss — is putting together great content that includes package execution, parent/child relationships, expressions and scripting, dynamic configurations, error handling, data flow internals, scalability and parallelism, automation, dynamic package generation, data warehousing patterns, and more!

As an aside, I am honored to announce that I have been re-awarded as Microsoft MVP for SQL Server. My husband took this picture of my trophy for me, which I think turned out really cool. ivermectin dog heartworm ๐Ÿ™‚

East Iowa SQL Saturday is Next Saturday! (Oct 1st)

East Iowa’s 3rd annual SQL Saturday is right around the corner! This year’s event will be on Saturday, October 1st. It’s being held in the same location as previous years, at the University of Iowa Capitol Centre in Iowa City.

We have a great mix of speakers this year, including nationally-recognized speakers such as Jason Strate and Ted Krueger. Topics include Denali, PowerPivot, High Availability, Disaster Recovery, PowerShell, SSIS, Analysis Services, Reporting Services, database tuning, and more. Don’t miss out on this great opportunity to network with SQL-savvy locals while receiving FREE training.

If you haven’t registered yet, please do so we can make sure we order enough food. Oh, and make sure to stick around for the after-party, too. The after-party is one of my favorite parts of the whole event. This year’s party will be held at The Mill, which is within walking distance of the conference center.

I hope to see you there! ๐Ÿ™‚

Yet Another PASS Summit Recap & Thoughts on PDW

The SQL blogosphere has been lit up with PASS Summit recaps.

I debated about whether or not to write my own post, until I remembered that this blog serves as a mini-journal for me too. I have a notoriously poor memory–my husband likes to say that my CPU and memory are good, but I must have an unusual clustering strategy–so maybe this blog post will be a good pointer for me when I start prepping for next year’s Summit. ๐Ÿ˜‰

This was definitely the best PASS Summit conference ever. While there will always be opportunities to do things better–improvement is a never-ending process–it was clear that the organizers of this event listened to the feedback they had received the previous year. One of the best changes? Backpacks. These were very useful, as evidenced by their presence everywhere. Nice job, organizers!

My absolute favorite thing about Summit is the chance to meet and reconnect with so many amazing SQL folks. There were entirely too many people to list out, but some highlights include meeting Crys Manson, Jorge Segarra, and Karen Lopez for the first time. I also had a chance encounter with Ola Hallengren in the Sheraton elevator. Apparently we were only staying a few rooms apart this year. We ended up having a couple of really great discussions about index fragmentation, the differences between our scripts, and things we’d like to see changed in future releases of SQL Server.

I had the opportunity to sit on the panel at the WIT luncheon. All of the women on the panel were amazing, and I was honored just to be sitting at the same table as them. I was especially pleased to meet Nora Denzel, a Senior Vice President at Intuit. Intelligent, confident, and witty, she is a great role model for young technical women, myself included. I can only hope that some of her gumption rubbed off on me due to our close proximity. ๐Ÿ™‚ After the event, I was pleasantly surprised by how many folks–men and women both–came up to me to tell me how much they enjoyed it. Thanks to the WIT VC for organizing another great event!

The lightning talk sessions were a new feature this year, and I think I like it. The format of the lightning session is 7 speakers presenting on a topic for 5 quick minutes. Watching these sessions is kind of like skipping right to the center of a tootsie pop: all content and no fluff. The standout lightning talk presentation for me was Adam Machanic’s. It was beautifully rehearsed and choreographed. Nice job, Adam!

Another of the many highlights of the week was meeting the Microsoft execs. In addition to meeting Ted Kummert, Mark Souza, and Donald Farmer–all very nice gentlemen–I had the opportunity to speak at length with Jose Blakely about Parallel Data Warehouse (PDW). PDW, formerly codenamed Madison, was officially launched at Summit. Jose was kind enough to explain the PDW architecture, both where it came from and the vision for where it’s going. I’d attempt to regurgitate it here, but I think the probability of me misquoting would be high.

Suffice it to say, this technology has me excited. Why? Quite frankly, I think PDW will do for data warehousing what SQL Server did for databases, and what Analysis Services did for BI: make it affordable. With a compelling cost-per-terabyte, an attractive scale-out approach, and an entry point at under $1 million, we’ll see more small-to-midsized companies implementing data warehousing and business intelligence. This is good news for those of us looking for an affordable data warehouse solution and for those of us who make our living with SQL Server. And for those of you who might suggest that few companies need a datawarehouse that can support multi-terabyte data, I’d like to point out that just 3 or 4 years ago, 100 GB was considered a lot of data.

I spent most of my week digging into the PDW architecture. It’s not all roses–it’s a first release and, as such, is immature compared to the much older and more established data warehouse systems–but again, it has a lot going for it, not least of all it’s easy integration within a SQL Server environment and the relatively low cost. We’re currently investigating this as a possible data warehouse solution for our business intelligence environment, so expect to see more from me about PDW as I learn more about it.

Not attending PASS Summit? Watch LIVE streaming events FOR FREE!

If you’ve not yet heard, the annual PASS Summit is less than 2 weeks away. This is the largest SQL Server and Business Intelligence conference _in the world_, sponsored by Microsoft and Dell. The return on investment of attending this conference is pretty huge, and I highly recommend you attend if you can swing it.

I am once more fortunate to be attending and presenting at the Summit. Here’s where you can find me speaking throughout the week:

Tuesday at 3PM
Lightning Talk – Page Compression
This year, PASS has decided to try something new. A daily Lightning Talk session will be held where speakers present for 5 quick minutes on interesting SQL topics. I’ll be presenting on Tuesday with 6 amazingly talented speakers. My topic is page compression — what is it, how to do it, and (most importantly, of course) how it affects performance.

Wednesday at 11:30am in the ballroom
Women-In-Technology (WIT) Luncheon
I’ll be speaking on this year’s WIT luncheon panel, which is sponsored by Contrary to common misconception, the luncheon is NOT just for women. In fact, men are encouraged to attend! If memory serves, last year’s luncheon had about 300 attendees, with a good mix of both genders. This year’s topic is focused on the recruitment, retention, and advancement of WIT. If you’re worried that this event will end up being a feminist bitch-fest, rest assured that’s most definitely not the case. I’ve always found the WIT events I’ve attended to be informative and thought-provoking. Plus, free lunch! ๐Ÿ™‚

Thursday at 2:30PM (room 3AB)
Heaps of Trouble, Clusters of Glory – A Look At Index Internals
You can click the link above to read my abstract, but in short, I’ll be taking attendees on a journey through indexes. You’ll come away with a much better understanding of the internal structures of indexes, which should help DBA’s with database design and performance tuning.

If you’re not able to attend in person, Summit does sell DVD’s of the event afterwards, which are well worth the investment. But this year, to make the event more accessible to the community, PASS and Dell have teamed up to present live streaming of the keynotes and WIT luncheon sessions.

Here’s details of the keynotes from the PASS press release:

Ted Kummert, Senior Vice President of the Business Platform Division at Microsoft Corp., will kick off PASS Summit on November 9 by highlighting the continued innovation across Microsoftโ€™s business and information platform. Kummert will explore Microsoftโ€™s key technical investments, as well as Mission Critical applications and the accessibility of Business Intelligence.

Quentin Clark, General Manager of Database Systems Group at Microsoft Corp., will showcase the next version of SQL Server on November 10 and will share how features in this upcoming product milestone continue to deliver on Microsoftโ€™s Information Platform vision. Clark will also demonstrate how developers can leverage new industry-leading tools with powerful features for data developers and a unified database development experience.

David DeWitt, Technical Fellow, Data and Storage Platform Division at Microsoft Corp., will be discussing SQL query optimization and address why it is difficult to always execute good plans in his highly anticipated technical keynote. DeWitt will also cover new technologies that offer the promise of better plans in future releases of SQL Server.

While all of the keynotes are interesting and definitely worth watching, I cannot recommend the David DeWitt keynote more highly. His keynote last Summit was outstanding. It was technical, thought provoking, and one of the best things of last year’s Summit.

You can find more information and register for the PASS Summit 2010 live streaming events at their website,

If you ARE attending Summit, make sure to swing by and say “hi” or message me via Twitter to see if there’s a time we can meet up. Meeting people is one of my favorite things about Summit. ๐Ÿ™‚

See you in Seattle!

My Summit abstract was accepted! I’m still a little surprised, but I’m also excited (okay, and a little nervous) to once more be presenting at the PASS Summit. If you’ll be at Summit this year — and I really hope you are, as it’s well worth the time and cost — then please make sure to say “hi” if you see me wandering around. Aside from the *excellent* content, my favorite thing about Summit is getting to meet so many great people. ๐Ÿ™‚

In other news, I’ve once more switched roles within GoDaddy. For the half dozen folks who’ve been following my blog from the beginning, you may remember that I originally started out on the traffic team working with tuning and VLDB’s, then took an opportunity to switch to the BI team to learn more about OLAP. Recently, a new team has been formed under the BI branch that’s tasked with developing a massive hybrid data warehouse (by hybrid, I mean half OLTP and half OLAP). “How massive is it?” Well, it’s SO massive, we’re expecting to be store petabytes of data when everything is said and done. I’m happy to say I’ll be on this new team. So yes, that means we have an opening for an OLAP developer. We’re also hiring SQL Server DBA’s. We have offices in Cedar Rapids, Denver, and the Phoenix area. Send me an e-mail at michelle at sqlfool dot com if you’re interested in learning more about this great job opportunity and company.

Lastly, I want to announce that SQL Saturday 50 is now open for registration! SQL Saturday 50 will be held in Iowa City, IA on Saturday, September 18th. We’re almost at 50% of our attendance capacity, so if you’re interested in attending, please register soon.

That’s it for now. I promise that my next blog post will be uber technical. ๐Ÿ™‚

SQL Saturday #50 – Call for Speakers

The Call for Speakers is now open for SQL Saturday #50, the East Iowa SQL Saturday event! This is our second time hosting a SQL Saturday, and we’re hoping to build upon the success of last year’s event. We’re looking for a wide variety of topics on SQL Server and related technologies (i.e. PowerShell, R2, LINQ, etc.). We also have had several requests for intro-level topics, such as beginning disaster recovery and basic performance tuning. ู†ุชุงุฆุฌ ู…ุจุงุฑูŠุงุช ูƒูˆุจุง ุงู…ุฑูŠูƒุง 2024 If you’re even remotely thinking about speaking, please submit an abstract!

Last year we had about 100 folks attend from surrounding areas. ุดูƒู„ ูƒู„ูŠูˆุจุงุชุฑุง ุงู„ุญู‚ูŠู‚ูŠ This year, we’re shooting for 125 attendees, which would max out our facility’s capacity. Not sure how far away Iowa City is? It may be closer than you think. Allow me to rehash my travel times from last year’s plea for speakers:

  • Chicago โ€“ 3.5 hours
  • Omaha โ€“ 3.5 hours
  • Milwaukee โ€“ 4 hours
  • Kansas City โ€“ 4.5 hours
  • Minneapolis โ€“ 5 hours
  • St. Louis โ€“ 5 hours
  • Indianapolis โ€“ 6 hours

The event will be held on September 18th at the University of Iowa in Iowa City. You can find more information, including an abstract submission form, on our event website at

Oh, and if you do make it to our SQL Saturday event, please make sure to stop me and say “hi! ููˆุฑู…ู„ุง ูˆู† ” ๐Ÿ™‚

Summit 2010 Abstract Submission

Jeremiah Peschka just tweeted about the looming deadline to submit an abstract for Summit 2010. I’ve been trying to think of a good topic to present on, and this finally got my butt in gear to submit one! For those interested, here’s what I submitted:

Heaps of Trouble, Clusters of Glory – A Look At Index Internals

Indexes are a crucial component of SQL Server, especially in performance tuning, yet many DBA’s don’t fully understand how indexes work. In this in-depth session, Michelle will examine the anatomy of indexes, from how they’re stored to how they get fragmented. And to make our inner geeks happy, she’ll look at page data to show you what’s happening behind the scenes. Topics covered will include index structure, fragmentation and defragmentation, index partitioning, and index filtering. If you’ve ever wondered just *what* goes on in an index, don’t miss this session!

Special thanks to Chris for help with the session title. ๐Ÿ™‚

Live Blogging: Women In Technology Luncheon

I was invited to live-blog for the Women In Technology (WIT) luncheon at the Summit on Tuesday.ย  As was the case with all of my “live blogging,” I mostly updated Twitter with near-real-time updates, which I then attempt to transcribe into a blog post for later reference. So here follows that transcription. ๐Ÿ™‚

11:49 AM PST
The room is filling up nicely! There’s a nice distribution of men and women in the room. This is great! I’m actually pleasantly surprised at the number of women at the Summit.ย  If it weren’t for this luncheon, I probably wouldn’t have even noticed just how many SQL ladies there actually are.

12:00 PM PST
The WIT luncheon is underway! We start off with watching an energetic video displaying various types of women and men who each state “I am a technical woman” or “I support technical women.”

Rushabh Mehta, Executive VP of Finance for PASS, opens the WIT luncheon with the statement “PASS Supports Technical Women.” He then hands the floor over to Wendy Pastrick (@wendy_dance) for moderation of today’s panel.

Today’s panel is:

12:10 PM PST
Kathi starts off with a great message: “I want to encourage women to enter technology, but I want to encourage boys too.” She started off as a pharmacist because “I was probably born 5 years too early” but later switched to technology. She makes less money but enjoys it more. Her daughter had to learn HTML for school and actually helped her get her first work in IT by showing her how to program. “I want both boys and girls to have the opportunity to discover lots of different things and to find what they love. And hopefully that’ll be technology.”

12:17 PM PST
The floor is then handed over to Jessica Moss, BI guru extraordinaire. She gives a great example of how influential a father can be in a young girl’s life: her father was the one who got her interested in technology and who encouraged her career. She says she never felt like she could *not* be technical because she was raised to believe she could do anything. She ends with a challenge for everyone at the Summit: talk to just one young woman and encourage her interest in technology.

12:23 PM PST
Cathi Rodgveller shares her background in education and how she started IGNITE (Inspiring Girls Now In Technology Evolution). The goal of IGNITE is to excite young women, minority races, and low-income youth, about technology and about technical careers. Rodgveller gives us a powerful message: “You can have an impact in your community. One [positive technology] event can change a young girl’s life.”

12:28 PM PST
Last, but certainly not least, Lynn Langit takes the floor. She starts off with a challenge to all audience members: tweet or text one person to say “I’m a technical women” or “I support technical women.” The room gets active while people are busy typing or texting, and Twitter is abuzz with various tweets and retweets. Lynn then takes the floor back and talks about her background and about her charity work. She mentions that every time someone buys one of her books, a donation is made to the MONA foundation. Langit also shares some of her experiences as a technical women: “I’m a developer evangelist. I’m often the only woman in the room, and I’M the one giving the presentation.”

12:35 PM PST
It’s now time for Q&A with the audience. I’ve also invited members of the Twitter community to send in their questions or comments, and we’ll do our best to get them answered. Following is a brief summary of the questions and answers provided:

Q: First up is a father of 2 teenage girls. He wants to know why WIT programs have continued to fall since 1985.
A: WIT is a low priority for schools. Schools have so many other priorities, and not enough time or funding, to address everything they need to. We need intervention from outside sources to stimulate change and ensure it’s being addressed. Rodgveller is working with her state Senator to try to enact change on the national level.

Moss mentions that studies show the top 2 issues for WIT are recruitment and retention. She also points out that middle school years are very formative and important for young women to foster their interest in technology. Rodgveller interjects that even high school is not “too late” to inspire young women.

Q: Another father asks, how can he remove stereotypes for children?
A: Parents are the best resources, period. Parents need to support their children at home and to let them know that stereotypes are negative and not okay. This includes not just gender issues, but also issues of race, religion, sexual orientation, etc.

Q: Today’s youth are concerned about the technical job market in US because of the prevalence of off-shores outsourcing. Is it still a good idea for young people to join today’s technical workforce, and how can we encourage them?
A: There are still plenty of opportunities in IT. In fact, one of the hottest trends today in technology is BI. The best way to ensure that your job is not outsourced is to stay relevant and keep up with the newest technology; those are not the jobs that are outsourced overseas.

Q: In the South, there are still lots of stereotypes. For instance, women frequently are not hired by companies for technical positions. Comments like “We can’t hire a woman for that job because it’s too valuable; what if she gets pregnant?” are still made. What can be done about this?
A: One of the executives from CA, the sponsor of the WIT luncheon, takes the stage to answer. He says in no uncertain terms that, at his company, those individuals making the disparaging remarks would be terminated. He says that the type of attitude described has to come from the top down, and the company is limiting itself by not hiring women. He ends with a message for employees to not tolerate discrimination and to go to HR whenever they see it happening.

Q: What can parents do to help WIT?
A: Parents are the greatest resource kids have. Parents set the attitude for their kids; if your attitude is positive, it will encourage your daughter to try new things and will open her up to opportunity whenever it presents itself. Also, parents need to raise the issue with schools, i.e. through PTA meetings, to make them realize that it’s important to you and it’s important for your children.

Moss: “But at the end of the day, it is the parent’s responsibility to expose your children to as much technology and as many experiences as possible.”

Kellenberger: “It is up to the parents to break stereotypes, for jobs, gender, race, etc. It’s the parent’s attitude that makes the difference.”

So that’s all I have for the Women In Technology luncheon. There was a lot of great content and some very positive messages from our panel. For more information on this topic, please check out the following resources:

Live Blogging: Keynote at PASS, Day 3

Today is the 3rd and final day of keynotes at the PASS Summit. Following is highlights of the keynotes. During the keynote, refresh often for updates!

8:36 AM PST
Keynote kicks off with Don’t Stop Believing by Journey. Awesome.

8:38 AM PST
PASS VP of Marketing Bill Graziano just takes the stage. He promises the shortest keynote of the conference, and appears to deliver on it. First up are Board announcements.

Outgoing Board Members are:

If you these folks at Summit, make sure to thank them for their hard work!

PASS President Wayne Snyder comes out to honor and thank Kevin Kline for his 10 YEARS of service.ย  Yes, that’s right, 10 YEARS. Wayne doesn’t get far into his speech before he gets choked up.ย  Great quote from Wayne:ย  “Kevin (@kekline) is a man of honor and integrity. He’s… well, he’s full of it.”ย  Kevin then gets a well-deserved standing ovation from the entire audience.

New Directors-at-Large are also announced:

Next year’s PASS Summit is also announced.ย  It will be in Seattle from November 8th – 11th, 2010. The decision was made to have the conference in Seattle because it’s a launch year, so access to Microsoft employees will be invaluable.ย  The registration rate is $995 if you register soon. Details and registration can be found on the PASS site at

8:52 AM PST
Dell keynote just started with Patrick Ortiz, Solution Architect with Dell’s Infrastructure Consulting Services for SQL Server & BI.ย  The keynote is pretty uneventful.

9:17 AM PST
Woot! Dr. David DeWitt, Technical Fellow, Data & Storage Platform Division at Microsoft, takes the stage. His presentation is entitled, “From 1 to 1000 MIPS.” He promises a very technical talk, against Microsoft Marketing’s wishes. He’s not going to be announcing any products, but instead plans to discuss the changes in database technology and what’s in store for us in the next 10 years.

Highlights (or at least, the ones that my simple mind was able to grasp):

  • Basic RDMS design is essentially unchanged, but the hardware has changed dramatically.
  • Interesting statistics in disk trends last 30 years: 10,000x capacity, 65x transfer rate, 10x avg seek time… not balanced at all
  • “CPU’s and disks are totally out-of-whack in terms of performance.”
  • The benefits of 1,000x improvement in CPU is almost negated by lack of improvement in disk
  • Transfer bandwidth/byte trends: 1980 = 0.015, in 2009 = 0.0001… 150x slower today! “It’s like trying to provide drinking water for the town through a garden hose.”
  • “Can incur up to one L2 data cache miss per row processed if row size is greater than size of cache line.”
  • DBMS transfers the ENTIRE ROW from disk to memory even though the query required just 3 attributes.
  • “Takeaway: DBMS must avoid doing random disk I/O as much as possible.”

In short, DeWitt shows us the power of indexing and vertical partitioning in very technical terms. He also gives us a taste for column-oriented design, which we’ll catch a glimpse of in SQL Server 2008 R2.ย  Awesomeness.

The keynote wraps up with a promise to include DeWitt’s presentation on the Summit DVD.ย  If you missed the conference, then trust me, DeWitt’s presentation is worth the cost alone; all of the sessions are just a nice bonus on top of that.ย  ๐Ÿ™‚

Live Blogging: Keynote at PASS, Day 2

So today is day 2 of the PASS Summit conference. The conference has been excellent so far, with tons of great content and tons of great folks. Yesterday’s keynote revealed some interesting tidbits, including SQL Server 2008 R2 setting the world record for TPC-E (transactions per second) and QphH (queries per hour). Be sure to refresh this page frequently for updates, or follow along on Twitter.

8:00 AM PST
Michelle shows up extra early today, having learned her lesson from yesterday. Coffee in hand. Ah, much better than yesterday. ๐Ÿ™‚

8:41 AM PST
Rushabh Mehta just took the stage to the tune of All Star.

8:46 AM PST
Interesting statistics:

  • 15% reduction in revenue
  • 40% growth in the SQL PASS community
  • 67% reduction in IT expenses

Rushabh encourages support in the PASS community. He challenges everyone to volunteer at least an hour a month with your local user group. If you would like to become more involved in the local East Iowa SQL Server user group, please let me know!

8:51 AM PST
Wayne Snyder just took the stage to present the PASSion Awards. Folks recognized include:

  • Tim Ford (@sqlagentman) – for his work on the Program Committee and Quiz Bowl. Tim was no where to be seen!ย  Probably eating left-over bagels.
  • Grant Fritchey (@GFritchey) – for his contributions to getting SQL Server Standard relaunched. Grant also had the opportunity to show off his fabulous kilt!
  • Jacob Sebastian (@jacobsebastian) – for his contributions to the Asia community. He has started 6 chapters in India. Wow!
  • Amy Lewis – for her contributions to the BI Virtual Chapters and growing the sub-chapters.

9:00 AM PST
2009 International PASSion Award recipient is Charlie Hanania. Accomplishments include Swiss Chapter Leader and managing the entire 2009 PASS European Conference.

2009 North American PASSion Award recipient is Allen Kinsel (@sqlinsaneo)! Allen’s accomplishments include 5 years with PASS, and Program Manager for the 2009 Program Committee. He helped tremendously with Summit 2009, so if you see him around Summit, be sure to thank him!

Read more about the PASSion Award winners here.

9:07 AM PST
Tom Casey just took the stage. His first challenge is for all Summit attendees to act as agents of change in their respective organizations. Interesting statistic: only 20% of business users have the data they need to do their job effectively. Tom’s call to arms is to improve that statistic.

9:13 AM PST
Tom Casey: “PASS Summit is the place to be for B-I.” Statistics from Summit 2009: 2 dedicated BI tracks with 50+ BI-related sessions. Last year, 20% of attendees expressed interest in BI; this year, that number is 31%.

9:16 AM PST
Ron VanZanten, Directing Officer of Business Intelligence for Premier Bank Card, just joined Tom on stage. He works with over 25 TB of data and has used BI to leverage the data that they collect. His 3,200 employees use this information to perform their job. VanZanten wanted a flexible BI stack that would grow with the company. He’s successfully tested Madison to improve performance & scale out his environment. This is very encouraging news for any large data warehouse environment.

9:23 AM PST
Tom recognizes that data is available in a wide variety of mediums and that the numbers of sources will only continue to increase. He also introduces a new term: “spreadmart” – spreadsheets that are used as data marts.

9:30 AM PST
Amir Netz just takes the stage to give everyone a demo of the self-service BI offerings in SQL Server 2008 R2. Features include PowerPivot and SharePoint services. Amir shows us an Excel spreadsheet PowerPivot table with 100m rows. It’s amazingly fast. While I shudder to think of end users actually asking for 100m rows in Excel (and trust me, I’m sure it’ll happen), this is going to be a great breakthrough for our BI admins and our power IW users. It’s a time saver and gives greater visibility into data. While I realize the self-service BI offerings are still young, I’m excited for the future of BI. I can’t wait to implement what I’ve seen in our reporting environments.

11:55 AM PST
Tom Casey returns to center stage. Tom announces a contest to win an X-Box 360. To enter, follow @powerpivot & retweet: “Want to learn more, go to & sign up for the November CTP #powerpivot.” Tom concludes the keynote with his call to arms to increase the number of business users who have access to the BI data that they need to do their job effectively (currently 20%).

So there you have it, today’s keynote. Check back tomorrow for details on Thursday’s keynote; it looks to be the best yet.