Marketing Deal Offers

  • Subscribe to our RSS feed.
  • Twitter
  • StumbleUpon
  • Reddit
  • Facebook
  • Digg

Saturday, 12 July 2008

Sybase IQ: A Different Kind of Columnar Database (Or Is It The Other Way Around?)

Posted on 12:18 by Unknown


I spent a fair amount of time this past week getting ready for my part in the July 10 DM Radio Webcast on columnar databases. Much of this was spent updating my information on SybaseIQ, whose CTO Irfan Khan was a co-panelist.

Sybase was particularly eager to educate me because I apparently ruffled a few feathers when my July DM Review column described SybaseIQ as a “variation on a columnar database” and listed it separately from other columnar systems. Since IQ has been around for much longer than the other columnar systems and has a vastly larger installed base—over 1,300 customers, as they reminded me several times—the Sybase position seems to be that they should be considered the standard, and everyone else as the variation. (Not that they put it that way.) I can certainly see why it would be frustrating to be set apart from other columnar systems at exactly the moment when columnar technology is finally at the center of attention.

The irony is that I’ve long been fond of SybaseIQ, precisely because I felt its unconventional approach offered advantages that few people recognized. I also feel good about IQ because I wrote about its technology back in 1994, before Sybase purchased it from Expressway Technologies—as I reminded Sybase several times.

In truth, though, that original article was part of the problem. Expressway was an indexing system that used a very clever, and patented, variation on bitmap indexes that allowed calculations within the index itself. Although that technology is still an important feature within SybaseIQ, it now supplements a true column-based data store. Thus, while Expressway was not a columnar database, SybaseIQ is.

I was aware that Sybase had extended Expressway substantially, which is why my DM Review article did refer to them as a type of columnar database. So there was no error in what I wrote. But I’ll admit that until this week’s briefings I didn’t realize just how far SybaseIQ has moved from its bitmapped roots. It now uses seven or nine types of indexes (depending on which document you read), including traditional b-tree indexes and word indexes. Many of its other indexes do use some form of bitmaps, often in conjunction with tokenization (i.e., replacing an actual value with a key that points to a look-up table of actual values. Tokenization saves space when the same value occurs repeatedly, because the key is much smaller than the value itself. Think how much smaller a database is if it stores “MA” instead of “Massachusetts” in its addresses. )

Of course, tokenization is really a data compression technique, so I have a hard time considering a column of tokenized data to be an index. To me, an index is an access mechanism, not the data itself, regardless of how well it’s compressed. Sybase serenely glides over the distinction with the Zen-like aphorism that “the index is the column” (or maybe it was the other way around). I’m not sure I agree, but the point doesn’t seem worth debating

Yet, semantics aside, SybaseIQ’s heavy reliance on “indexes” is a major difference between it and the raft of other systems currently gaining attention as columnar databases: Vertica, ParAccel, Exasol and Calpont among them. These systems do rely heavily on compression of their data columns, but don’t describe (or, presumably, use) these as indexes. In particular, so far as I know, they don’t build different kinds of indexes on the same column, which IQ treats as a main selling point. Some of the other systems store several versions of the same column in different sort sequences, but that’s quite different.

The other very clear distinction between IQ and the other columnar systems is that IQ uses Symmetrical Multi-Processing (SMP) servers to process queries against a unified data store, while the others rely on shared nothing or Massively Multi-Processor (MMP) servers. This reflects a fundamentally different approach to scalability. Sybase scales by having different servers execute different queries simultaneously, relying on its indexes to minimize the amount of data that must be read from the disk. The MPP-based systems scale by partitioning the data so that many servers can work in parallel to scan it quickly. (Naturally, the MPP systems do more than a brute-force column scan; for example, those sorted columns can substantially reduce read volumes.)

It’s possible that understanding these differences would allow someone to judge which type of columnar system works better for a particular application. But I am not that someone. Sybase makes a plausible case that its approach is inherently better for a wider range of ad hoc queries, because it doesn’t depend on how the data is partitioned or sorted. However, I haven’t heard the other vendors’ side of that argument. In any event, actual performance will depend on how the architecture has been implemented. So even a theoretically superior approach will not necessarily deliver better results in real life. Until the industry has a great deal more experience with the MPP systems in particular, the only way to know which database is better for a particular application will be to test them.

The SMP/MPP distinction does raise a question about SybaseIQ’s uniqueness. My original DM Review article actually listed two classes of columnar systems: SMP-based and MPP-based. Other SMP-based systems include Alterian, SmartFocus, Infobright, 1010Data and open-source LucidDB. (The LucidDB site contains some good technical explanations of columnar techniques, incidentally.)

I chose not to list SybaseIQ in the SMP category because I thought its reliance on bitmap techniques makes it significantly different from the others, and in particular because I believed it made IQ substantially more scalable. I’m not so sure about the bitmap part anymore, now that realize SybaseIQ makes less use of bitmaps than I thought, and have found that some of the other vendors use them too. On the other hand, IQ’s proven scalability is still much greater than any of these other systems—Sybase cites installations over 100 TB, while none of the others (possibly excepting Infobright) has an installation over 10 TB.

So where does all this leave us? Regarding SybaseIQ, not so far from where we started: I still say it’s an excellent columnar database that is significantly different from the (MPP-based) columnar databases that are the focus of recent attention. But, to me, the really important word in the preceding sentence is “excellent”, not “columnar”. The point of the original DM Review article was that there are many kinds of analytical databases available, and you should consider them all when assessing which might fit your needs. It would be plain silly to finally look for alternatives to conventional relational databases and immediately restrict yourself to just one other approach.

Email ThisBlogThis!Share to XShare to Facebook
Posted in analytical database, column data store, column-oriented database, columnar database, database technology, sybase iq, vertica | No comments
Newer Post Older Post Home

0 comments:

Post a Comment

Subscribe to: Post Comments (Atom)

Popular Posts

  • eBay Offers $2.4 Billion for GSI Commerce: More Support for Marketing Automation
    eBay ’s $2.4 billion offer for e-commerce services giant GSI Commerce has been described largely in terms of helping eBay to compete with ...
  • Selligent Brings a New B2C Marketing Automation Option to the U.S.
    I’m writing this post on my old DOS-based WordPerfect software, to get in the proper mood for discussing business-to-consumer marketing auto...
  • Infer Keeps It Simple: B2B Lead Scores and Nothing Else
    I’ve nearly finished gathering information from vendors for my new study on Customer Data Platform systems and have started to look for patt...
  • 4 Marketing Tech Trends To Watch in 2014
    I'm not a big fan of year-end summaries and forecasts, mostly because I produce summaries and forecasts all year round.  But I pulled to...
  • NICE Buys Causata to Extend Its Customer Experience Management Position
    So, there I was around 7:30 Eastern time this morning, sending out reminder notices to vendors I need to interview for an upcoming report on...
  • thinkAnalytics Helps Marketers Optimize Customer Treatments
    Summary: thinkAnalytics provides a robust decision engine to help make optimal recommendations across channels. Too bad more people don...
  • Vocus Marketing Suite: Still Mostly Social But Marketing Automation is On the Way
    If you’ve heard of Vocus at all, it’s probably as vendor serving public relations professionals. Its core offerings include a huge databas...
  • So Many Measures, So Little Time
    I’ve been collating lists of marketing performance metrics from different sources, which is exactly as much fun as it sounds. One result th...
  • Marketo Raises Another $50 Million: Where Does the Money Go?
    Marketo this morning announced a new $50 million funding round, almost exactly one year to the day after raising $25 million in November 2...
  • Advanced Analytics and Still More Reasons I Love QlikView
    I’m at the National Center for Database Marketing Conference this week. NCDM is always a good place to get a feel for what’s on people’s mi...

Categories

  • [x+1]
  • 1010Data
  • 2009 trends
  • 2010 predictions
  • 2011 predictions
  • 2013 marketing automation revenues
  • 2014 predictions
  • account data in marketing systems
  • acquisitions
  • acquistions
  • act-on software
  • active conversion
  • activeconversion
  • acxiom
  • ad agencies
  • ad servers
  • adam needles
  • adobe
  • adometry
  • advertising effectiveness
  • advocate management
  • affiliate marketing
  • agilone
  • aida model
  • aimatch
  • algorithmic attribution
  • alterian
  • analysis systems
  • analytical database
  • analytical databases
  • analytical systems
  • analytics tools
  • app exchange
  • app marketplace
  • application design
  • aprimo
  • are
  • artificial intelligence
  • ascend2
  • asset management
  • assetlink
  • atg
  • attribution analysis
  • attribution models
  • automated decisions
  • automated dialog
  • automated modeling
  • autonomy
  • b2b demand generation
  • b2b demand generation systems
  • b2b email marketing benchmarks
  • b2b lead scoring
  • b2b marketing
  • b2b marketing automation
  • b2b marketing automation industry consolidation
  • b2b marketing automation industry growth rate
  • b2b marketing automation revenues
  • b2b marketing automation systems
  • b2b marketing automation vendor rankings
  • b2b marketing data
  • b2b marketing industry consolidation
  • b2b marketing strategy
  • b2b marketing system comparison
  • b2c marketing automation
  • b2c marketing automation vendors
  • balanced scorecard
  • balihoo
  • barriers to marketing success
  • barry devlin
  • beanstalk data
  • behavior detection
  • behavior identification
  • behavior targeting
  • behavioral data
  • behavioral targeting
  • big data
  • birst
  • bislr
  • blogging software
  • brand experience
  • brand marketing
  • business intelligence
  • business intelligence software
  • business intelligence systems
  • business marketing
  • businses case
  • callidus
  • campaign flow
  • campaign management
  • campaign management software
  • causata
  • cdi
  • cdp
  • channel management
  • channel marketing
  • channel partner management
  • chordiant
  • cio priorities
  • clickdimensions
  • clicksquared
  • clientxclient
  • cloud computing
  • cmo surveys
  • cms
  • collaboration software
  • column data store
  • column-oriented database
  • columnar database
  • community management
  • compare marketing automation vendors
  • compiled data
  • complex event processing
  • consumer marketing
  • contact center systems
  • content aggregation
  • content distribution
  • content grazing
  • content management
  • content marketing
  • content matrix
  • content recommendations
  • content selections
  • content syndication
  • context automation
  • conversen
  • coremetrics
  • crm
  • crm integration
  • CRM lead scores
  • crm software
  • crm systems
  • crmevolution
  • cross-channel marketing
  • crowd sourcing
  • custom content
  • custom media
  • customer database
  • customer analysis
  • customer data
  • customer data integration
  • customer data management
  • customer data platform
  • customer data platforms
  • customer data quality
  • customer data warehouse
  • customer database
  • customer experience
  • customer experience management
  • customer experience matrix
  • customer information
  • customer management
  • customer management software
  • customer management systems
  • customer metrics
  • customer relationship management
  • customer satisfaction
  • customer success
  • customer support
  • cxc matrix
  • dashboards
  • data analysis
  • data cleaning
  • data cleansing
  • data enhancement
  • data integration
  • data loading
  • data mining
  • data mining and terrorism
  • data quality
  • data transformation tools
  • data visualization
  • data warehouse
  • database management
  • database marketing
  • database marketing systems
  • database technology
  • dataflux
  • datallegro
  • datamentors
  • david raab
  • david raab webinar
  • david raab whitepaper
  • day software
  • decision engiens
  • decision engines
  • decision management
  • decision science
  • dell
  • demand generation
  • demand generation implementation
  • demand generation industry
  • demand generation industry growth rate
  • demand generation industry size
  • demand generation industry trends
  • demand generation marketbright
  • demand generation marketing automation
  • demand generation software
  • demand generation software revenues
  • demand generation systems
  • demand generation vendors
  • demandforce
  • digiday
  • digital marketing
  • digital marketing systems
  • digital messaging
  • distributed marketing
  • dmp
  • dreamforce
  • dreamforce 2012
  • dynamic content
  • ease of use
  • ebay
  • eglue
  • eloqua
  • eloqua10
  • elqoua ipo
  • email
  • email marketing
  • email service providers
  • engagement engine
  • enteprise marketing management
  • enterprise decision management
  • enterprise marketing management
  • enterprise software
  • entiera
  • epiphany
  • ETL
  • eTrigue
  • event detection
  • event stream processing
  • event-based marketing
  • exacttarget
  • facebook
  • feature checklists
  • flow charts
  • fractional attribution
  • freemium
  • future of marketing automation
  • g2crowd
  • gainsight
  • Genius.com
  • genoo
  • geotargeting
  • gleanster
  • governance
  • grosocial
  • gsi commerce
  • high performance analytics
  • hiring consultants
  • hosted software
  • hosted systems
  • hubspot
  • ibm
  • impact of internet on selling
  • importance of sales execution
  • in-memory database
  • in-site search
  • inbound marketing
  • industry consolidation
  • industry growth rate
  • industry size
  • industry trends
  • influitive
  • infor
  • information cards
  • infusioncon 2013
  • infusionsoft
  • innovation
  • integrated customer management
  • integrated marketing management
  • integrated marketing management systems
  • integrated marketing systems
  • integrated systems
  • intent measurement
  • interaction advisor
  • interaction management
  • interestbase
  • interwoven
  • intuit
  • IP address lookup
  • jbara
  • jesubi
  • king fish media
  • kwanzoo
  • kxen
  • kynetx
  • large company marketing automation
  • last click attribution
  • lead capture
  • lead generation
  • lead management
  • lead management software
  • lead management systems
  • lead managment
  • lead ranking
  • lead scoring
  • lead scoring models
  • leadforce1
  • leadformix
  • leading marketing automation systems
  • leadlander
  • leadlife
  • leadmd
  • leftbrain dga
  • lifecycle analysis
  • lifecycle reporting
  • lifetime value
  • lifetime value model
  • local marketing automation
  • loopfuse
  • low cost marketing software
  • low-cost marketing software
  • loyalty systems
  • lyzasoft
  • makesbridge
  • manticore technology
  • mapreduce
  • market consolidation
  • market software
  • market2lead
  • marketbight
  • marketbright
  • marketgenius
  • marketing analysis
  • marketing analytics
  • marketing and sales integration
  • marketing automation
  • marketing automation adoption
  • marketing automation benefits
  • marketing automation consolidation
  • marketing automation cost
  • marketing automation deployment
  • marketing automation features
  • marketing automation industry
  • marketing automation industry growth rate
  • marketing automation industry trends
  • marketing automation market share
  • marketing automation market size
  • marketing automation maturity model
  • marketing automation net promoter score. marketing automation effectiveness
  • marketing automation pricing
  • marketing automation software
  • marketing automation software evaluation
  • marketing automation success factors
  • marketing automation system deployment
  • marketing automation system evaluation
  • marketing automation system features
  • marketing automation system selection
  • marketing automation system usage
  • marketing automation systems
  • marketing automation trends
  • marketing automation user satisfaction
  • marketing automation vendor financials
  • marketing automation vendor selection
  • marketing automation vendor strategies
  • marketing automion
  • marketing best practices
  • marketing cloud
  • marketing content
  • marketing data
  • marketing data management
  • marketing database
  • marketing database management
  • marketing education
  • marketing execution
  • marketing funnel
  • marketing integration
  • marketing lead stages
  • marketing management
  • marketing measurement
  • marketing mix models
  • marketing operating system
  • marketing operations
  • marketing optimization
  • marketing performance
  • marketing performance measurement
  • marketing platforms
  • marketing priorities
  • marketing process
  • marketing process optimization
  • marketing resource management
  • marketing return on investment
  • marketing ROI
  • marketing sales alignment
  • marketing service providers
  • marketing services
  • marketing services providers
  • marketing skills gap
  • marketing software
  • marketing software evaluation
  • marketing software industry trends
  • marketing software product reviews
  • marketing software selection
  • marketing software trends
  • marketing softwware
  • marketing suites
  • marketing system architecture
  • marketing system evaluation
  • marketing system ROI
  • marketing system selection
  • marketing systems
  • marketing technology
  • marketing tests
  • marketing tips
  • marketing to sales alignment
  • marketing training
  • marketing trends
  • marketing-sales integration
  • marketingpilot
  • marketo
  • marketo funding
  • marketo ipo
  • master data management
  • matching
  • maturity model
  • meaning based marketing
  • media mix models
  • message customization
  • metrics
  • micro-business marketing software
  • microsoft
  • microsoft dynamics crm
  • mid-tier marketing systems
  • mindmatrix
  • mintigo
  • mma
  • mobile marketing
  • mpm toolkit
  • multi-channel marketing
  • multi-language marketing
  • multivariate testing
  • natural language processing
  • neolane
  • net promoter score
  • network link analysis
  • next best action
  • nice systems
  • nimble crm
  • number of clients
  • nurture programs
  • officeautopilot
  • omnichannel marketing
  • omniture
  • on-demand
  • on-demand business intelligence
  • on-demand software
  • on-premise software
  • online advertising
  • online advertising optimization
  • online analytics
  • online marketing
  • open source bi
  • open source software
  • optimization
  • optimove
  • oracle
  • paraccel
  • pardot
  • pardot acquisition
  • partner relationship management
  • pay per click
  • pay per response
  • pedowitz group
  • pegasystems
  • performable
  • performance marketing
  • personalization
  • pitney bowes
  • portrait software
  • predictive analytics
  • predictive lead scoring
  • predictive modeling
  • privacy
  • prospect database
  • prospecting
  • qliktech
  • qlikview
  • qlikview price
  • raab guide
  • raab report
  • raab survey
  • Raab VEST
  • Raab VEST report
  • raab webinar
  • reachedge
  • reachforce
  • real time decision management
  • real time interaction management
  • real-time decisions
  • real-time interaction management
  • realtime decisions
  • recommendation engines
  • relationship analysis
  • reporting software
  • request for proposal
  • reseller marketing automation
  • response attribution
  • revenue attribution
  • revenue generation
  • revenue performance management
  • rfm scores
  • rightnow
  • rightwave
  • roi reporting
  • role of experts
  • rule-based systems
  • saas software
  • saffron technology
  • sales automation
  • sales best practices
  • sales enablement
  • sales force automation
  • sales funnel
  • sales lead management association
  • sales leads
  • sales process
  • sales prospecting
  • salesforce acquires exacttarget
  • salesforce.com
  • salesgenius
  • sap
  • sas
  • score cards
  • search engine optimization
  • search engines
  • self-optimizing systems
  • selligent
  • semantic analysis
  • semantic analytics
  • sentiment analysis
  • service oriented architecture
  • setlogik
  • setlogik acquisition
  • silverpop
  • silverpop engage
  • silverpop engage b2b
  • simulation
  • sisense prismcubed
  • sitecore
  • small business marketing
  • small business software
  • smarter commerce
  • smartfocus
  • soa
  • social campaign management
  • social crm
  • social marketing
  • social marketing automation
  • social marketing management
  • social media
  • social media marketing
  • social media measurement
  • social media monitoring
  • social media roi
  • social network data
  • software as a service
  • software costs
  • software deployment
  • software evaluation
  • software satisfaction
  • software selection
  • software usability
  • software usability measurement
  • Spredfast
  • stage-based measurement
  • state-based systems
  • surveillance technology
  • sweet suite
  • swyft
  • sybase iq
  • system deployment
  • system design
  • system implementation
  • system requirements
  • system selection
  • tableau software
  • technology infrastructure
  • techrigy
  • Tenbase
  • teradata
  • test design
  • text analysis
  • training
  • treehouse international
  • trigger marketing
  • twitter
  • unica
  • universal behaviors
  • unstructured data
  • usability assessment
  • user interface
  • vendor comparison
  • vendor evaluation
  • vendor evaluation comparison
  • vendor rankings
  • vendor selection
  • vendor services
  • venntive
  • vertica
  • visualiq
  • vocus
  • vtrenz
  • web analytics
  • web contact management
  • Web content management
  • web data analysis
  • web marketing
  • web personalization
  • Web site design
  • whatsnexx
  • woopra
  • youcalc
  • zoho
  • zoomix

Blog Archive

  • ►  2013 (55)
    • ►  December (4)
    • ►  November (5)
    • ►  October (4)
    • ►  September (3)
    • ►  August (5)
    • ►  July (5)
    • ►  June (5)
    • ►  May (6)
    • ►  April (6)
    • ►  March (1)
    • ►  February (6)
    • ►  January (5)
  • ►  2012 (56)
    • ►  December (4)
    • ►  November (3)
    • ►  October (6)
    • ►  September (4)
    • ►  August (7)
    • ►  July (3)
    • ►  June (4)
    • ►  May (5)
    • ►  April (3)
    • ►  March (4)
    • ►  February (8)
    • ►  January (5)
  • ►  2011 (74)
    • ►  December (9)
    • ►  November (8)
    • ►  October (6)
    • ►  September (5)
    • ►  August (5)
    • ►  July (3)
    • ►  June (6)
    • ►  May (5)
    • ►  April (6)
    • ►  March (8)
    • ►  February (7)
    • ►  January (6)
  • ►  2010 (75)
    • ►  December (9)
    • ►  November (9)
    • ►  October (5)
    • ►  September (6)
    • ►  August (7)
    • ►  July (3)
    • ►  June (6)
    • ►  May (9)
    • ►  April (4)
    • ►  March (6)
    • ►  February (6)
    • ►  January (5)
  • ►  2009 (96)
    • ►  December (2)
    • ►  November (4)
    • ►  October (5)
    • ►  September (9)
    • ►  August (7)
    • ►  July (16)
    • ►  June (9)
    • ►  May (5)
    • ►  April (11)
    • ►  March (11)
    • ►  February (11)
    • ►  January (6)
  • ▼  2008 (59)
    • ►  December (6)
    • ►  November (3)
    • ►  October (8)
    • ►  September (1)
    • ►  August (5)
    • ▼  July (8)
      • How to Report on Ease of Use?
      • How Do You Classify Demand Generation Systems?
      • Two Acquisitions Extend SQL Server
      • Sybase IQ vs. Vertica: Comparisons are Misleading,...
      • QlikView 8.5 Does More, Costs Less
      • Sybase IQ: A Different Kind of Columnar Database (...
      • LucidEra Takes a Shot at On-Demand Analytics
      • The Value of Intra-Site Web Search: A Personal Exa...
    • ►  June (5)
    • ►  May (5)
    • ►  April (6)
    • ►  March (3)
    • ►  February (3)
    • ►  January (6)
  • ►  2007 (84)
    • ►  December (4)
    • ►  November (6)
    • ►  October (6)
    • ►  September (1)
    • ►  August (4)
    • ►  July (7)
    • ►  June (16)
    • ►  May (20)
    • ►  April (20)
Powered by Blogger.

About Me

Unknown
View my complete profile