Savvy Web sites
are watching every move of their customers, tracking their page viewing
preferences and buying habits. This tracking is fed into sophisticated
correlation engines that can accurately predict those products and services
that the customer is likely to buy. Using this information, the Web page
content is customized. We are seeing the dawn of applied artificial
intelligence in eCommerce, and the major database vendors are creating
products to help.
The ability to
track the behavior of customers on an eCommerce site is critical, and Oracle
offers several tools and techniques to assist in this process. If Web sites
can create dynamic Web page content featuring those items that their customer
likes to buy, they can greatly improve the shopping experience for the
customer, and greatly increase sales.
properly, intelligent customer correlation analysis techniques are a win-win
technology. Customers are happy because the eCommerce site highlights those
products they want to buy. The eCommerce company is also happy because of
millions in extra revenue from "impulse" sales. Marketing experts estimate
that impulse sales account for billions of dollars in yearly consumption, and
the eCommerce vendor can dramatically improve revenue by using custom HTML
improperly, customer correlation analysis can lead to broken lives and
lawsuits. There is an urban legend that an Oracle data warehouse for a Hotel
chain analyzed patterns of hotel usage, and sent targeted mailing at customers
who rented rooms on weekdays. Sadly, the coupons were sent to the customer's
home address, and not their work address. Legend says that more than six wives
learned that their partners were renting rooms in local hotels during the
week, and several divorce proceedings were started because of this poor choice
of mailing address.
explores this exciting new technology and reveals the secrets mechanisms used
by Oracle9iAS Personalization Engine to build intelligent Web page content.
While the technology behind the scene is very sophisticated, the mechanisms
are well-known and have been employed in marketing systems for years.
is Watching You
While most Web
users profess a desire to remain anonymous, the eCommerce software must track
visitor's behavior and use their prior behavior to customize their Web page
content. Using a combination of cookies and referrer statistics, many
eCommerce sites can watch your every move and store your page viewing habits
inside an Oracle database.
Before we move
into the Oracle specifics, let's take a closer look at how page view tracking
works. Every time you make an HTTP request from your browser, you are sending
three bits of information:
1. Your IP
address — This is needed for the Webserver to relay the page back to your
browser. For most Web users, their IP address changes every time they connect
to their Internet Service Provider (ISP)
2. The URL you
want — This is the page that you seek to display.
3. The URL of
the last page you visited — This is a "referrer" that says the URL of the page
immediately preceding this request. If this URL contains "content strings,"
you can record a great deal of information about the viewing habits of the
URL that you are requesting gets resolved to an IP address, and these three
items are sent to the target IP address. The most important of these three
items is the referrer URL because it provides a history of Web page usage. In
their quest for Internet privacy, a host of products such as the
browsers from sending referrer URLs.
used by eCommerce sites for tracking your visits are borrowed from the
principles in the landmark management spy book Unobtrusive Measures by
Eugene Webb. The idea behind unobtrusive measures is simple: "Customers
will change their behavior when they know they are being watched."
It is well
documented that "anticipated evaluation" will change consumer behavior, and a
valid data collection mechanism requires an unobtrusive tool. In the marketing
industry, researchers have long been aware of the anticipated evaluation bias
and spend millions of dollars on Web systems that will unobtrusively monitor
The tools used
by eCommerce sites to monitoring your behavior are very unobtrusive and the
casual user is not aware that their movements are being tracked.
What Does an
eCommerce Site Track?
gather detailed visitor information from the referrer statistics that are
coded inside each HTTP request, and store them inside an Oracle database for
analysis. Because the Web site knows the last URL that you issued (the
referrer URL), you can gather this "referrer" information and see how people
got to you site. If your visitor used a search Engine such as Google to find
your Web site, the referrer URL may look like this:
looking at this URL, we can see that this visitor issued a Google search for
pages using the keywords "remote oracle support." Because keyword information
is included inside the URL, referrer statistics can be a gold mine for
tracking Web browsing behavior.
It is important
for eCommerce sites to add data inside URLs, and the more sophisticated the
tracking mechanism, the longer the URLs. For example, here is a URL to display
a book on Amazon.com:
From looking at
the URL above, it is clear that eCommerce sites are using the URL to help
capture detailed data about you behavior.
Webserver can capture this referrer URL chain in a database, then a whole
history of your activity on the Web site can be stored and analyzed. Standard
Webserver software (apache, asp, cgi) collects the referrer statistics for all
visitor information (refer to Listing 1).
Listing 1: A referrer report of Google keywords.
referrer statistics are critical to companies who rely on search engines to
bring them traffic. On the Web site in Listing 1, we see exactly the keywords
that are used by new visitors to find the Web site, and the frequency of the
tracking and storing details about your visit is only half the story, and to
be efficient, the Web site must have details about your demographics.
More About You
collection is a critical part of a successful formula for personalization
technology. Without demographics, we cannot develop multivariate correlations
that compare your viewing and purchasing behavior with known groups of
When you sign
on to an eCommerce site for the first time, you will most likely provide
enough personal information to uniquely identify yourself. The eCommerce
server takes your identity information and collects demographics about you
(refer to Figure 1). These demographics include your sex, age, income level,
credit history, and other publicly available information about your income,
social status, and personal interests.
Figure 1: Demographics collection mechanism for eCommerce.
this information is stored in a large Oracle database. Using this data,
complex multivariate (Chi-square) analysis is performed, and detailed
predictive models are created. These models are used to customize the Web
At this point,
the Web site knows details about your browsing behavior, but we next must have
a means to positively identify each visitor so we can spy on their browsing
behavior. Web developers for eCommerce systems have faced the challenge by
writing cookies into the PC of the customer. A cookie is a small flat-file
that serves to positively identify each visitor.
interesting side-note, these files were called cookies after the famous Cookie
Monster rouge software on the MIT student system. For more than a year, a
hidden program named cookie monster would randomly interrupt user session and
demand "Gimme Cookie." When the student typed "cookie," the software thanked
them and restored their computing session.
As we have
noted, we cannot use the IP address of the browser to verify the identity of a
customer. Most ISPs use dynamic IP addressing where by the fourth octet of the
IP is assigned at login time for example, an ISP may have the IP of
172.16.10.xxx, and xxx is a number assigned at ISP connect time. Without a
static IP to identify the customer eCommerce vendors must rely on cookies to
get a positive ID on an incoming customer request.
You can easily
see all of the cookies on your MS-Windows system (refer to Figure 2).
Figure 2: The location of your cookies directory on MS-Windows.
For fun, you
can look inside cookie file and get an idea about how cookies help the
eCommerce site spy on your behavior. Here is an example of an Amazon cookie
(refer to Listing 2).
Listing 2: A sample cookie file.
understand cookies, you can have fun with eCommerce sites either by removing
the cookie or editing their contents. Next time you use such a site, take a
minute to edit and change the cookie file before you go to the site.
Let's wee how
cookies influence customized Web content. Below, we access www.Amazon.com
without a Cookie file (refer to Figure 3).
Figure 3: Amazon home page without a cookie.
Once we replace
our cookie, we access Amazon and get a customized recommendation list (refer
to Figure 4). This list of recommendations is produced by a personalization
engine that has carefully analyzed your viewing behavior, compared your
behavior to other customers, correlated your demographics, and develops a
customized predictive model of those items that you have the highest
propensity to purchase.
Figure 4: Accessing Amazon with a Cookie.
At this point,
we should now have an understanding of how eCommerce sites collect information
about you and your behavior. Next, let's look at how they use the data to
recommend purchases for visitors.
Solution to Personalization
referrer data on a busy eCommerce site is a formidable computing challenge.
Companies such as Oracle have developed tools such as the
ORACLEiAS Personalization engine and
Oracle Data Mining Suite (formerly the Darwin product) to assist with this
complex process. The nature of this analysis is very resource intensive, and
almost all large eCommerce sites must devote large servers exclusively for
developing predictive recommendations.
of dollars each year in impulse purchases at stake, IT marketing professionals
know that it is critical of get the right products onto your custom page. To
be successful, the system must be able to accurately predict your propensity
to buy a product, based on two factors:
1. Prior buying
and browsing patterns
2. Buying patterns of like-minded customers (customer profiling)
in developing these models is accurately placing visitors into consumer
groups. A consumer group is a group of customers with similar
demographics, and hence, similar buying patterns. Some shops tag these groups
with "handles" such as yuppies, dinks (double income, no kids), and so on.
Figure 5 shows
the process of analyzing demographic information to place visitors into
consumer groups. A visitor can be placed into a consumer group in two ways:
1 - Their
demographic category (collected from personal information)
2 - Their pattern of page views (collected from referrer URLs)
Figure 5: Categorizing users into consumer groups.
Once we have
defined consumer groups, we next start a data mining procedure to correlate
the patterns of each consumer group to specific products (refer to Figure 6).
The customized HTML personalization is based on data from three sources:
consumer group data — These are predetermined summaries of consumer group
rankings of pages viewed — This is a measure of the popularity of product
pages according to each consumer group.
history data — This is historical sales data, correlated by consumer group.
Figure 6: Correlating consumer groups to product preferences.
After we have
the consumer groups ready, the Oracle database can quickly generate customized
HTML (refer to Figure 7). Here we see three steps:
Request — The customer browser makes an HTTP request to the Web listener.
2. Get the
Cookie — The Web server sends a request to the browser PC to get the
the custom page — The cookie file verifies the identity of the visitor,
and customized HTML is created by the app server based on the identity, and
the custom HTML is delivered back to the browser.
Figure 7: Delivering custom HTML pages.
inherent complexity of personalization technology, vendors are creating
interfaces to the software. The Oracle product even offers a GUI interface to
aid in the complex analysis of referrer statistics (refer to Figure 8).
Figure 8: The Oracle iAS Personalization GUI.
These tools use
sophisticated data mining technology to constantly peruse the database looking
for statistically-significant correlations. Often, these predictions are
obvious (i.e., long underwear does not sell well in Hawaii), but other
predictions are amazing. Some predictive models have been able to accurately
target new products to exactly the type of people who will want them. This is
the classic "need creation" model used by successful companies like Sony. For
example, Sony launched the first Walkman because they had empirical evidence
that customers would look at the product and say "I need that," even though
the customer had no prior knowledge that they needed the product.
technology surrounding personalization of Web content is already extremely
sophisticated, IT professionals are constantly making it even more complex.
Artificial Intelligence routines are now being employed to develop detailed
models about how consumers spend their disposable income.
Corporation is making headway into this market with their Data Mining Suite
and Personalization Engine software. These products are sophisticated programs
that interface with the Oracle database.
threes advances will come even more concerns about the loss of privacy on the
Internet, but for now, customers appear only too happy to provide the raw data
needed to analyze their spending patterns.