Resume

Jeremiah W.,

 

 

 

We only do consulting. 

No permanent job inquiries please.

 

Click here for Consulting rates.

 

 

BIOGRAPHY

 

 

     

 

Jeremiah was a founding member and a veteran of Amazon.com’s Database Systems team.  He was one of the original architects of Amazon.com’s IT infrastructure and a top level Amazon.com escalation point for Oracle problem resolution.  In addition to his extensive database administrator expertise, Jeremiah is a regular lecturer and author at industry and user group conferences.  Jeremiah has been honored by Oracle Corporation as one of the first five Oracle 9i Certified Masters.  On numerous occasions Jeremiah has been interviewed and featured in the Wall Street Journal,  Information Week, Oracle Magazine and Select Magazine.   His professional areas of interest include high availability theory, massive systems concurrency - scalability, rapid incident recovery and systems internals.   Throughout Jeremiah's many years of professional IT experience he has always been customer-focused.

 

 


Expert IT Skills

 

  • Product

     Oracle 6.0.22 – Oracle 10.1.0,
    SQL*Plus, SQL*Loader, Export/Import, RMAN, etc.
    Precise SQL
    BMC Patrol
    GNU tools
    HP ServiceGuard
     

  • Platforms

     Linux 2.1 – 2.4 and GNU tools on Intel
    HP-UX 64-bit 11.00 – 11.11 on HP 9000-series servers
    Compaq Tru64 Unix (Digital Unix/OSF) 4.0B – 4.0F on Compaq (DEC) Alpha
    Vax/VMS on DEC Vax model 11
     

  • Programming Languages

    Perl
    PL/SQL

 

Oracle Certifications

  • 2001     Oracle 9i Certified Master (Hon.) Oracle Corp
    .

  • 2000     Oracle Internals Training (Oracle Support CoE)
     

  • 1999     Hewlett-Packard ServiceGuard Cluster training, HP Education
     

  • 1996     SAS Certification, SAS Institute

 

Education

  • 1993     Bachelor of Arts (BA) International Studies / East Asia, Earlham College, Indian
                 (US News and World Report second-tier ranking)
     

  • 1992     Chinese Language Certificate, Nanjing University, China

 

Employment History
 

  • Sep. 2003 – Nov. 2003  
    Amazon.com              Seattle, WA          Database Administrator, Technical Lead
     

  • Sep. 2002 – Sep. 2003  
    Amazon.co.uk           Slough, UK             Database Administrator, Technical Lead
     

  • May 1997 –  Aug. 2002   
    Amazon.com            Seattle, WA            Database Administrator, Technical Lead
     

  • Jul. 1995 –  May 1997    
    CompuServe             Seattle, WA           Database Administrator, Programmer
     

  • Oct. 1994 – Jul. 1995    
    LaserDirect                Kent, WA               Oracle Programmer
     

  • Oct. 1993 – Oct. 1994 
    Earlham College       Richmond, IN          Oracle Programmer
     

  • Sep. 1992 – Aug. 1993  
    Earlham College           Richmond, IN          Cabling Specialist
     

  • Sep. 1991 – Aug. 1993  
    Earlham College       Richmond, IN          System Operator, Helpdesk Staff
     

  • Nov. 1987 – Aug. 1990
    MetroVideo               Wellesley, MA         Machine Vision Technician
     

Experience Details

Amazon.com

My greatest professional experience
Ninety percent of my IT knowledge comes from my involvement with Amazon.com. My interest, drive and skills in IT prior to my employment with Amazon.com were miniscule in comparison to today. I cannot overstate the intensity of my experiences with Amazon.com, including 72-hour straight crisis management sessions and many nights spent on the floor of my office, or worse, at the keyboard. I have scooped water into buckets in a flooded machine room, killed the ventilation system for our offices to prevent tear gas being sucked in during anti-WTO protests, and managed a critical media recovery during a building evacuation. Working for Amazon.com has been an intense and draining experience. Through my involvement with Amazon.com, I have been inspired to become a true industry activist, speaking at conferences, working on publications, and actively participating in mailing lists and newsgroups.

The first DBA
I was the first DBA hired at Amazon.com, during the company’s pre-IPO period. Prior to my employment, the company’s databases were managed by a handful of software developers. I established the Database Systems group. I played a key role in building the group, which now numbers 40 employees, and handles all design, modeling, systems engineering, database administration and systems administration for Amazon.com’s database systems. I have probably interviewed over 300 candidates since being hired. Among the key values I tried to impart to the group are:

Using intelligence over procedures

  • Hiring smart people rather than focusing on experience

  • Retaining group responsibility for and expertise in the whole system (programming standards and API, data model,   application design, schema, database operations, software, systems administration, hardware design, network, etc.).

  • Understanding system internals

  • Developing a close working relationship with the upper-levels of Oracle’s development and support organization (BDE/DDR, development leadership).

Scaling and availability

From the beginning, the clear challenge at Amazon.com has been achieving the dual goals of scalability and availability. These goals were made more challenging by three years exponential growth, a rapidly changing application set, and hundreds of Oracle software defects that were uncovered by our unusual load characteristics. During my tenure, the company has grown from:

  • One Amazon.com website to 50 websites, including borders.com, toysrus.com and target.com

  • Two Oracle instances to over 150 Oracle instances

  • 200 employees to thousands of employees

  • 200,000 customers to over 50 million customers

  • 50 orders per minute to thousands of orders per minute

  • One country to six countries

  • One product line to hundreds of product lines
     

As a result of these challenges, I developed skills in a variety of areas:

Standby database

  • I developed scripts that allowed planned failovers and failbacks to be achieved without reinstantiation of the standby. Also I developed techniques that reduced the overall failover time to less than two minutes.

  • I developed techniques for maintaining a standby that reduced the amount of DBA intervention necessary to maintain it. These techniques included:

Intelligent exception-handling log application scripts

  • Automatic datafile create capability

  • Intelligent monitoring of nologging activities on the primary and appropriate file refresh on a file-by-file basis.

  • Zero downtime instantiation using hot backup mode and network copy.

Clusters

  • Writing original cluster failover scripts

  • Fastest possible downtime reduction techniques

Shared-nothing architecture

Availability tiers/factoring

  • Architectures with no single point of failure

Software design for availability

  • Intelligent reconnect

  • Wait-based brownout detection

Replication (multimaster and other)

  • Tricking snapshot logs into not purging during secondary snapshot site builds

  • Understanding the replication catalog and API

Shared Server (MTS)

  • Best practices for exponential scaling with Shared Server

  • Multi-interface and many-listener configurations
     

Application caching strategies

  • Host (shared memory) vs. process (private memory) application caching strategies to reduce database load and access times.

Designing to “solve the problem” rather than to “use the product/feature”

  • Seeking the solution to a problem, rather than evaluating random “products”

  • Challenging gratuitous use of in-vogue languages and features

Wait and resource-based tuning strategy (v$session_wait, v$session_event, v$sesstat)

  • Overall philosophy that the most important measure of scaling and availability is how long it takes to provide the results.

  • Subsequent breakdown from the client-level, through the application, service and the database   (v$session_wait, v$session_event, v$system_event), of time spent making requests.

  • Focus placed on largest time chunks and how to reduce them through novel approaches.

  • Secondary focus on resource statistics and query optimization as tools to reduce time spent per unit work.
        CPU, memory and execution statistics as recorded by v$sesstat and v$sysstat.
        Query execution plan and optimizer tuning (histograms)

  • Timing using empirical sampling tools such as Precise SQL

  • Whole-session analysis using 10046 trace and parser such as tkprof or homegrown.

Rapid problem solving and crisis management skills

  • Development of problem-solving bailout rules

  • Required diagnostic steps before bounce

  • Clear escalation and collaboration plan for major incident problem solving
    “Mission Critical” is my middle name

  • Truly enjoy the heat of troubleshooting major problems

Stability through the CBO

  • Cost-based optimizer can provide stability during data volume changes

  • CBO diagnostics using 10053 trace

Traditional and novel recovery strategies

  • Single-table point-in-time recoveries

  • Logminer backout of user “oopses”

  • Faster recovery through recovery tuning

Resource and time-saving backup strategies

  • Backup schedule based on rate of change of tablespace

  • Third-mirror/clone partition splits

  • RMAN, Veritas Netbackup

Management of issues within Oracle Support

  Understanding of what to collect for Oracle support

  • Systemstate/hanganalyze dumps for resource pileups that cannot be untangled via the dynamic performance views

  • Errorstack(debugger backtrace)/strace/truss/tusc of processes that are spinning but not doing work

  • Errorstack on error number of sessions encountering error but no evident reason why

  • Numerous other dumps as listed in oradebug dumplist.

  Extensive experience escalating and managing issues within Oracle Support

  High working efficiency


  Maximized productivity through use of time-saving Unix shell features in bash and zsh

  • Command history

  • Tab completion

  • Keyboard shortcut command-line navigation

 Platform agnostic


 Happy to use MS or Unix platforms interchangeably or interoperatively
 

  Other skill areas developed at Amazon.com

  • Basic system administration and networking skills in Compaq Tru64 (OSF), HP-UX, Linux and Solaris

  • Logical Volume managers and “raw” datafiles (Compaq/Digital LSM, HP-UX LVM, Veritas Volume Manager)

  • Asynchronous I/O

  • Basic Oracle Financial Applications experience (patches, concurrent manager)

  • Oracle Networking (multiple listeners, multiple interfaces, failover, round-robin)

  • Table partitioning


    Previous Experience Details
    CompuServe
    I started out with SPRY, producers of “Internet in a Box.” The company was eventually bought by CompuServe, and re-designated the CompuServe Internet Division. I was part of a group of six who built custom corporate web applications and scalable websites for the earliest web presences of Oracle Magazine, United Stationers and others. We created end-to-end solutions, including database, C, C++ and Perl CGI, server software, hardware and hosting. The greatest skill I gained at CompuServe was how to program in Perl and perform basic database administration tasks (install, configure, start, stop, backups, basic tuning and optimization). I was the only DBA in the group.

    LaserDirect
    LaserDirect was a “Direct Marketing” firm. They used Oracle to generate massive promotional mailings. In essence, I was responsible for a variety of projects executing the requirements of marketing campaigns, extracting names and addresses, and generating large numbers of mail pieces using large service bureau printers. At LaserDirect, I was introduced to Oracle v7.x, and honed my skill in query tuning, PL/SQL and SQL*Plus formatting features.

    Earlham College
    My alma mater was the first environment in which I worked as an Oracle professional. After graduating, I was invited to apply for a position as an Oracle programmer. I jumped at the opportunity, and spent over a year working on the college’s administrative computing systems, which are Oracle-based. I gained excellent mentoring from mature professionals in the optimizer, SQL, SQL*Forms, SQR and application design. Before graduation, I held a number of positions within the Computing Services department as a student employee, including networking specialist and helpdesk staff.