by, Marc Staimer

June, 2022

Research Premise

It is a well-known fact that few if any DBA pros enjoy or simply welcome performing any database upgrade. It’s easy to understand why. The process has historically been tedious, time-consuming, and labor-intensive together with downtime and, often, weekend work. The database applications frequently break after the upgrade, requiring more time to troubleshoot the root cause and fix the problems. More applications translate into more problems to fix.

It’s no wonder why DBAs are reluctant to embrace a database upgrade, especially when everything is working before the upgrade. DBAs by their nature tend to be philosophically conservative. It can be summed up by the adage, “If it isn’t broken, why fix it?” Why indeed.

Many of the DBAs out there may be nodding their heads in agreement as they read this. However, times have radically changed. Over the past few years, unprecedented cyber-attacks have fundamentally altered the IT landscape. It is no longer okay to put off or ignore upgrading mission-critical systems. These systems are regularly targeted by cyber-thieves. Mission-critical systems contain data that is extremely valuable to enterprises. One of the absolute mandates of cyber security is to keep all software up to date.

Make no mistake, databases that are not updated are increasingly vulnerable, and the cyber crooks know this. Every single time software is updated or patched, a list of vulnerabilities closed are included. The Cyber-criminal organizations immediately adjust their code to go after those vulnerabilities because they know that far too many IT organizations either do not update or delay updating their software. This is known as vulnerability exploitation. According to Mandiant, 2021 saw much greater vulnerability exploitations than ever before—more than the previous three years combined.

Most pros remember the ransomware WannaCry attack that tore across the globe in 2017. It infected more than a quarter million machines in 150 countries. At the time it was the largest ransomware attack ever. It was also a vulnerability exploitation attack on Windows machines that had not been kept up to date with the latest patches. Many of them could not be updated because they were pirated licenses.

Since then, cyber-attacks have massively spiked. Last year Check Point Software Technologies saw a 40% increase in 2021 over 2020. An attack occurs on average every 11 seconds. Google too saw a rise in vulnerability attacks. In their 2021 ‘Cost of a Data Breach’ report IBM noted that the average cost of a data breach was $4.62M.

Ransomware continues to evolve and is becoming more dangerous. The latest variants no longer just encrypt the data and throw away the key. Now they copy out the sensitive data first to use as leverage to force organizations to pay the ransom by threatening to release or sell the data. Even when the ransom is paid, the cyber-thieves will often sell that data on the dark web. So just paying ransom is obviously not a very good plan.

Cyber-insurance is not an answer for failing to update mission-critical databases that become compromised. When an organization makes a claim, one of the first things cyber-insurance firms do is check whether the IT organization was following strong security protocols such as keeping mission-critical software up to date. If not, the claim and payment is likely to be rejected. That can’t be good for anyone’s career. Keep in mind that as the ransoms in ransomware keep climbing from thousands into millions of dollars, euros, yen and pounds—they are only a small part of the cost.

That’s only one part of the increasingly dangerous IT environment when failing to keep mission-critical databases updated. Worldwide regulatory agencies are another. Privacy laws and regulations have made protecting personally identifiable information (PII) essential to every IT organization. And these laws, rules, and regulations have seriously sharp teeth. The fines can be ridiculously high when IT organizations are breached and lose control of PII because of IT negligence such as not keeping mission-critical systems updated. This can be interpreted as an intentional disregard of compliance requirements with fines as high as 4% of worldwide revenues over the past 12 months. Not a trivial fine by any measure.

These are the most urgent reasons why it is career endangerment when putting off upgrading Oracle Database 12c (or earlier) to 19c. There are also quite a few positive reasons to upgrade from Oracle Database 12c to 19c, including new features, capabilities, and innovations. The combination has the potential to be a career enhancing move because of the many problems it solves.

This Wikibon research examines in more detail the pros and cons of upgrading from Oracle Database 12c to 19c and the production-proven methodologies that simplify, automate, and greatly reduce manual labor, time, complexity, and the cost of performing that upgrade.

Oracle Database Version 12c, Version 19c, and Support Background

There are two distinct types of Oracle Database releases. The first is called a “Long-Term” release. The second is called an “Innovation” release. They have different target markets and support matrixes.

Long-Term Oracle Database releases have the highest stability level and longest time frame for error correction support with up to 8 years of exceptional support. Long-Term releases are aimed at those customers requiring less frequent upgrades to newer releases. There are three support levels for Long-Term releases.

1. Premier Support
   • Complete support for 5 years from effective date of the purchase order.
   • It provides complete access to product updates, enhancements, secure patching, technical assistance, and support resources necessary to maintain the Oracle Database, performance optimize it, and effectively implement new features and functionality.
2. Extended Support
   • Nearly equivalent support for 3 years after the end of Premier Support.
   • The combination with Premier Support (8 years total) is what Oracle calls the “Error Correction Support period”.
3. Sustained Support – a.k.a. Knowledge Base Support
   • Becomes available after the end of Extended support.
   • Customers can manually download existing patches and documentation.
   • Customers cannot request new patches or certifications.

Innovation Oracle Database releases are delivered in between Long-Term releases. Innovation releases commonly include many database enhancements and new capabilities. These enhancements and capabilities will be included in the next Long-Term release,and some may be backported to the current Long-Term release. Innovation releases are aimed at those customers that require ongoing and consistent leading-edge technologies to rapidly develop or deploy new applications or augment existing applications.

1. Premier Support
   • Complete support for just 2 years from effective data of the purchase order.
   • But there is no Extended Support.
2. Sustained Support – a.k.a. Knowledge Base Support
   • Becomes available after the end of Premier support.
   • Customers can manually download existing patches and documentation.
   • Customers cannot request new patches or certifications.

These customers should be willing to upgrade their Oracle Databases every two years. This is why Innovation releases have a short Premier support timeline and no Extended support.

Oracle released Version 12c[1] back in 2014. It’s an Innovation release and was the first Oracle Database version specifically architected to work in a public cloud and on-premises. That’s why there is a “C” designation. Version 12c made it simple to migrate on-premises Oracle Database to Oracle Cloud Infrastructure (OCI) or other public clouds running 12c or greater.

That means only Sustained support is currently available on Oracle 12c Databases. There is one exception for Oracle 12c version 12.1.0.2 where Extended support expires at the end of July 2022. Support has essentially lapsed.

Version 19c[2] is the Long-Term support release for the 12c Oracle Database family. Aside from the latest security enhancements, Version 19c adds a lot of capability to the Oracle Database including Automatic Indexing based on machine learning (ML) that delivers automated enhanced system performance, lower cost, and much higher index efficiencies than can be provided by even the most experienced DBA; Query Quarantine that automatically quarantines queries, stopping them from running in a loop that overloads system resources; JSON support with simplified syntax for JSON functions enabling users to perform partial JSON updates while also providing SODA APIs for Node.js, C, Python, and Java; Hybrid Partitioned Tables empowers DBAs to easily manage tables both inside and outside the database including read-only datastores outside the database on-premises or in a public cloud; Blockchain Tables, and Active Data Guard that eliminates downtime and data loss while DBAs implement database repairs, upgrades, patching, and incidental updates, while a synchronized replica/standby is maintained.

These are just a few key capabilities gained in upgrading to Oracle Database 19c. There are literally hundreds of new Version 19c capabilities compared to what’s available in Version 12c. For more detail on all the features included in 19c, go to Oracle Database Features and Licensing. Interviews with current Oracle Database 19c customers reveals that Automatic Indexing alone more than justifies the upgrade due to the improved performance on the same hardware. And whereas support for Version 12c has essentially lapsed, Premier support for Version 19c is available through April 2024 with Extended support available through April 2027.

Pros and Cons of Upgrading from Oracle Database 12c to 19c

Any objective analysis requires an in-depth look at the pros and cons of upgrading from 12c to 19c.

Pros

Security

As noted earlier, the biggest advantage by far is security. Version 19c comes with the latest Oracle Database security and patches. All known cyber vulnerabilities are patched. And because both Premier and Extended support are still available, future vulnerabilities will also be patched.

That latest security includes functionality not available on 12c. Unique functionality such as:

• Immutable Block Chain Tables—implemented from 21c to 19c
• Ability to create a User-Defined Master Encryption Key
• Automatic support for Both SASL and Non-SASL Active Directory Connections
• Enhanced diagnosability of Oracle Database
• Passwords removed from Oracle Database accounts
• Secure cluster communication
• Signature-based security for LOB locators
• Unified auditing top-level statements
• Updated support for Micro Edition Suite (MES) for FIPS 140.2

That’s quite a bit of enhanced security. That’s just the enhanced security between Version 19c and 18c. There’s even more when comparing the available security features from Version 19c to 12.1 or 12.2. In the current rapidly evolving cyber-attack environment, there is never enough security.

Cyber-security professionals will readily admit that there is simply no silver bullet that will guarantee the prevention of malware, ransomware, and security breach attacks. It just doesn’t exist. Cyber-security objectives are to make it as difficult as possible for those attacks to succeed. It’s called layered defense. As it becomes more difficult for the cyber-criminals to successfully complete their attack, they are very likely to stop wasting their time and move on to easier targets.

This is a very important factor to cyber-insurance vendors. Before they take on a client and even more so before they pay out a claim, they examine how difficult it was for the cyber-criminals to carry out their attack. If the IT organization is not patching and not keeping the mission-critical software up to date, as previously stated, insurance vendors are prone to reject paying the claims. Like all insurance vendors, they make a profit when they pay out less money in claims than they collect in premiums. They will minimize their risk.

Version 19c makes it much more difficult for the cyber-crooks to compromise or breach the Oracle Database. Oracle is constantly improving Oracle Database security with every release. The company is constantly evaluating past and current security concerns with previous versions and making them more secure in the next version and patch update. Note that difficult does not mean impossible. The weakest security link has been and continues to be the human operators and users.

Support

Another advantage of upgrading to Version 19c is the Oracle lifetime support policy, which only follows this version of the 12.x.x.x family. Failing to upgrade from older versions such as Version 12c means support will lapse. Without that support, there will be limited-to-no Oracle assistance for issues that are likely to arise. Issues such as Version 12c potentially developing frustrating problems when the underlying OS is upgraded.

Functionality That Simplifies and Automates, While Reducing DBA Time, Labor-intensive Tasks, and Cost

That old adage of “If it isn’t broken, don’t fix it,” is based on a false premise when it comes to database processes. The premise is that those database processes aren’t broken. Far too many of those processes are very labor-intensive and time-consuming. Version 19c is specifically architected to minimize and automate many of those labor-intensive DBA tasks. Automatic Indexing alone will save hundreds to thousands of DBA hours in developing, tuning, and modifying indexes—a thankless task. It makes the databases run faster and more efficiently without changing the underlying system. Testing has noticeably shown that Automatic Indexing is able to deliver more capable indexes than the most experienced DBA. Oracle has demonstrated that in less than 24 hours, Automatic Indexing was able to deliver fewer and patently superior indexes that are more efficient and faster than those developed by many expert NetSuite DBAs over a period of 15 years.

Automatic Indexing is just one—albeit one of the most important ones—of an additional 108 functions/features/capabilities above and beyond the security enhancements that are simply not part of or available to Version 12c through all previous releases up until 19c. That number increases as the Version 12c release gets older (12.2.0.1, 12.1.0.2, and 12.1.0.1). One underappreciated advantage in moving to Oracle Database 19c, is that customers gain the ability to easily migrate to the public cloud and the unique Oracle Autonomous Database.

Those 108+ functions/features/capabilities translate into significant savings in DBA tasks, time, and cost. Greater DBA productivity turns into faster time-to-market on projects, programs, services, and products. That in turn becomes unexpected and results in previously unobtainable increased revenues and lower costs.

The fact of the matter is that there is always going to be a need to increase DBA productivity. This is currently being exacerbated by the rapidly dwindling supply of experienced, knowledgeable, and skillful DBAs as baby boomers and the Jones[3] generation retire. Businesses are not standing still. There are significantly increasing requirements for AI, machine learning, block-chain, and analytics in general, all calling for DBA resources.

Upgrading from Version 12c to 19c Pros Summary

The advantages of upgrading to Version 19c are significant. Version 19c boosts security and provides exceptional ongoing support while adding key functionality that improves security and automates functions that reduces the demands on DBAs by orders of magnitude.

Each of these advantages on their own are more than enough to justify upgrading. Taken together, they’re compelling.

Yeah but, what about the cons? And there are definitely some cons.

Cons

Poor Application Documentation

The bane of every developer is documentation. Their code needs to be documented within application code and written down in a technical document. It is not fun, it takes time, and slows development. Few developers enjoy the process and not all do it well. Many do the bare minimum. Some even less than that.

Therefore, upgrading the application’s database can cause undocumented use of database parameters to break. When there’s no documentation or nominal documentation, troubleshooting on the part of DBAs becomes extensive. It becomes worse for multiple applications. Attempting to figure it out before the database upgrades take place is frequently a very large time sink not worth the effort.

Database Upgrades and Application Testing Takes a Lot of Time and Effort

There are three (3) production-proven methodologies to upgrade from Oracle Database 12c to 19c as documented in the Oracle Whitepaper “Upgrade and Migrate to Oracle Database 19c”.

1. Database Upgrade, using the command-line upgrade with dbupgrade.
2. Transportable tablespaces (TTS) export and import, using the Oracle Database feature full transportable export/import, or the traditional TTS mode.
3. Oracle Data Pump Export/Import, using either dump files or network mode.

Each is a tried and true playbook to effectively upgrade from Oracle Database 12c to 19c. Unfortunately, they all require time, effort, and expertise.

Upgrading from Version 12c to 19c Cons Summary

There are a couple of “Cons” to upgrading from Version 12c to 19c. They revolve around the perception of database applications breaking after the upgrade. Followed by the effort, skill, and time spent in troubleshooting those applications to fix the alleged breakage.

Oracle Database 19c’s Unique Fixes to Upgrade Hesitancy

AutoUpgrade

Oracle Database 19c comes with a new automated methodology called the AutoUpgrade tool. It’s a small, very powerful command line tool that enables DBAs to upgrade auto-magically one, several, or many databases quickly, easily, and repeatably in a controllable manner. It has become the only Oracle Database methodology currently recommended by Oracle. This is because the AutoUpgrade tool eliminates many of the time-consuming manual upgrade tasks and human errors typical of database upgrades. It performs the manual work of running pre-checks against multiple databases. It sets a restore point for those Murphy’s law occasions when something goes wrong. Then the AutoUpgrade tool upgrades one or multiple Oracle Databases concurrently. After the upgrade, it does the post-upgrade, recompilation, and time zone adjustment. DBAs only need to input a config file in text format for each database being upgraded.

No muss, no fuss, nominal effort, very little DBA time consumed. Oracle has an exceptional AutoUpgrade “how-to” video tutorial on YouTube that goes into detail with a simple step-by-step process. It’s thorough and quite easy to follow even for the novice DBA.

Yeah but…what about application testing? That annoying consequence of a database upgrade. There appears to be an effective answer for that too.

Real Application Testing

Oracle has a very easy-to-use and low cost Real Application Testing option for Oracle Database upgrades, consolidation, or changes. Real Application Testing eliminates much of the manual efforts previously required to test applications. It provides highly accurate simulated application load testing of the upgraded Oracle Database. It does this by capturing the application production workloads. Then it uses those workloads to assess and determine how the upgraded Oracle Database affects them. This minimizes unforeseen application instabilities or problems before production deployment.

Oracle Real Application Testing ensures application integrity, data management, and privacy for sensitive data when testing. It does all of this with the following essential functions:

• SQL Performance Analyzer (SPA)
• Database Replay
• Concurrent Database Replay
• SPA Quick Check
• Database Consolidation Workbench
• Test data management masking and data subsetting to prevent sensitive data exposure to non-production users

Oracle Real Application Testing is intuitive to use, inexpensive, helps avoid performance problems with closed-loop automated tuning, and is a highly proven proactive methodology that diminishes unforeseen upgrade induced application issues. It radically reduces DBA’s troubleshooting, effort, and wasted time in Oracle Database upgrades and more time for strategic productivity.

Summary and Conclusion on Upgrading from Oracle Database 12c to 19c

Oracle Database customers using Version 12c have only two choices.

1. Continue to run Version 12c. However, the severe consequences for that decision include:
   • Reduced security—just as cyber-attacks have grown exponentially.
   • Lapsed or soon to be lapsed support—customers are essentially on their own when there are problems and/or patches, leading to more security issues.
   • No ability to take advantage of Version 19c automation capabilities that significantly reduce a DBA’s tasks, time, and effort.
2. Or upgrade to Version 19c that provides:
   • Dramatically enhanced security with functionality such as Block Chain Tables and more.
   • Oracle’s Long-Term support which translates into increased security and reduced problems for DBAs to run down and solve.
   • Functionality that automates many of the manually labor-intensive DBA tasks.
   • The ability to migrate to Oracle Autonomous Database and to more easily move to the cloud.
   • All of this for the one-time effort of upgrading the databases and testing the applications. The release of Version 19c has slashed huge amounts of DBA manual upgrade efforts using automation with Oracle’s AutoUpgrade tool. It’s further slashed with Oracle Real Application Testing.

Anyone making a critical reasoning analysis of the pros and cons for each choice will conclude that upgrading from Oracle Database 12c to 19c is the only rational choice. This is especially true if they want to avoid the wrong kind of CEM—Career Ending Move—and, instead, make it a Career Enhancing Move.

[1] Version 12c encompasses Versions 12.1.0.1, 12.1.0.2, 12.2.0.1, 12.2.0.2 a.k.a. 18c

[2] Version 19c is a.k.a. 12.2.0.3

[3] The Jones generation is a new category for those born between 1955 and 1965. Baby boomers are the generation born between 1945 and 1955.