The 5 major benefits of ECPUs in the Oracle Autonomous Database

Oracle Autonomous Database (ADB) recently introduced a more advanced ECPU billing metric and is now retiring the legacy OCPU billing metric for Autonomous Data Warehouse and Autonomous Transaction Processing in the next 12 months. Oracle recommend switching from the OCPU billing metric to the ECPU billing metric, which will not incur any downtime or service interruptions.

ECPUs will provide the same great price-performance as OCPUs and continuous improvements in price-performance over time. Updating to the ECPU billing metric provides the following benefits:

• 50% lower entry cost: The smallest Autonomous Database that can be provisioned with ECPUs is 50% less expensive ($0.672 per hour vs $1.3441 per hour with OCPUs)
• Finer granularity for database scaling: Each incremental increase in ECPU database size is only $0.336
• Lower storage costs: Autonomous Data Warehouse storage price reduced from $118.40 to $25.00 per TB per month and Autonomous Transaction Processing storage can be provisioned in increments of 1GB (this is a huge thing!), with a minimum of 20GB – this brings the ADW in-database storage price on par with the object storage and thus this helps to build data lakes solely on the architectural requirements and not focusing on cost
• Up to 87% lower costs with database consolidation: Elastic Resource Pools, available on ECPU ADB Serverless databases, help consolidate deployments leading to major cost savings
• New features for Autonomous Database may only be available with ECPU’s

Note that the prices mentioned above are the current list prices for Autonomous Databases with the License Included license type.

There are also differences in backups between OCPU’s and ECPU’s. ECPU’s backup storage is billed separately, and the backup retention period may be selected between 1 and 60 days. With OCPU’s, 60-days of backup storage is included in the storage price. This new ECPU customer-controllable backup is beneficial because customers can now control the backup storage size and further reduce the cost of dev/test environments. Here is how I reduced the size from 60 to 31 days (later on I did reduce it to 7 days).

I did scale down my database in 2 phases: (1) I switched to the ECPU model (1 OCPU –> 4 ECPUs) and then (2) reduced the ECPU count from 4 to 2 and the storage from 1024GB to 20GB (those two in one go with no downtime).

Here are some general guidelines related to the new ECPU metric:

1. Provision all new Autonomous Data Warehouse and Autonomous Transaction Processing databases or clones with the ECPU billing metric
2. Update all existing databases to the ECPU billing metric, which is a simple and seamless button click or API call
3. Note that if you choose not to update your existing databases’ billing metric at this time, Oracle may convert your databases from the OCPU billing metric to the ECPU billing metric in the future

Updating your Autonomous Database Serverless to the ECPU billing metric will have no impact to your service and incur no downtime. Oracle Autonomous Database will be retiring the OCPU-based SKUs and replacing them with the ECPU-based SKUs. Starting in August 2023, some new Autonomous Database features may be available only on ECPU’s. For example, Elastic Resource Pools are only available with ECPU’s.

Note that ECPU’s have also already been introduced for MySQL Heatwave on AWS, and other services may also offer ECPU’s in the future.

Source: juliandontcheff.wordpress.com

Continue reading

Oracle Database 23c: A Comprehensive Guide

Introduction to Oracle Database 23c

Oracle Database 23c, also known as Oracle Database 23c, is the latest iteration of Oracle's powerful, enterprise-grade database management system. This version introduces a multitude of features and enhancements that aim to revolutionize data management, streamline processes, and enhance performance and scalability. As organizations increasingly rely on data to drive decision-making, Oracle Database 23c stands out as a critical tool for managing and leveraging large volumes of data efficiently.

Key Features of Oracle Database 23c

1. Advanced AI and Machine Learning Integration

Oracle Database 23c introduces advanced AI and machine learning (ML) capabilities directly within the database. This integration allows for more intelligent data processing and analytics. With these features, users can build, train, and deploy machine learning models using SQL, making it easier for data scientists and developers to create sophisticated applications without needing external tools.

2. Enhanced Multitenancy

The concept of multitenancy in Oracle Database 23c has been significantly improved. This feature allows multiple pluggable databases (PDBs) to run within a single container database (CDB). The new enhancements provide better resource allocation, isolation, and management of these PDBs, enabling more efficient use of hardware and reducing operational costs.

3. Autonomous Database Enhancements

Oracle's Autonomous Database capabilities have been further refined in 23c. These enhancements include improved self-driving, self-securing, and self-repairing functionalities. The database can automatically handle routine tasks such as tuning, backups, and security updates, freeing up valuable time for database administrators (DBAs) and ensuring optimal performance and security.

4. Blockchain Table Enhancements

Blockchain technology within Oracle Database 23c has seen notable improvements. Blockchain tables provide immutable and tamper-evident storage of data, ensuring data integrity and security. These enhancements include better performance, scalability, and support for more complex transactions, making it easier to implement blockchain solutions within the database.

5. In-Memory Database Advancements

Oracle Database 23c continues to push the boundaries of in-memory database technology. The advancements in this area include faster data processing, improved query performance, and more efficient use of memory resources. These improvements enable real-time analytics and faster decision-making processes.

Performance and Scalability Improvements

1. Optimized Query Performance

Oracle Database 23c introduces numerous optimizations to improve query performance. These include advanced indexing techniques, improved execution plans, and better use of hardware resources. These enhancements ensure that even the most complex queries are executed quickly and efficiently, providing faster access to critical data.

2. Improved Storage Management

Storage management in Oracle Database 23c has been enhanced to provide more efficient data storage and retrieval. This includes better compression algorithms, automated storage tiering, and improved support for large datasets. These features help reduce storage costs and ensure that data is always available when needed.

3. Scalable Architecture

The scalable architecture of Oracle Database 23c allows it to handle increasing workloads without compromising performance. This is achieved through better parallel processing, improved load balancing, and support for distributed databases. As a result, organizations can scale their database infrastructure to meet growing data demands seamlessly.

Security Enhancements

1. Advanced Encryption Techniques

Security is a top priority in Oracle Database 23c, with the introduction of advanced encryption techniques. These techniques ensure that data is protected both at rest and in transit, preventing unauthorized access and data breaches. The database also supports transparent data encryption, making it easier to implement and manage encryption policies.

2. Enhanced User Authentication

User authentication mechanisms have been strengthened in Oracle Database 23c. This includes multi-factor authentication (MFA), single sign-on (SSO) integration, and improved password policies. These features help ensure that only authorized users can access sensitive data and systems.

3. Comprehensive Auditing and Compliance

Oracle Database 23c offers comprehensive auditing and compliance features. This includes detailed logging of database activities, automated compliance checks, and support for regulatory requirements such as GDPR and CCPA. These features help organizations monitor and manage database security more effectively.

Simplified Database Management

1. Unified Management Console

The unified management console in Oracle Database 23c provides a centralized interface for managing all aspects of the database. This includes configuration, monitoring, and troubleshooting tools. The console simplifies database administration tasks, making it easier for DBAs to manage and optimize their database environments.

2. Automated Maintenance

Automated maintenance features in Oracle Database 23c help reduce the workload on DBAs. This includes automated backups, patching, and updates. These features ensure that the database is always up-to-date and running smoothly, without the need for manual intervention.

3. Integrated Development Environment

Oracle Database 23c includes an integrated development environment (IDE) that supports a wide range of programming languages and tools. This includes support for SQL, PL/SQL, Java, and more. The IDE provides a rich set of features for developing, testing, and deploying database applications, making it easier for developers to build robust and scalable solutions.

Conclusion

Oracle Database 23c represents a significant leap forward in database technology. With its advanced AI and ML integration, enhanced multitenancy, autonomous capabilities, and robust security features, it provides a powerful platform for managing and leveraging data. The performance and scalability improvements, combined with simplified management and enhanced development tools, make it an ideal choice for organizations looking to optimize their data infrastructure.

For those seeking to harness the full potential of their data, Oracle Database 23c offers unparalleled capabilities and a clear path to innovation and efficiency.

Continue reading

ORA-04030 out of process memory when trying to allocate - 3 Step Resolution

ORA-04030: out of process memory when trying to allocate bytes, occurs when an Oracle process runs out of operating system memory.

The error is caused by either:

• Exhausting total machine physical memory
• Exhausting designated space in the Program Global Area (PGA).

◉ Understanding ORA-04030

The ORA-04030: out of process memory when trying to allocate bytes, occurs when an Oracle process runs out of operating system memory. 

This can be caused by either:

◉ Exhausting total machine memory. i.e. there isn't enough physical RAM on the machine

Or

◉ Exhausting designated space in the Program Global Area, known as the PGA.

The error message will show how much memory the process tried to allocate and provide details of where the allocation failure happened.

ORA-04030 can occur in either a client or database process.

If raised by a database process then an entry will be made in the database alert log, which will point to a trace file containing more details, which can be useful to identify the cause.

Resolving an ORA-04030 error typically involves addressing the memory limitations that caused it.

This might be:

• Increasing available RAM
• Adjusting PGA size
• Optimizing resource usage
• Reviewing operating system limits
• Identifying and resolving memory leaks

◉ ORA-04030 Error Troubleshooting Steps

The basic Fix Flow steps for ORA-04030 are:

1. Use Autonomous Health Framework to generate an ORA-04030 Service Request Data Collection (SRDC)

In the first step we’re going to use Autonomous Health Framework to generate an ORA-04030 diagnostic collection.

The Autonomous Health Framework provides a series of capabilities aimed at making it easier for you to maintain your database.

It’s the same technology Oracle uses in the cloud to keep the Autonomous Database running, and it’s available for you at no extra cost as part of your existing software support agreement.

You can download Autonomous Health Framework by logging into My Oracle Support and searching – or you can just go to http://bit.ly/oracleahf, which will take you directly to the download page.

Autonomous Health Framework is capable of collecting various types of diagnostic collections, with problem specific SRDCs being one of them.

Before Autonomous Health Framework, Oracle Support might have asked you to go through many different steps as part of SRDC or Service Request Data Collection for ORA-04030. This would involve collecting and trimming different logs and running scripts, to understand how to perform it correctly required watching a video and reading other knowledge documents.

Now all this SRDC logic is packaged within Autonomous Health Framework and you can capture it all with a single command.

Login to the machine where the ORA-04030 occurred and as the Oracle user run the command:

tfactl diagcollect –srdc ORA-04030

You’ll be prompted to enter the date and time of the ORA-04030 you’re interested in, and then the database name

For example

$ tfactl diagcollect -srdc ORA-04030

Enter the time of the ORA-04030 [YYYY-MM-DD HH24:MI:SS,<RETURN>=ALL] : 

Enter the Database Name [Required for this SRDC] :

Components included in this collection: OS DATABASE CHMOS SOSREPORT

Preparing to execute support diagnostic scripts.

    Executing DB Script srdc_db_sid_memorysizes_11gplus.sql on CDB12 with timeout of 300 seconds...

Collecting data for all nodes

TFA is using system timezone for collection, All times shown in PDT.

Scanning files from 2024-03-25 10:07:36 PDT to 2024-03-25 10:40:07 PDT

Collection Id : 20240325104016mymachine03

Detailed Logging at : /opt/oracle.ahf/data/repository/srdc_ora4030_collection_Mon_Mar_25_10_40_19_PDT_2024_node_all/diagcollect_20240325104016_mymachine03.log

Waiting up to 120 seconds for collection to start

2024/03/25 10:40:24 PDT : NOTE : Any file or directory name containing the string .com will be renamed to replace .com with dotcom

2024/03/25 10:40:24 PDT : Collection Name : tfa_srdc_ora4030_Mon_Mar_25_10_40_18_PDT_2024.zip

2024/03/25 10:40:24 PDT : Collecting diagnostics from hosts : [mymachine04, mymachine03]

2024/03/25 10:40:25 PDT : Getting list of files satisfying time range [03/25/2024 10:07:36, 03/25/2024 10:40:07]

2024/03/25 10:40:25 PDT : Collecting Additional Diagnostic Information...

2024/03/25 10:40:44 PDT : Collecting ADR incident files...

2024/03/25 10:41:06 PDT : Executing TFA rdahcve with timeout of 600 seconds...

2024/03/25 10:41:08 PDT : Executing IPS Incident Package Collection(s)...

2024/03/25 10:41:10 PDT : Generating IPS Pack for 1 incidents on database cdb12

2024/03/25 10:41:17 PDT : Executing SQL Script db_feature_usage.sql on cdb12 with timeout of 600 seconds...

2024/03/25 10:41:17 PDT : Executing Collection for OS with timeout of 1800 seconds...

2024/03/25 10:41:25 PDT : Executing Collection for SOSREPORT with timeout of 1860 seconds...

2024/03/25 10:42:30 PDT : Completed Collection of Additional Diagnostic Information...

2024/03/25 10:42:35 PDT : Completed Local Collection

2024/03/25 10:42:35 PDT : Not Redacting this Collection on Exadata with no redaction option passed ..

2024/03/25 10:42:35 PDT : Not Redacting this Collection ...

2024/03/25 10:42:35 PDT : Remote Collection in Progress...

2024/03/25 10:42:56 PDT : Collection completed on host: mymachine04 

2024/03/25 10:42:56 PDT : Collection completed on host: mymachine03 

2024/03/25 10:42:56 PDT : Completed collection of zip files.

.---------------------------------------.

|           Collection Summary          |

+-------------+-----------+------+------+

| Host        | Status    | Size | Time |

+-------------+-----------+------+------+

| mymachine04 | Completed | 21MB | 113s |

| mymachine03 | Completed | 36MB | 131s |

'-------------+-----------+------+------'

Logs are being collected to: /opt/oracle.ahf/data/repository/srdc_ora4030_collection_Mon_Mar_25_10_40_19_PDT_2024_node_all

/opt/oracle.ahf/data/repository/srdc_ora4030_collection_Mon_Mar_25_10_40_19_PDT_2024_node_all/mymachine04.tfa_srdc_ora4030_Mon_Mar_25_10_40_18_PDT_2024.zip

/opt/oracle.ahf/data/repository/srdc_ora4030_collection_Mon_Mar_25_10_40_19_PDT_2024_node_all/mymachine03.tfa_srdc_ora4030_Mon_Mar_25_10_40_18_PDT_2024.zip

Once it's finished Autonomous Health Framework will package everything for you in a zip file for each machine, as you progress you'll only need the one from the node where the problem occurred.

2. Use ORA-04030 Troubleshooting Tool to find recommendations

Log into My Oracle Support and search for ORA-04030, or alternatively go to http://bit.ly/ORA-04030 to access it directly.

When you get to the troubleshooting tool click the Next button at the top right

Select the first radio button to choose to upload a Trace File Analyzer package. (Trace File Analyzer is one of the components of Autonomous Health Framework)

Then click the “Choose file” button, select the zip from the failing node that Autonomous Health Framework captured for you in step 1.

Then press the Upload button.

Once this is uploaded click the Next button at the top right again.

The troubleshooting tool will then analyze the contents of the diagnostic collection and compare the log entries against its list of known problems and recommend a solution.

If you go through the MOS troubleshooting tool and can’t find a solution, or you just need some more help, then Oracle Support is just a click away.

3. Log a new SR using the diagnostic collection

Press the Create SR button at the bottom.

You’ll then be prompted to clarify your:

• Product
• Product Version
• Support Identifier
• Operating System
• SR severity

Then press the Create SR button.

And you’ll get a new SR number.

Source: oracle.com

Continue reading

Oracle Globally Distributed Database addresses multi-country data residency challenges for Munich Re HealthTech

Customer Success story with Oracle Globally Distributed Database:

Oracle has helped the leading global specialist in digital solutions for the health insurance industry achieve scalability and data residency through Oracle Globally Distributed Database.

The insurance sector has been correlated with economic growth radically – Undoubtedly, the COVID-19 pandemic has resulted in a huge impact on insurers across the world. Digitalization is a critical driver in shaping politics, economics, and the balance of power in global events. The amplification of growth in technology is changing customer expectations, and in turn, prompting the incumbent and larger insurance companies to accelerate digitization and adopt new ways of working to offer seamless services to their customers.

While the industry heads forward with the changes happening in the ecosystem and shifting regulations with digital initiatives, a few more direct impacts that convert to lower profit margin and growth are compelling insurers to cut costs and perform more efficiently at the same time. Legacy IT infrastructure and the overly manual nature of certain processes and capabilities have challenged insurers’ ability to respond to such crises. This success story will take you through the nuances of the insurance vertical, and how Oracle Globally Distributed Database is helping enterprises in responding to these challenges.

Data Sovereignty in the Post-pandemic Insurance Industry

As insurers and regulators navigate through the digital revolution, the challenges of cyber risk pose obstacles in this thriving data-rich universe. Insurance companies have to collect massive amounts of data with the goal of risk mitigation, optimizing performance, and meeting the rocketing expectations of customers hence, data is key to remaining competitive. For ages, insurance companies have been aggregating a great deal of data, however, they face challenges, preventing them from making the most of the power offered by data governance frameworks and meeting the data sovereignty requirements such as:

◉ Maintaining control over encryption and access to your data

◉ Giving organization visibility and control over provider operations

Organizations across the globe are now prioritizing their data sovereignty to stay ahead of the curve. Residing the data close to the consumers is the best way to deal with such issues.

Solution: 

Oracle Globally Distributed Database meets data sovereignty requirements and supports applications that require low latency and high availability. It offers user-defined data placement for complying with regulatory requirements.

Oracle Globally Distributed Database provides comprehensive data sovereignty solutions that focus on the following aspects:

Data Residency: Data can be distributed across multiple shards, which can be deployed in different geographical locations.

Data Processing: Application requests are automatically routed to the correct shard irrespective of where the application is running.

Data Access: Data access within a region can be restricted further using the Virtual Private Database capability of Oracle Database.

Derivative Data: Ensuring that the data is stored in an Oracle Database, and using Oracle Database features to contain the proliferation of derivative data.

Data Replication: Oracle Globally Distributed Database can be used with Oracle Data Guard or Oracle GoldenGate to replicate data within the same Data Sovereignty region.

Customer story: 

Oracle has helped the leading global specialist in digital solutions for the health insurance industry achieve scalability and data residency through Oracle Globally Distributed Database.

Watch the Video

Read their story

Company’s Overview :

• Munich Re HealthTech (MRHT) is a division of Munich Re, one of the world’s leading providers of reinsurance, primary insurance and insurance-related risk solutions. MRHT, which is a leading global specialist in digital solutions for the health insurance industry, has been operating since 1995 and its solutions currently support 27 organizations in 19 countries in Europe, MEA, Asia and Latin America.
• They have an application called SMAART i.e Munich Re HealthTech’s all-in-one tool for sophisticated insurance claims reserving, advanced portfolio performance monitoring, and pricing system for the health insurance business. It provides you with a 360-degree view of your business performance, allowing you to monitor your portfolio profitability from a product and policy perspective.
• SMAART is built using the low-code Oracle APEX development platform within the Oracle Database

Business Challenges:

Data Sovereignty - Privacy & Residency

• SMAART is experiencing growth across multiple countries, and it requires adherence to data residency laws
• Clients that are using their systems are companies with a business that spans across several countries: Regulators (UAE, Saudi Arabia so far, and more countries in the ME in the coming future) are enforcing data residency requirements where personal and/or non-personal data must be stored locally
• There was a need to store some of the client data locally in the country of origin to comply with data residency regulations 
• Clients do not want to have a limited view of data per country, they need to be able to see the overall picture
• The maintainability of multiple infrastructures for a single client in multiple countries is an issue due to the costs involved
• Maintenance costs for MRHT are also higher due to increased support
• The customer evaluated several vendors to build a globally distributed database. However, migrating the data model and modifying the code would take several years if they build a globally distributed database. Some of the other shortcomings included a lack of full concurrency, transaction, and update support or joins

SMAART Got Intelligent Architecture With Oracle Globally Distributed Database

In-country data storage with a global view

• Oracle Globally Distributed Database architecture enabled a single logical database to be distributed over multiple geographies to achieve data residency requirement
• The application stack is deployed in one region, while database shards are stored in different countries
• Support from APEX to run multi-shard and single-shard queries and VPD (Oracle VPD) to handle row-level security 

The SMAART Approach

In-country data storage with a global view

1. Central ‘Catalog’ database and two sharded nodes one in each remote site
2. Selected the Sharding Key (in this case Client Organization ID)
3. We uploaded data in the Catalog database and verified that they were directly stored in the relevant sharded tables depending on the organization ID
4. Could connect to and query the Catalog database transparently, getting results for all sharded tables as if they were stored in the same centralized location
5. Could connect directly to the shard and retrieve information only for the data of the particular shard
6. Installed Oracle APEX on the Catalog database and installed the SMAART application
7. Applied VPD and tested VPD through APEX
8. Tested different types of user access through APEX

Business Benefits:

The client gained notable efficiencies from the comprehensive system overhaul:

High Performance 

• Sharded architecture delivered a single global database for the customer, offering higher performance than ever before
• This architecture substantially reduced global data replication and selectively replicate data only when needed to satisfy joins against local data
• APEX automated the data extraction from spreadsheets reducing the time to build analytical dashboards from 15 days to an hour and reducing human error and security risks
• Enabled customer to perform shard-specific DML or cross-shard DML without any impact on performance

Faster reporting response 

• Enabled customers to run reports and analytics against a single global database as and when needed
• This robust architecture eliminated the need to copy large amounts of data to a central warehouse as distributed parallel queries run against all shards with local data
• The architected solution offered a bird’s eye view of their entire cross-country portfolio, and at the same time limited access of users to specific areas that how much they need to see

The actuary is making better decisions now

• Oracle capabilities empowered them to build AI forecasting models by using the R interface with Oracle Machine Learning in Oracle Database
• The client achieved enhanced performance and isolated data manipulations at the local shard site with the support of R, ODI, and ODM
• Fully supports Oracle Analytics Server and enables cross-shard access and single-shard access based on user setup and configuration (can be enabled in RPD)

This is how Oracle Globally Distributed Database made it possible to satisfy data privacy regulatory requirements related to data residency where regional data would remain in respective regions and storing data close to its consumers that delivered data proximity.

To conclude, Oracle Globally Distributed Database is the perfect solution for covering data residency requirements while keeping central maintenance of their application and users. The solution can deliver hybrid functionalities leveraging Cloud and on-premises infrastructures.

The Way Forward:

Since 2014, top insurance players had delivered an annual Total Shareholder Return of 20% by the end of 2019. However, by the second quarter of 2020, the insurance sector underperformed the market by approximately 12%. The insurance industry’s proportionality with economic growth and fallouts is crystal clear. Going forward, technological enhancements and data sovereignty will have a significant impact on the growth potential of the insurance industry and ultimately the global economy. The road ahead will necessarily be paved by the growing digital and technological capabilities of countries as they aim for newer advancements in technology and digital sovereignty. Oracle Globally Distributed Database is all set to help its customers in this upcoming wave of disruption.

Source: oracle.com

Continue reading

Don’t Be Fooled by Misleading Data Egress Announcements

Part 1: The Hotel California plan meets the European Commission

Recent announcements in response to the European Data Act have led some to conclude that major Cloud Service Providers (CSP) have eliminated Data Movement and Data Egress charges, but in the ways that matter most to companies running their business in the Cloud on an ongoing basis, that’s not true. Most CSPs still charge for data moved between Availability Zones / Domains in the same Region (which is free with Oracle), between Regions, and ongoing Data Egress from their Clouds. Oracle still allows you to move a lot more data before charges start, and the rates are *much* lower once they do. That hasn’t changed.

There are a lot of articles and blogs out there that refer to the common CSPs model of charging high fees for data egress as the ‘Hotel California’ plan, in reference to the old Eagles song that ends, “You can check out anytime you like, but you can never leave.” While I’ve always loved the song, the analogy is a bit strained. You could always leave a cloud provider, but it’s been a matter of paying a potentially very hefty bill when you did so – and the more data you have to move, the higher that bill.

Data Egress charges often come as one of the biggest surprises to an organization that’s relatively new to the cloud world. For a great explanation of what they are and why they exist, check out Cloud Data Egress Costs: What They Are & How to Reduce Them. As noted in that article, there are quite a few things you can do to help reduce the charges you incur, but many of them require extensive planning up front, and potentially making changes to the architecture of an application you plan to migrate to the cloud.

Different Types of Data Movement Charges

For this discussion, I’d like to differentiate between three different types of data movement charges. First, let’s look at (1) Data Movement charges which encompass transfer of data within the CSP’s ecosystem, e.g. from Availability Zone to Availability Zone or Region to Region within the same Cloud, and thus might not be truly considered to constitute Data Egress. Then, within the realm of true Data Egress charges there are those that occur (2) as part of normal ongoing business processes, or (3) on a one-time basis as part of a migration away from utilizing the CSP’s services altogether – perhaps in order to move instead to another Cloud where either the services available or the pricing model is a better fit for your needs. Let’s look at some examples.

Data Movement within the Same Cloud

As an example of Data Movement within the same cloud, if for Disaster Recovery purposes you wanted to keep a replica of a key database in another Region you’d need to move data from one region to the other to keep your DR copy in sync. This is a very common use case as companies spread their data across multiple cloud regions. This may be primarily to provide resiliency in the event of a region-level failure or service interruption, or just to reduce latency by making data available closer to where those who need to access it are located. In Part 2 of this blog we’ll look at a detailed comparison of data movement costs in that database DR scenario with the various CSPs.

As many of you are no doubt aware, many cloud Regions consist of more than one datacenter located in close proximity. Each of these is usually referred to as an Availability Zone (AZ) or Availability Domain (AD). In concert, they provide resiliency in the event of an interruption of service that impacts an entire datacenter. In order to get the benefit, a customer would need to distribute their data at minimum, and potentially the ability to process that data as well, across multiple AZs. But moving your data from one AZ to another, whether to keep multiple copies in sync or as part of your normal processing, may come with data movement charges.

While events that impact an entire cloud region are relatively rare, they do happen. As recently as June 13, 2023, AWS experienced failures in Lambda function invocations across the US-East-1 (Northern Virginia) region that also impacted several other services as a result. While the problem was resolved fairly quickly (most services were reported to be fully recovered in a little less than 4 hours), as the primary manager of a customer AWS environment at the time I was very happy we had complete redundancy in US-West-2 and had the option to shift our critical processing there for much of the afternoon until US-East-1 had recovered. In order to have this capability we maintained synchronized replicas of a couple of important databases that then resided in AWS in both Regions. Keeping them in sync in order to be able to shift processing during the service outage comes with an associated cost including not only the additional storage required, but also – you guessed it – data movement charges.

Figure 1: Data Movement Within the Same Cloud

 

Unlike Amazon Web Services (AWS), Microsoft Azure, and Google Cloud (GCP), Oracle Cloud (OCI) doesn’t charge for data movement within the same Region, even between services. So, types (1) and (2) of Data Transfer as illustrated in Figure 1 are free with OCI.

Data movement between regions within the same cloud (Type (3) as shown in Figure 1 is typically billed based on amount of data transferred out from the source region, just like egress from the cloud to the public internet but often at lower rates. CSPs typically provide a monthly allowance of Data Egress before billing commences based on volume of data transferred out. AWS and Azure allow 100 GB/month, GCP gives 200 GiB/month, whereas OCI provides a whopping 10 TB/month for free. Once charges do kick in they’re often tiered and can also vary from region to region but Oracle consistently charges significantly less as shown here.

Ongoing Data Egress Charges

Next, and also very common, are ongoing Data Egress charges from the Cloud to either the public internet, or to the customer’s on-premises datacenter. Examples might include a webserver feeding data out to the browsers of all the users that visit the website over the internet (At scale this would often entail usage of a CDN or Content Delivery Network, in many cases another service from the CSP), or a bucket in cloud object storage where data is periodically deposited by one party and subsequently retrieved for processing either on-premises or in a different cloud by another.

In our Disaster Recovery example discussed above, if you wanted to go so far in your plan as to have redundancy across multiple Clouds, the data you moved between the CSPs to keep things in sync would typically incur Data Egress charges.

Figure 2: Data Egress from Cloud to Public Internet or Customer Data Center

 

Just as with our Region-to-Region scenario above, when it comes to egress to the public internet, Data Transfer type (4) as illustrated in Figure 2, CSPs typically provide the same monthly allowance of Data Egress before billing commences based on volume of data transferred out. In this case AWS and Azure allow 100 GB/month, GCP gives 200 GiB/month, whereas OCI again provides a massive 10 TB/month for free. Once charges do start to accrue they’re often tiered and can also vary from region to region but once again Oracle consistently charges significantly less than the other CSPs as shown here.

An even greater contrast in pricing model can be seen in the case of private or dedicated line connectivity, as indicated by Data Transfer type (5) in Figure 2. Where the other CSPs have both port and data volume charges for this type of connection, the OCI FastConnect service has no volume charges, just the port fee. This results in much lower and more predictable cost.

One-time, Unidirectional Migration of Data

Finally, we have the one-time, unidirectional movement of data away from the CSP altogether. Compared to the other types of ongoing data movement we’ve discussed this is a rare occurrence. If a large customer were to decide they wanted to move all their data away from AWS, for example, to another Cloud, depending on the volume of data in question these costs could be quite significant and constitute a barrier to the ability for that customer to freely move from one CSP to another. As a result, these charges have been seen as a potential contributor to vendor lock-in, not only by customers but also by government regulators including the European Commission, the US Federal Trade Commission, and UK regulator Ofcom.

Regulation of Data Movement Charges: The European Data Act

On January 11, 2024 the European Commission issued a Press Release announcing the enaction of the European Data Act. One of the points referenced states “Furthermore, the Data Act will allow customers to switch seamlessly (and eventually free of charge) between different cloud providers.” The Data Act comes into force after 20 months, on September 12, 2025.

Also on January 11, 2024 Google Cloud issued an announcement stating that starting on that date “Google Cloud customers who wish to stop using Google Cloud and migrate their data to another cloud provider and/or on premises, can take advantage of free network transfer to migrate their data out of Google Cloud.” Coincidence? Seems unlikely.

On March 5, 2024, AWS appeared to follow the precedent with release of this blog post on “Free data transfer out to internet when moving data out of AWS”. This post explicitly states that the new policy “…follows the direction set by the European Data Act”, and, while it states, “We don’t require you to close your account or change your relationship with AWS in any way.” It also links to an FAQ which states that “After your move away from AWS services, within the 60-day period, you must delete all remaining data and workloads from your AWS account, or you can close your AWS account”.

The following week, on March 13, 2024, Microsoft also published an update titled “Now available: Free data transfer out to internet when leaving Azure.” This one also explicitly cites alignment with the European Data Act.

Worth noting is that if you dig just a bit deeper in the links provided in each of these announcements, you’ll find that getting credit to offset these Data Egress charges for migration requires the customer in question to contact either their Account Team or the CSP’s Support organization and request approval. That only makes sense as any business wants to retain their customers and will surely ask probing questions into why they’re migrating away. In most cases they’ll probably offer incentives in a final attempt to entice the customer to change their mind about leaving.

A (small) Step in the Right Direction

While the opportunity to have Data Egress charges that are incurred during a migration away from GCP, AWS, or Azure waived is a positive development for any customers who do want to migrate from one Cloud to another (and also allows those CSPs to proclaim their compliance with new regulations), there have been many indications that some may have interpreted these announcements as having broader implications than would appear to actually be the case.

Of the three types of Data Movement charges we defined above, these policy changes as described in their respective announcements apply only to the last, i.e. Data Egress on a one-time basis as part of a migration away from utilizing the CSP’s services altogether. They have no impact on either of the first two types of data movement charges, i.e. those incurred either within the same cloud or ongoing Data Egress charges as a result of conducting business in a hybrid or multicloud model.

In the great majority of cases these two types of data movement constitute a significantly larger volume than do those of actual migrations away from one of the major CSPs. In a previous role, I managed the operations of both the AWS and GCP environments used by my employer at the time, and I can attest that these changes would not have reduced our data movement charges from either vendor during the course of a normal month by so much as a single cent. I’d assert that this would also be the case for the vast majority of cloud customers during the course of their normal operations.

In summary, it’s nice that these large CSP’s are prepared to be more flexible and reduce the financial penalties they’ve historically assessed when customers want to stop using their platforms. But don’t read into this any more than that. If you’re skeptical, click through the links to the individual announcements provided above and read them closely, then follow the links they contain to additional documentation and FAQs from the respective vendors. If you still expect these announcements are going to bring any relief from the Data Movement charges you incur every month during your normal operations, have a discussion with your Account Team. I think you’ll find that the Hotel California model would appear to be as alive and well as it ever was. You can check out anytime you like. And now they won’t even charge you by the ounce for each piece of your own luggage you want to take with you. Well, actually they still will, but if you ask nicely and tell them why you’re leaving they might give you a credit back. Either way, for all the fees they charge you during your stay with them – nothing has changed.

Source: oracle.com

Continue reading

Make Better Maps for Your Apps with Spatial Vector Tiles and H3 in Oracle Database 23ai

Oracle Spatial is introducing these new features for developers in Oracle Database 23ai:

• Vector tile support: Do you want to create maps from spatial data in your Oracle database?  You can now generate vector tiles via an in-database function and then stream your spatial data as vector tiles for scalable, flexible maps for web and mobile applications. 
• H3 support: You can use easy-to-view hexagon cells to create compelling visuals and summaries of very large volumes of point data.

As you’ve probably heard, Oracle just announced the general availability of Oracle Database 23ai. In alignment with the focus of the release on AI and developer productivity, Oracle Spatial offers several improvements to simplify the experience of developing applications by removing the complexity associated with your database interaction. These improvements range from streamlined SQL syntax that makes it easier to work with the spatial vector data type, to a REST API for raster data and more functions for working with Lidar point clouds. 

Now, back to vector tiles and H3.  These are very exciting new features, as ITIS, a solutions provider for the transportation and safety sector, told us - see the quote above.

What are spatial vector tiles?

In the world of maps and spatial, the vector tile format is a highly efficient, compressed data format optimized for web applications to consume and render spatial data (note: spatial vector tiles are different from AI vectors for generative AI, a powerful Oracle Database 23ai capability you may have heard of). Like GeoJSON, vector tiles enjoy wide support from developer web kits and spatial applications. But what distinguishes vector tiles is their superior performance: they allow you to get much larger volumes of data very quickly into mapping applications.

Vector tiles are also versatile, supporting dynamic styling. That is, map styles (which define the look of elements in your map) can be applied on the fly in the client, so you have flexibility in applying different styles to the same data. In addition, vector tiles provide fast performance, smooth map interactions, and dynamic map queries. They’re the industry’s preferred method for scalable delivery of spatial data to mapping client applications.

A thematic map generated from vector tiles

 

How do you create vector tiles in the database?

So, what we’ve done in Oracle Database 23ai to support vector tiles is to add a function that lets you generate vector tiles – just with a simple SQL call. It’s a straightforward function called sdo_util.get_vectortile().

The get_vectortile() function accepts the geometry column in a table, tile identifier, and attributes as parameters, and returns the corresponding vector tiles.

Here is an example where we’re creating a vector tile based on the geometry column in the CUSTOMERS table. Customer attribute information will also be returned to the vector tile, so that further filtering or analysis can be performed in the browser client.

SELECT sdo_util.get_vectortile(

  'CUSTOMERS',                                                                                -- table name

  'GEOM',                                                                                            -- geometry column name

  6,                                                                                                      -- tile x

  14,                                                                                                    -- tile y

  25,                                                                                                    -- tile zoom

sdo_string_array('NAME', 'ZIP', 'CITY', 'STATE', 'SALES_2022') -- attributes

FROM customers

You can also use an SQL query to specify criteria for data you want in your vector tile map. For example, you may want to map all the sales territories in a specific region with more than 1000 customers between the ages of 25-40.

The function returns the vector tile data as a binary structure. It is not human-readable, but it is a highly efficient, compressed format containing map data that is tailored to be consumed by web clients. And because spatial data in the database is converted to vector tiles right in the database your architecture is simplified.

To be useful to end users, the result of the vector tile query needs to get streamed out to a web application that can consume it. A great way to do this is through REST APIs, which are a convenient way for backend systems to talk to web applications. Thanks to Oracle Database as a converged database, it’s very easy to create REST APIs in the database with Oracle REST Data Services (ORDS) which takes any SQL query and exposes it as a REST API. The end result is that modern web mapping clients can consume vector tile data generated through the get_vectortile() function and exposed with REST APIs.

Now, let’s look at H3, a spatial data structure that’s great at visualizing and making sense of very large point data sets.

What is H3?

Hexagons are a very visually appealing and useful way to view data on maps, especially when you have very large amounts of dense point data, and want to easily view summaries, patterns, and clusters in data. They also have advantages in terms of geospatial analysis, coverage, and accuracy.

Oracle Spatial now provides support for hexagonal hierarchical spatial indexing (H3) in Oracle Database 23ai. H3 is an open source indexing system developed by Uber that uses hexagons in a grid system to cover the earth’s surface. With H3, data points are bucketed in hexagon cells and attributes can be aggregated and summarized over these hexagons. For very large volumes of point data, H3 cells are very useful for visualizing aggregated attribute data in thematic maps.

From the technical point of view, H3 takes dense, large-volume point data and constructs summaries of the data in a hierarchical structure. For example, you might have a data set with all the car crashes that occurred in California in 2023 with more than 100,000 data points. Visualizing these thousands of points on a map as is will make the visual representation of the data very difficult. Instead, you can use H3 to generate a data structure that summarizes the total number of crashes in each hexagon cell at different zoom levels. You can then view these on a map as color-coded hexagons – instead of as a confusing smudge of too many points. This makes it much easier to view and interpret large datasets – enabling an at -a-glance view of hot spots, clusters, and areas of low activity.

Visualization of H3 hexagons color coded by the number of incidents in each hexagon

How does H3 indexing work in Oracle Database?

In order to deliver map visualizations to your client application, you will go through two main steps – generating the H3 index and then turning the H3 into vector tiles.

First, you generate the H3 index for the point data we’re interested in using the H3 hierarchical grid system. H3 index creation is done with the new function SDO_UTIL.H3SUM_CREATETABLE(), which creates a new table in the database for the H3 representation of the data. You specify your input data (such as crash locations) and pick the type of aggregation you want (such as the count of crashes in each cell).

Second, you turn H3  hexagons into vector tiles using the new function SDO_UTIL.H3SUM_VECTORTILE(). This is similar to what we described with the get_vectortiles( ) function above, except that this function is specifically for converting H3 hexagons into vector tiles. With these 2 simple steps, you can aggregate a large data set into an aggregated hexagon data set and visualize it as thematic maps that are easy to interpret and visually appealing. Your web applications can easily consume the hexagon data as vector tiles exposed with a REST API.

Summary

Vector tiles and H3 are very popular techniques for maps in web and mobile applications, so we made vector tiles and H3 developer-ready in Oracle Database 23ai – you can work with them right out of the database. These new features will help you create powerful maps from your data and enhance your web applications, for data-driven applications with integrated spatial analytics.

Source: oracle.com

Continue reading

Consolidating Development Environments with Autonomous Database Elastic Pools

You are dying to get a reservation at that hot new restaurant for you and another couple. You get a tip that a table may be available for this weekend. Since you want to keep the spot, you call them and make the reservation for a table of 4, then try to track down your friends.

Saturday comes around, and your friends still have not committed, so you head to the restaurant anyway. They seat the 2 of you at a table of 4, and your friends never show. Afraid the restaurant will charge you for no-show friends, you are pleasantly surprised to pay for the meal for the two even though you had a lovely big table ready for 4.

Autonomous Database Elastic Pools

Similarly, Autonomous Database Elastic Pools (Elastic Pools) allow you to plan for what you could use but only pay for what you actually do use in terms of ECPUs. You choose a set pool of ECPUs (e.g., 128, 256, 512,1024, 2048, 4096) but can use up to 4X that amount of ECPU (e.g., 4 * 128 = 512, 4* 256 = 1024) when needed. Because you have allocated the required space in the pool for each database when it needs to expand to that size, the capacity is there for you, but you only pay based on the pool utilization each hour. Elastic Pools also give you finer-grained controls via ECPUs with any database in the pool since Elastic Pools only require a minimum of 1 ECPU instead of the standard 2 ECPU for non-pooled databases.

Development and Test Environments

Your development and test environments are probably notorious for having resources that are either overallocated or have resources sitting idle. Idle and overallocated resources contradict the benefits of running in the cloud. I worked with a financial services customer that runs 11 non-production databases for every major application. All these environments, though, are not required to be fully on all the time. This is the situation where the elasticity of the cloud starts to pay off. With hundreds of applications spawning many different non-production copies, the chance of you having idle or overallocated databases is high if not properly managed.

Autonomous Database Autoscaling and scheduled startup and stops can help you at a database level by controlling the minimum ECPUs and shutting them down when they are not in use. Although this will help to cut down on unused asset costs, it does not take advantage of the economies of scale with a large amount of highly transitory databases you find in development.

Moving your development databases to an Elastic Pool minimizes or eliminates the time and effort to manage resources for transitory databases, which may start and stop multiple times per sprint. The Elastic Pool allows you to manage resources at a pool level with databases able to consume what they need and then release those resources back to other development databases without administrator intervention.

Example

Assume you have a shared development environment for 50 applications with five development/test environments each. Development may use these databases for a single sprint or less, but they must be sufficiently sized when in use and able to start up on demand without administrative wait time.

Configuring with Autoscaling

Using an Autoscaling Autonomous Database, you would also have to start and stop each database and individually manage each database.

• With 50 Applications * 5 environments = 250 possible databases
• In a non-pooled environment, you could start with a 2 ECPU with autoscaling up to 3X the initial size for each database.

Configuring with Elastic Pools

If you run this in an Elastic Pool, you can improve your efficiency further and get more resources and finer-grained controls.

• The first benefit is that each database minimum is 1 ECPU
• Some of these databases could be allocated to as low as 1 ECPU, while you may allocate test databases to a higher capacity.
• As long as you stay below the 512 (4X of 128) pool capacity of a 128 ECPU Elastic Pool, you would only need a 128 ECPU Elastic Pool.
• If a database only uses 1 ECPU but suddenly needs more capacity, it would have up to the allocated amount ready, which you should reflect in the allocation level. Elastic Pools immediately release those extra ECPUs to the pool when the workload scales back down.

Development High Peak Use Case

In the rare case that all 250 databases ran at low usage during a peak hour, in an autoscaling configuration, you would consume 250 databases at 2 EPCU each = 500 ECPU for that hour. Elastic Pools would handle this differently by scaling the Elastic Pool to meet the concurrent databases. At 250 databases with a minimum of 1 ECPU, this is half of the autoscale usage, and you could handle with a 256 ECPU Elastic Pool for that hour and scale back to 128 as the demand drops. The result is nearly halving the cost of development ECPUs for that hour.

Idle Development Database Use Case

If databases are used but left idle and not shut down, they will continue to be billed at the minimum ECPU in an autoscaling configuration. Databases in an Elastic Pool will consume the ECPUs in the Elastic Pool only when utilized. For example, 20% of the 250 development databases, each with a minimum configuration of 2 ECPUs, are left running by mistake. In an autoscaling configuration, this would be 25 databases * 2 ECPU, or 50 ECPU consumption for every hour they are left on. In Elastic Pools, unless you are utilizing these databases, they will not consume ECPUs from the pool, resulting in a savings of the cost of up to 50 ECPUs.

Conclusion

Autoscaling is a way to manage your capacity for individual databases. Elastic Pools takes this further by providing an innovative way to consolidate your database environment by simplifying Autonomous Databases' overall management and capacity planning across large estates of development databases. By understanding and using the concept of 4X capacity and allocating databases properly, you can put a pool in place that will lower overall costs and ensure the capacity is there when needed. The development environment is one of many use cases in which Elastic Pools can simplify and cut costs in a modern database estate.

Source: oracle.com

Continue reading