Monday, February 12, 2024

Simplified Data Safe notifications improve data security visibility

Notifications in OCI provide a common, efficient infrastructure for operational alerts and application integration. OCI notifications are reliable, low latency, and implement a publish and subscribe paradigm that reduces the need for polling by connected applications. Setting up notifications from scratch can be challenging, however. This is why Data Safe now makes it easier to configure notifications. Notifications allow you to quickly set up and receive messages informing you about a variety of different events and conditions in Data Safe. Best of all, you can now define notifications directly within the Data Safe console without navigating away from the feature you are working on. Notifications can improve operational security by alerting you with an email, SMS message, or Slack message when something in your database environment changes.

What are notifications?


To understand how we've made notifications easier to use, it’s necessary to first understand a little bit about how OCI notifications work. OCI notifications involve creating and associating OCI resources to create operational alerts. These resources include:

  • Events –structured messages that indicate changes in Data Safe resources.
  • Rules –filter the stream of events to trigger actions such as notifications for selected event conditions.
  • Notifications –messages that are created when some defined event that meets a rule’s condition is generated in OCI.
  • Topics –communication channels that are used to forward notifications to endpoints.
  • Subscriptions –endpoints to which notification messages are sent. For example, a subscription endpoint could be an email address, phone number, or Slack channel.

Putting it all together, an event that meets conditions defined by a rule triggers a notification that is sent to subscribers of a topic.

We've added contextual notifications to the Data Safe console to make setting up notifications easy for users. We are calling these contextual notifications because the events you choose from are now displayed within the context of the Data Safe function you are working with. No more scrolling through hundreds of different events trying to find the ones that are relevant to you!

A new notifications tab can be found in the various features of Data Safe, allowing you to create and manage notifications for a Data Safe feature while working in the context of that feature. You will find notification tabs in many areas of the Data Safe console, for example:

  • User assessment
  • Security assessment
  • Discovery
  • Masking
  • Activity auditing
  • SQL Firewall

To make getting started with notifications even easier, we created quick-start templates for the most common use cases. Let's look at an example of creating a simple but useful notification to show you how this works.

Security configuration drift.


Many data breaches can trace their cause to administrator error. A frequent source of that error is configuration drift. A setting or parameter is changed – often to facilitate troubleshooting or solve an urgent operational issue – and that change weakens the system's security. Data Safe helps you identify this type of "drift" away from an approved security baseline, and notifications can bring that drift quickly to your attention. Let's use the new notifications capability to email us if Data Safe spots configuration drift. We first open the Security assessment page and click the new notifications tab.

Simplified Data Safe notifications improve data security visibility

On the notifications tab we see quick start templates – these are ready-to-go notification policies that make sense in the context of the Data Safe area you are working in. We’ll click A security assessment has drifted from baseline to create our notification.

Simplified Data Safe notifications improve data security visibility

To create our notification, we add three things to the template:

  • Name of the notification rule – this is what you want to call the notification, and it will appear in the email you get when security drift occurs
  • Topic name – this is what you will call the OCI resource to which you’ll add notification subscriptions. Something to watch out for when entering a topic name is that there can be no spaces in the topic name.
  • Subscription – these are the locations to which you want notification messages to be sent. The most common form of subscription is an email, so you would enter an email address. Other types of subscriptions include SMS or Slack messages, RESTful API calls, or calls to OCI functions.

That’s all it takes – click Create notification, and the next time Data Safe detects that the most recent security assessment doesn’t match the baseline, you’ll get an email letting you know.

Note: for email subscriptions, Data Safe will first send an email to the address you entered asking you to confirm you wish to subscribe to the notification. Until you confirm the subscription, Data Safe will not actually send a notification email to that address.

Where can you find notifications within Data Safe?


You’ll find the new notification tabs throughout Data Safe – wherever it makes sense in the context of the screen you are working with to proactively notify you that something has changed. These include:

  • Security Assessment
  • User Assessment
  • Data Discovery – Sensitive data models
  • Data Masking – Masking policies
  • Activity auditing – Audit profiles, policies, trails, Archive data retrievals, and reports
  • SQL Firewall
  • Alerts – Reports and Target-policy associations

Notifications work the same way throughout Data Safe except for Target-policy associations in the alert console. We'll look at an example of that next.

Target-policy notifications.


Data Safe includes several predefined alert policies for conditions like SQL Firewall violations, user profile changes, user entitlement changes, etc. When Data Safe detects the conditions defined by one of these policies, an alert is created, which you can view in the All alerts report. Alerts can also trigger an alarm, which in turn can be used to trigger a notification. Target-policy associations allow you to enable an alert policy for a Data Safe target database.

Simplified Data Safe notifications improve data security visibility

At first glance, these are similar to other notifications – two quick start templates are defined.

  • An alert was generated – this will send a notification anytime Data Safe detects an alert triggered for any target database with an alert policy associated with it
  • More than 1000 alerts were generated in 5 minutes – this is just what it sounds like – Data Safe will send you a notification if more than 1,000 alerts are triggered within five minutes. That can be for ANY of your targets, and the 1,000 count is cumulative across them.

These predefined templates make sense for Data Safe instances with just a few target databases. But as the number of target databases increases, the number of alerts generated in the system may also increase, and it might make sense to define your notification more narrowly. Let's take a look at a common use case – user entitlements.

Advanced alert notifications


We’re going to narrow down the conditions Data Safe will check to send us an alert so that we don’t receive notifications for things we don’t care about. In the case of user entitlement changes, I might only care if a new entitlement is granted, but not care if an entitlement is revoked. To create this kind of custom notification, we will click Create notification (the grey button) instead of using one of the templates. Then, we’ll click Advanced alarm notification. This opens a form where we can create our advanced notification. In the screenshot below you’ll see that I’ve added callouts to indicate where changes are entered into the form – for this use case, there are eight of them. That number will vary depending on how many rules you create.

  1. Alarm name – similar to the simplified notification templates like Security assessment drift from baseline, this is just what you want to call your notification. It appears as the description field in a notification message.
  2. Event type – there is only one choice here, Alert Generated.
  3. Click + Another condition to add a new condition. We’ll select Attribute for the condition, and then displayName for the attribute name. In Attribute values enter “User entitlement changes” – this is the name of the alert policy. You can find a list of alert policy names in the Alert policies resource page.
  4. Click + Another condition to add a second condition. This time, we'll select operation, and enter GRANT as the value we want to look for.
  5. Metric – Choose DeliverySucceedEvents and leave the rest of the fields at their default values.
  6. Trigger rule – This is where you select how many alarms need to occur before you will be notified. In this case, we want a notification anytime someone grants a new privilege, so we’ll choose “equal to” for the Operator, set the Value to one, and enter a Trigger delay in minutes of one.
  7. Now we are back on familiar ground – enter a topic name (don’t forget – no spaces allowed).
  8. Enter the email address you want to receive the notification.

Click Create notification and you’re done!  You’ll be notified whenever Data Safe detects a user entitlement change AND that change is a grant.

By stacking different conditions you can narrow your notification focus to just those alerts you care about, reducing the “noise” of unnecessary notifications.

Note: conditions are always cumulative – always an “and” operation. There is no ability to have an “or” or “not” condition.

Simplified Data Safe notifications improve data security visibility

And that’s all there is to it. Notifications are extremely flexible and can be tailored for a wide variety of use cases. If you’d like to see the new feature in operation, here is a short video that walks you through several ways of using it. 

Source: oracle.com

Friday, February 9, 2024

Oracle Globally Distributed Database supports RAFT Replication in Oracle Database 23c

Distributed databases are gaining increasing popularity and significance due to their capacity to manage large and intricate datasets, scalability, flexibility, and compatibility with contemporary application designs and architectures. As organizations embrace cloud-based solutions and microservices-based architectures, there is an anticipated surge in the demand for distributed databases. Businesses are on a continual quest for innovative database solutions that facilitate horizontal scalability, ensure data durability, and maintain high availability while adopting more adaptable strategies.

Scaling databases is imperative for handling transactional volume and ensuring optimal performance. Additionally, in an increasingly global business landscape, geographic scaling is crucial to tailor services to customers located in different parts of the world.

Resilience is a vital consideration when scaling databases, as businesses strive to guarantee survival in the event of the failure of an entire Availability Zone/Availability Domain (AZ/AD) or cloud provider.

The escalating demand for distributed databases, coupled with the necessity for horizontal and geographic scaling, is propelling businesses toward adopting next-generation methodologies such as Oracle Distributed Database. This ensures efficient performance and resilience, meeting the expectations of modern consumers. Oracle Distributed Database functions as a data distribution system, employing advanced techniques to partition data across multiple servers or shards, thereby delivering exceptional performance, availability, and scalability.

Oracle Database 23c introduces a noteworthy feature in the form of Raft replication, a consensus-based replication protocol that facilitates automatic configuration of replication across all shards. Raft replication seamlessly integrates with applications, providing transparency in its operation. In case of shard host failures or dynamic changes in the distributed database's composition, Raft replication automatically reconfigures replication settings. The system takes a declarative approach to configure the replication factor, ensuring a specified number of replicas are consistently available.

Swift failover is a key attribute of Raft replication, enabling all nodes to remain active even in the event of a node failure. Notably, this feature incorporates an automatic sub-second failover mechanism, reinforcing both data integrity and operational continuity. Such capabilities make this feature well-suited for organizations seeking a highly available and scalable database system.

When Raft replication is enabled, a distributed database contains multiple replication units. A replication unit (RU) is a set of chunks that have the same replication topology. Each RU has multiple replicas placed on different shards. The Raft consensus protocol is used to maintain consistency between the replicas in case of failures, network partitioning, message loss, or delay.

Replication Unit

When Raft replication is enabled, a distributed database contains multiple replication units. A replication unit (RU) is a set of chunks that have the same replication topology. Each RU has three replicas placed on different shards. The Raft consensus protocol is used to maintain consistency between the replicas in case of failures, network partitioning, message loss, or delay.

Each shard contains replicas from multiple RUs. Some of these replicas are leaders and some are followers. Raft replication tries to maintain a balanced distribution of leaders and followers across shards. By default each shard is a leader for two RUs and is a follower for four other RUs. This makes all shards active and provides optimal utilization of hardware resources.

Raft Group

Each replication unit contains exactly one chunk set and has a leader and a set of followers, and these members form a raft group. The leader and its followers for a replication unit contain replicas of the same chunk set in different shards as shown below. A shard can be the leader for some replication units and a follower for other replication units.

All DMLs for a particular subset of data are executed in the leader first, and then are replicated to its followers.

Replication Factor

The replication factor (RF) determines the number of participants in a RAFT group. This number includes the leader and its followers.

The RU needs a majority of replicas available for write.

  • RF = 3: tolerates one replica failure
  • RF = 5: tolerates two replica failures

In Oracle Globally Distributed Database , the replication factor is specified for the entire database, that is all replication units in the database have the same RF. The number of followers is limited to two, thus the replication factor is three.

Raft Log

Each RU is associated with a set of Raft logs and OS processes that maintain the logs and replicate changes from the leader to followers. This allows multiple RUs to operate independently and in parallel within a single shard and across multiple shards. It also makes it possible to scale the replication up and down by changing the number of RUs.

Changes to data made by a DML are recorded in the Raft log. A commit record is also recorded at the end of each user transaction. Raft logs are maintained independently from redo logs and contain logical changes to rows. Logical replication reduces failover time because followers are open to incoming transactions and can quickly become the leader.

The Raft protocol guarantees that followers receive log records in the same order they are generated by the leader. A user transaction is committed on the leader as soon as half of the followers acknowledge the receipt of the commit record and writes it to the Raft log.

Transactions

On a busy system, multiple commits are acknowledged at the same time. The synchronous propagation of transaction commit records provides zero data loss. The application of DML change records to followers, however, is done asynchronously to minimize the impact on transaction latency.

Leader Election Process

Per Raft protocol, if followers do not receive data or heartbeat from the leader for a specified period of time, then a new leader election process begins.

The default heartbeat interval is 150 milliseconds, with randomized election timeouts (up to 150 milliseconds) to prevent multiple shards from triggering elections at the same time, leading to split votes.

Node Failure

Node failure and recovery are handled in an automated way with minimal impact on the application.

The failover time is sub-3 seconds with less than 10 millisecond network latencies between Availability Zones. This includes failure detection, shard failover, change of leadership, application reconnecting to new leader, and continuing business transactions as before.

The impact of the failure on the application can further be abstracted by configuring retries in JDBC driver and end customer experience will be that a particular request took longer rather than getting an error.

The following is an illustration of a distributed database with all three shards in a healthy state. Applications requests are able to reach all three shards, and replication between the leaders and followers is ongoing between the shards.

Oracle Globally Distributed Database supports RAFT Replication in Oracle Database 23c

Leader Node Failure

When the leader for a replication unit becomes unavailable, followers will initiate a new leader election process using the Raft protocol.

As long as a majority of the nodes (quorum) are still healthy, the Raft protocol will ensure that a new leader is elected from the available nodes.

When one of the followers succeeds in becoming the new leader, proactive notifications are sent from the shard to the client driver of leadership change. The client driver starts routing the request to the new leader shard. Routing clients (such as UCP) are notified using ONS notifications to update their shard and chunk mapping, ensuring that they route traffic to the leader.

During this failover and reconnection period, the application could be configured to wait and retry with the retry interval and retry counts settings at the JDBC driver configuration. These are very similar to the present RAC instance failover configuration.

Upon connecting to new leader, the application will continue to function as before.

The following diagram shows that the first shard failed, and that a new leader for the replication unit whose leader was once on that first shard has been replaced by a new leader in the second shard.

Oracle Globally Distributed Database supports RAFT Replication in Oracle Database 23c

Oracle Globally Distributed Database provides commands and options in the GDSCTL CLI to enable and manage Raft replication in a system-managed sharded database.

Source: oracle.com

Wednesday, February 7, 2024

Oracle Database@Azure now Generally Available in Azure East US Region to accelerate your data center exit

We’re proud to announce the general availability of Oracle Database@Azure, a joint Oracle and Microsoft service that allows customers to easily provision, access, and operate enterprise-grade Oracle database services as native resources directly in Microsoft Azure. Customers can purchase Oracle Database@Azure in the Azure Marketplace and apply their Oracle database cost directly to their Microsoft Azure Consumption Commitment (MACC).

In this blog post, we provide a detailed explanation of provisioning and management.

Service overview


Since 2019, we have partnered with Microsoft to deliver the Oracle Interconnect for Azure, which offers secure, private interconnections with sub-2 millisecond latency, in 12 global regions to date. This high-performance network foundation enables customers to build applications with components straddling OCI and Azure. However, the flexibility to connect any resource in either cloud requires customers to configure and manage the connections between the cloud environments, DNS entries, and more.

With Oracle Database@Azure, Oracle and Microsoft deliver the same Oracle Database services that run on OCI, physically in Microsoft Azure data centers. By collocating OCI database services in Azure data centers, Oracle Database@Azure has low latencies like other Azure services, and provides the same high levels of performance, scale, security, and availability that we offer in OCI today. In a few steps, you can procure, deploy, and manage the service through the existing Azure portal, use Azure developer tools, software development kits (SDKs), and APIs. Metrics, logs, and events for the Oracle databases you create using the service will be available in Azure tooling for unified telemetry and monitoring in Azure environments.

We’ve made it easier for customers to migrate and modernize their Oracle Database workloads to Azure. With Oracle Database services running locally in Azure, you can benefit from proximity to familiar application development tools and frameworks to modernize your workloads and run cloud native applications with direct access to data in Oracle databases with the same security and compliance across the entire solution.

Purchasing

Purchasing Oracle Database@Azure is a simple process, facilitated by private offer in the Azure Marketplace. Azure permissions might be required for the user completing the private offer purchase.

Oracle Database@Azure now Generally Available in Azure East US Region to accelerate your data center exit

Private offer for database consumption

All who purchase Oracle Database@Azure must have a private offer created and made available in the Azure marketplace. To have your private offer created, existing Oracle customers can reach out to their Sales Representative.

Private offers for Oracle Database@Azure can be completed in a week for actively engaged customers. Working with your sales representative, you can customize a quote based on your specific needs, then sign and approve. Oracle then generates your private offer in the Azure marketplace. To access your private offer, log in to the Azure marketplace and view your private offers or use the Oracle-provided offer ID through your email. Move through a few screens to accept your private offer, and you can start provisioning Oracle Database@Azure directly from your Azure portal.

You must provide your Azure billing account ID to your Oracle sales representative to ensure single invoicing for the service from Azure. You can find your billing account ID through one of the following methods:

◉ Contact your Azure sales representative

Existing Azure and Oracle agreements are eligible for use with Oracle Database@Azure including MACCs, license included, or Bring Your Own License (BYOL). Database consumption from the service might also be eligible for Oracle Support Rewards. The Oracle databases you provision through the Oracle Database@Azure service apply to your existing Azure agreements just like other native Azure resources—no extra agreement required. If you aren’t an existing Azure customer, select Get Started on the Azure or Oracle webpage to create an account or contact an Azure Sales Representative. 

Onboarding


The following summary shows the onboarding process for Oracle Database@Azure with components in the Azure portal and the Oracle Cloud Console. Detailed explanations for many of the tasks are linked throughout this summary. Oracle and Azure specialists are available to address questions and provide any required support.

OCI tenancy

An OCI tenancy is required for billing orchestration, some database management tasks, such as provisioning and management of container databases, and for Oracle to provide infrastructure and software maintenance updates for your database service. You can create an OCI tenancy or link an existing tenancy to your Azure subscription. Click the corresponding link in the welcome email you received upon completing private offer acceptance.

Oracle Database@Azure now Generally Available in Azure East US Region to accelerate your data center exit

Identity federation

We recommend federating identities between Entra ID and OCI to reduce the overhead of managing two identity directories. This federation allows users to access their OCI account with an Azure login and helps enforce the same identity structure across both cloud providers. Oracle Database@Azure uses two sets of groups with specific policies: Groups in Azure and 

Groups in OCI. To set up federation, the Azure user must have the following Entra ID privileges:


To complete manual federation setup between Azure and OCI, follow the Identity Federation guide.

If you choose not to federate your OCI tenancy with Azure's identity service, you might need to create and maintain separate credentials for accessing the Oracle Cloud Console using the OCI Identity and Access Management (IAM) service.

Groups and roles

To ensure proper operation and permissions for the service, you must create groups, roles, and attached policies in Azure and OCI. Groups in Azure include infra-admins, virtual machine (VM) cluster admins, and others to administrate Exadata database resources and VM clusters in Azure. Other groups with attached policies are replicated in OCI during federation. Roles in Azure include database owner, and infra-amin, VM cluster admin, subscription manager, and database reader. Standard permissions apply, including read, write, delete, and action. Groups and policies are also created in OCI to manage database actions.

Provisioning


Provisioning Oracle Database@Azure resources is streamlined requiring a few simple steps in each portal, such as creating infrastructure and VM clusters in Azure and creating databases in the Oracle Cloud Console. Established workflows in the Azure portal and Oracle Cloud Console guide through each step. The summarize steps are included with links for readers seeking more details. Reach out to your Sales Representative if you want to chat with an engineering specialist.

Creating a resource

You can deploy resources directly from the Azure portal. Select Oracle Database@Azure from the Azure services on the home page, or search for Oracle Database@Azure in the search bar.

Oracle Database@Azure now Generally Available in Azure East US Region to accelerate your data center exit

As a prerequisite, the user must be part of the Exadata infrastructure administrators group in Azure and have sufficient limits assigned to the region.

On the Oracle Database@Azure overview page, you can access tutorials, create Oracle Exadata infrastructure, VM clusters, and more. Selecting Create cloud Exadata infrastructure opens the following flow for infrastructure creation:

  1. Basics: Select the Azure subscription, assign to a new or existing resource group, name the infrastructure, select a region, and select an availability zone.
  2. Configuration: The infrastructure model defaults to Exadata X9M. Customize the number of database servers (2–32) and storage servers (3–64) you need. OCPUs and storage (in TBs) automatically populate based on your selection.
  3. Maintenance: Set your maintenance method, schedule, and points of contact (up to 10).
  4. Consent: Agree to terms of use, privacy policy, and access permissions.
  5. Tags: Enter a name and value for your infrastructure resource.
  6. Review and create: Validate all information and create resource.

Provisioning an Exadata VM cluster

From the home page or Oracle Database@Azure page, you can quickly provision VM clusters.

Oracle Database@Azure now Generally Available in Azure East US Region to accelerate your data center exit

For prerequisites, ensure the appropriate permissions for VM cluster admins and Oracle Database@Azure, and create a virtual network (vNet) with two delegated subnets to Oracle Database@Azure. It requires the following permissions for an Oracle Database@Azure subscription:

  • Microsoft.Network/virtualNetworks/*
  • Microsoft.Network/virtualNetworks/subnets/*

On the Oracle Exadata VM Clusters tab of the Oracle Exadata Database@Azure page, click the Create link. Like Exadata infrastructure creation, the following multistep process displays:

  1. Basics: Select the Azure subscription, resource group (new or existing), name the cluster, select region, select infrastructure, license type, time zone, and set security with SSH keys and key pair name.
  2. Configuration: Select compute count, system shape, OCPU count per VM, memory per VM, and local storage per VM. Total OCPU count, requested memory, and local storage are computed based on the local values that you accept or select. Exadata sparse snapshots, and local backups are options that can only be set at this time before the Oracle Exadata VM cluster has been provisioned.
  3. Networking; Enter virtual network, client subnet, and DNS information.
  4. Diagnostics collection: Events, health monitoring, logs, and tracing are selected by default. Change if required.
  5. Consent: Agree to terms of use, privacy policy, and access permissions.
  6. Tags: Enter a name and Value for your infrastructure resource.
  7. Review and create: Validate all information and create resource.

Create a database

Database creation, deletion, and management is facilitated in the Oracle Cloud Console. Navigation through single sign-on (SSO) is provided in the Azure portal on the Resource page for the VM Cluster you want to connect. To proceed, on the VM Cluster overview page in the Azure portal, select “Go to OCI.”

Oracle Database@Azure now Generally Available in Azure East US Region to accelerate your data center exit

Oracle Database@Azure now Generally Available in Azure East US Region to accelerate your data center exit

As a prerequisite, you must set up Identity Federation with trust between Azure Entra ID and OCI identity domains and have groups created in Azure that are replicated in OCI.

When you click “Go to OCI” in the Azure portal, you arrive on a corresponding page in the Oracle Cloud Console. For example, if you were on the VM Cluster overview page in Azure, you’re not on the VM Cluster overview page in OCI. Using the Console or Database APIs, you can create database homes, container databases, and pluggable databases at any time.

Creating container databases requires entering basic information, such as name, Database service version (19c), creating a database home, admin credentials, and backup configuration. When you’re finished, select Complete, and your new database progresses from “Provisioning” to “Available.” An initial pluggable database is created at the time of container database creation, and extra pluggable databases are easy to provision by navigating to the Pluggable Database tab on the Container Database detail page and selecting Create.

Infrastructure as Code (IAC)

Oracle Database@Azure supports both GUI and CLI operations through PowerShell in Azure and Terraform in OCI. PowerShell is only applicable for resources within Azure. You can use Terraform in the following ways:

  • Native OCI service in Resource Manager: For operations in OCI, such as network operations in a virtual cloud network (VCN), container or pluggable database operations, and other permissions-based operations.
  • Standalone Terraform application: For running operations in Azure and OCI as two separate Terraform providers. Each provider requires its own permission set and operations.

Management


Management responsibility of Oracle Database@Azure is shared between Oracle, Microsoft, and the customer. The following list and chart describes each component of managing the service and ownership distribution between OCI, Microsoft, and the customer.

  • Oracle responsibilities: Exadata hardware, host infrastructure, Oracle Database@Azure API endpoint.
  • Azure responsibilities: Azure physical network, Azure SDN, Azure application infrastructure.
  • Oracle and Azure shared responsibilities: Data residency and support.
  • Customer responsibilities: The following table describes service components and location for task completion in the Azure portal and Oracle Cloud Console:

  Exadata
Infrastructure
VM Cluster
Management 
Database
Management 
Database
Maintenance 
Applications  Identity
Management 
Deployment  Azure Azure OCI OCI Azure Azure/OCI
Lifecycle  Azure  Azure  OCI  OCI  Azure  Azure/OCI 
Configuration  Azure  Azure  OCI  OCI  Azure  Azure/OCI 
Support          Azure  Azure/OCI 

Customer management activity in the Azure portal is facilitated from the Oracle Database@Azure service page, while activity in the Oracle Cloud Console is facilitated on the Oracle Exadata Database Service page.

Billing


A single invoice provided by Azure is viewable on the subscriptions page in the Azure portal. All familiar capabilities, including detailed marketplace invoices, cost analysis, and more, are available like your other Azure subscriptions. Detailed Oracle database consumption information is accessible in the Oracle Cloud Console.

Oracle Database@Azure now Generally Available in Azure East US Region to accelerate your data center exit

Monitoring


You can monitor Oracle Database@Azure from a single screen in the Console. Because Oracle manages infrastructure monitoring, telemetry exposure to customers isn’t required. VM cluster telemetry is fully available for customer consumption and is automatically integrated with Azure Monitor. Telemetry for Oracle databases will be integrated with Azure and available for complete customer monitoring in early 2024.

Oracle Database@Azure now Generally Available in Azure East US Region to accelerate your data center exit

Source: oracle.com

Friday, February 2, 2024

Extract key values with Oracle Analytics and OCI Document Understanding

Extract key values with Oracle Analytics and OCI Document Understanding
Example passport images used in Oracle Analytics to recognize text and expiration dates based on a pre-trained OCI AI model.

Oracle Analytics solutions now integrate with Oracle Cloud Infrastructure (OCI) Document Understanding!

OCI Document Understanding is an OCI AI service that enables developers to extract text, tables, and other key data from document files through APIs and command-line interface tools. With OCI Document Understanding, you can automate tedious business processing tasks with prebuilt AI models and customize document extraction to fit your industry-specific needs.

You can use pre-trained models for text extraction, table extraction, key value extraction, and document classification. Once you choose a pre-trained model, you can upload your images, then register the model in Oracle Analytics Cloud (OAC), and apply your model in a data flow to extract the key values from a resume, a passport, a receipt, or an invoice.

Here's how you can create a data visualization project based on OCI Document Understanding in 4 steps under 5 minutes. The steps are to:

1. Upload sample passport images to Oracle Cloud, into a private (non-public) bucket object storage.
2. Register the OCI Document Understanding model in Oracle Analytics Cloud.
3. Create a data flow to apply the AI model to example passport images.
4. Add the recognized key values to a Dataset and use this dataset in a workbook to visualize the data.
5. A prerequisite is to have an Oracle Analytics Cloud instance with a connection to OCI.

Step 1 – Create a bucket in OCI

1. Connect to OCI at this URL: https://www.oracle.com/cloud/sign-in.html
2. Click the menu option, then Storage, create a Bucket, and set a name.
3. Upload into the bucket all the document images needed to train and test your model.
4. Ensure the bucket is in the same tenancy as OAC. In this example, the bucket is “Bucket-vision-ai” that was previously created, and I uploaded 9 sample passport pictures under a folder named AID.

This step generates a location where OAC will be able to access uploaded images and apply the AI model.

Step 2 – Register your model in Oracle Analytics Cloud

1. On the OAC Home Page, click the 3 little dots menu ("...", or the ellipsis) at the top right corner. 
2. Select “Register Model/Function” and select “OCI Document Understanding Models”.
3. Once selected, choose your OCI connection. If it does not exist, you will need to create a new connection in OAC (Create > Connection > OCI Resource).
4. The window “Select a Model” will pop up. Select the model type “Pretrained Document Key Value Extraction”.
5. In the right-side panel, select your OCI Bucket and select the document type. In this example, it's “Passport”.

Extract key values with Oracle Analytics and OCI Document Understanding
Example of a Selection of Pre-trained Document Key Value Extraction AI models in OAC.

Step 3 – Apply the AI model to your images

1. Create a new data flow in OAC.
2. Create a dataset using a CSV file including your Bucket URL.
3. Add the Dataset to the data flow.
4. Add a step “Apply AI Model” to apply your pre-trained AI model to the images. Select the bucket URL in the Parameters, or File Location if you use a dataset showing images as a line item.
5. Select “Documents” as Input Type if you use itemized images or “Buckets” if you use your bucket URL.
6. Add a step to save the data in a new dataset. The saved data should contain the image name and URLs as well as the key values text/number extracted.

The data flow will load the images, analyze them with the pre-trained AI model, and extract the key values from the images. In this example, the passport documents. Then the data flow will load all the information into a dataset that you can use to explore and visualize the data.

Extract key values with Oracle Analytics and OCI Document Understanding
This screenshot shows an example of the data flow with the results generated by the AI model.

Step 4 – Visualize the results in Oracle Analytics.

1. Create a new Workbook in OAC.
2. Add the new dataset generated by the data flow.
3. Add an Image plugin visualization object to see all the images in your bucket.
4. Use this Image object as a filter by clicking the top left corner filter icon.
5. Create a new table to show all columns of your dataset.
6. Click one of the passport images to see the key value extracted in the table.
7. You can now use this workbook as a starting point to create additional calculation metrics to filter and analyze your documents. In this example, I decided to analyze the expiry date and create a conditional formatting donut to show if it’s expired (Red) or passed (Green).

Extract key values with Oracle Analytics and OCI Document Understanding
Example of the OAC workbook using the images plugin, OCI Document Understanding, and conditional formatting on calculation metrics.

Customers are using OCI Document Understanding to recognize multiple types of documents at scale, from passports to invoices to receipts and resumes. Oracle Analytics allows you to apply and visualize the data in a matter of minutes, helping you go quickly from data to insights, actions, and decisions.

Source: oracle.com

Wednesday, January 31, 2024

Dashboards: Dead or Alive? The evolution from data graveyards into data gold mines

Recently I’ve seen the subject of analytics dashboards reaching the end of their usefulness raised on more than one occasion, whether in a live in-person event or in a social post. But is this really the case? Are dashboards indeed an outdated form of presenting data to stakeholders and making decisions?

Are dashboards really dead?

What are traditional BI dashboards?


Classic dashboards have been around for decades, intended to empower business users with data-driven insights without the need for technical skills. When I began my analytics career nearly 20 years ago, "dashboards" were already prevalent. They served to simplify data access, letting executives explore key metrics without knowledge of the underlying data architecture.

Building those early dashboards was no simple task; it required deep expertise across data visualization tools, storage systems, and accessing and transforming data into business concepts. While the creation was complex, the consumption was simplified. Users could understand and action insights.

Traditional dashboards removed the complexity of data for businesspeople. But creating them still demanded technical specialization. These static, legacy dashboards remain common today despite lacking modern flexibility and intelligence. The next generation of analytics must balance technical depth with intuitive business exploration. Insights for all, complexity for none.

Dashboards: Dead or Alive? The evolution from data graveyards into data gold mines
Figure 1: A traditional dashboard example in Oracle Business Intelligence Enterprise Edition

What's the problem with traditional BI dashboards?


Legacy dashboards often rely on dated technologies and methods, lacking capabilities such as natural language, ML predictions, and built-in augmented data preparation. Developers are resistant to adopt (or adapt to) emerging innovations. The old "if it ain't broke, don't fix it" mentality has many organizations still using old approaches that lag behind current best practices.

For example, traditional dashboards struggled with real-time analytics, taking minutes or hours to refresh. Interactivity was limited too. Some allowed basic filtering or drill-down within preset data sources, but venturing outside the developer's predefined lanes was impossible.

When people say things such as “dashboards are where good data goes to die,” they're typically referring to the traditional type of dashboards. The data presented on these dashboards often becomes stagnant and isn't actively used for decision-making, gradually diminishing its value over time. These dashboards are generally constructed and maintained centrally by IT. Making changes can be challenging, and the process of creating dashboards is often rigorous and time-consuming. As business needs evolved, many dashboards were created but rarely removed. Consequently, organizations have been amassing huge repositories of thousands of inactive dashboards. Business users perceive little value in them, often turning to alternate visualization tools or even resorting back to spreadsheet-based solutions instead. Sound familiar?

This type of traditional dashboard is what Gartner refers to as “mode 1” business intelligence (BI). BI that was created centrally by a dedicated team, posted to a repository for business users and left there for the rest of eternity.

How are modern dashboards different?


A modern analytics dashboard (which leverages cloud technology) offers enhanced accessibility, real-time interactivity, user-accessible data integration, and collaborative capabilities that distinguish it from the older style of IT-led BI dashboards prevalent in legacy on-premises tools. Modern dashboards excel on two fronts compared to traditional, IT-led approaches. First, embedded AI/ML augments human analysis with machine intelligence. Second, self-service analytics empowers business users with autonomy.

AI and ML boost productivity by automating tedious data preparation and enrichment tasks. They also detect hidden insights such as anomalies, unseen patterns, and future predictions. This enhances data stories with machine learning, minus the manual effort.

However, technology alone isn't the answer. True enablement comes by putting analytics directly into the hands of business users by fostering a culture of data literacy and by providing intuitive tools for self-service exploration and decision-making. Self-service removes reliance on IT or specialists to build reports. Now cross-functional users can freely access, analyze, and share data on their own. With IT empowering through governance, not limiting through control, the possibilities are endless. People ask their own questions, find their own answers, and tell impactful data stories that positively affect the business without bottlenecks.

The future of dashboards is democratization, not restriction. Blending embedded AI/ML with self-service exploration makes modern dashboards a portal into limitless insights.

How has Oracle Analytics evolved?


Previous versions of Oracle Analytics (such as OBIEE) were designed for centralized, IT-controlled business intelligence. The semantic model simplified access to complex data architectures, transforming them into business metrics.  But there were drawbacks, such as IT becoming a bottleneck trying to keep up with business requests. Now with the option to upgrade to Oracle Analytics Server (OAS) on-premises or hosted in cloud, OBIEE customers gain the opportunity to give businesspeople the modern analytics tools and interfaces that their users demand while still retaining their investment in the corporate semantic model.

Today's Oracle Analytics platform (both Oracle Analytics Cloud and Oracle Analytics Server) has expanded into self-service for business users. Now cross-functional teams can easily obtain insights without relying on specialists. AI and ML augment human analysis with recommendations and productivity enhancers such as generative AI chat interfaces.

Dashboards: Dead or Alive? The evolution from data graveyards into data gold mines
Figure 2: Oracle Analytics platform YouTube product tour

IT retains governance for compliance and reliability. But the focus has shifted from centralized reporting to decentralized self-sufficiency. User autonomy, not technical complexity, now sits at the core. By combining governance with flexibility, Oracle enables actionable analytics. People ask their own questions and find answers quickly. Backed by machine learning, human intelligence scales further. Oracle Analytics allows organizations to become truly insights-driven by making analytics accessible, intelligent, and empower users to be self-sufficient.

Dashboards: Dead or Alive? The evolution from data graveyards into data gold mines
Figure 3: Oracle Analtyics Cloud with live interactive data visualizations embedded into a slide. Live visualizations allow people to focus on the results and making the right decision.

Call to action


Sure, there are most certainly many examples of BI or analytics projects that still have traditional style centrally created static dashboards where data indeed goes to die. But hopefully it’s clearer that all dashboards aren’t created equal, and the same fate doesn't apply to modern dashboards that are designed for self-service and that use AI/ML throughout.

For clarity, data doesn’t die and dashboards aren't dead. But traditional mode 1 style BI can stagnate data, making it less useful and untrustworthy. Business users and executive stakeholders that aren’t savvy enough to connect to and retrieve the data that they need will always rely on some form of dashboard. In the absence of a good solution, they'll resort to spreadsheets – and no one wants that.

My recommendation is to stop building old-school centralized dashboards (also known as mode 1) that are used briefly and then forgotten. Instead empower users with self-service and modern technologies (such as AI/ML) to answer their own questions, tell their own stories, and make better more analytics-driven decisions (also known as mode 2).

Source: oracle.com

Monday, January 29, 2024

New AI capabilities with Oracle Analytics

Oracle Analytics delivers ongoing innovation through the year, and offers a wide range of AI capabilities. It provides users capabilities augmented with AI, Machine Learning, and Data Science, as well as leveraging Oracle Cloud Infrastructure (OCI) integrated AI Services.

Oracle Analytics offers 3 types of AI capabilities:

1. Built-in, AI-augmented capabilities.
2. Integrated ML capabilities.
3. Integrated AI capabilities.

New AI capabilities with Oracle Analytics
AI capabilities of Oracle Analytics - Author: Philippe Lions, VP Product Management, Oracle.
 
Here's a list to help you discover some of the key AI features for Oracle Analytics:

Built-in AI Augmented capabilities

AI Auto-Insights: This feature generates data visualizations automatically from a specific dataset. It will create an optimal visualization for the data elements selected.

New AI capabilities with Oracle Analytics
AI Auto Insights, Explain and Segmentation

Explain: This feature uses Machine Learning to automatically explain a metric or attribute - like “Attrition” in the example below - with a single click. It calculates the correlations, drivers, segmentations and anomalies for the dataset.

New AI capabilities with Oracle Analytics

1-Click ML/AI feature: this feature enables you to create statistics and ML-generated insights, like clustering with ML or forecasting, using algorithms like ARIMA or ETS with a single click.

New AI capabilities with Oracle Analytics

ML Integrated Capabilities

Drag and Drop ML and Data Science: this feature allows you to train models using numeric prediction, multi-classifier, binary classifier, or clustering with different built-in algorithms. These ML algorithms can be customized, trained, tuned, and then published to the wider analytics user community.

New AI capabilities with Oracle Analytics
ML Data Flow with Prediction by Low Code No Code
 
Integrated AI capabilities

Oracle Cloud offers multiple AI services: https://www.oracle.com/artificial-intelligence/ai-services/

Currently, Oracle Analytics is integrated with 3 of them and more are planned in our Product Roadmap. Integration means that you can create an AI service in OCI, register the AI model, and ingest the results into Oracle Analytics as a dataset to create visual data stories.

Below are some key OCI AI services currently integrated with Oracle Analytics:

1. AI Vision
2. AI Language
3. AI Document Understanding

Below you can review these AI services and discover data visualizations built using them.

AI Vision Integration

AI Vision provides Object Detection and Image Classification. There are live demos for each in the public OAC instance.

A blog and demo showing object detection is available here:

New AI capabilities with Oracle Analytics
Data Visualization showcasing the use of an OCI AI Vision - Author: Benjamin Arnulf

A blog and demo showing image classification is available here:

New AI capabilities with Oracle Analytics
Data Visualization showcasing the use of an OCI AI Image Classification trained model with 3,000 public MRI - Author: Benjamin Arnulf

AI language Integration

Oracle Analytics has the capability to analyze language using AI:    

New AI capabilities with Oracle Analytics
Oracle Analytics data disualization showing the use of an OCI AI Language with a pre-trained model - Author: Philippe Lions

AI Document Understanding

Oracle Analytics is integrated with OCI AI Document Understanding and can recognize and extract values from a passport, receipts, resumes (aka CVs), invoices and more. See a blog and demo below:

New AI capabilities with Oracle Analytics
Oracle Analytics Data Visualization showing the use of OCI AI Document Understanding with a pre-trained model for key value extraction. Authors: Benjamin Arnulf and Philippe Lions

AI Digital Assistant

This will be integrated into Oracle Analytics, as announced at the keynote of T.K. Anand at OCW23. In summary, you will be able to ask questions like “What is the root cause for attrition and the trend for the past 3 quarters?”, and the integration will leverage Cohere or other Large Language Models (LLM) to create an answer and generate relevant data visualizations. 

New AI capabilities with Oracle Analytics
Oracle Analytics and the OCI AI Digital Assistant using large language models to generate answers and data visualizations.

Source: oracle.com