Unlock the Power of Your Data: Seamlessly Create and Refresh Datasets from Files Stored in OCI Object Storage

In this post, I provide an overview and examples of creating datasets from files stored in Oracle Cloud Infrastructure (OCI) Object Storage. I also explore the new capabilities with centralized file storage that enables the ability to schedule periodic reloads of the data stored in Object Storage. I review the creation of an OCI Resource connection including the creation of a dataset using the new connection, using the new UI to search and navigate the compartments, buckets, and objects, and creating datasets based on some files. Finally, I describe how you can manually reload or schedule a periodic reload of the dataset by updating files in Object Storage.

What Is OCI Object Storage?

OCI Object Storage enables you to securely store any type of data in its native format. With built-in redundancy, OCI Object Storage is ideal for building modern applications that require scale and flexibility, because it can be used to consolidate multiple data sources for analytics, backup, or archive purposes.

Creating an OCI Resource Connection

In order to access files stored in OCI Object Storage, you first create an OCI Resource Connection using an API Key. This connection is the same type of connection required for connecting Oracle Analytics to OCI functions and OCI models such as Vision and Language.

Creating Datasets from Files in OCI Object Storage

Once a connection has been successfully created, you can start the process of creating datasets from files in the OCI Object Storage buckets. Start the process by creating a dataset by clicking Create Dataset from the home page. Notice that the OCI Resource Connection is displayed as one of the data sources in the Create Dataset dialog.

Region Selection

After selecting the OCI Connection, use the dialog to change the default region if necessary, and to easily search for the right compartments, buckets, and objects that could include folders, subfolders, and files. Review the default region and change if necessary with the drop-down list.

Navigating and Searching Compartments

After selecting or keeping the default region, either manually navigate or enter a full or partial search string to search all the compartments. The search results are filtered to display only those compartments that meet the search criteria. The search is a wildcard, case-insensitive search.

Navigating and Searching Buckets

After clicking the correct compartment where the buckets reside, do the same type of navigation or wildcard search on the buckets. After locating the bucket where the files reside, click it and notice that all the objects in the selected bucket are displayed on the right-hand panel.

Navigating and Searching Objects and Selecting a File

Again, manually navigate the objects in the bucket, which can consist of folders, subfolders, and files, or perform a wildcard case-insensitive search. After locating the file to import into the dataset, click OK. The system imports the file into OAC and provides a preview of the contents of the file for review. After the review, click OK to bring the file into the Dataset Editor, where a representative sample is extracted and the deep semantic profile is triggered and the results are displayed in the form of the Data Quality Insights for the contents of the file.

Adding Another File from the Same OCI Connection

After creating the table based on the file from the OCI Object Storage bucket, notice that the connection (My OCI Connection), the resource (OCI Object Storage), and the imported file are listed in the left-hand panel. To add another file from the same connection, click the icon to the right of the resource. After clicking that icon, the navigation dialog is displayed again, and you can drill into the bucket again to get the second file. Add as many files as you need and join them to create the dataset. You can also join files from OCI Object Storage with database tables and other files.

Extract Credit – Scheduling a Dataset Reload

One of the really cool advantages of creating datasets from files in OCI Object Storage buckets is that you can create a recurring workflow where an upstream process can place updated files of the same name in the same bucket periodically. You can then schedule dataset refreshes to automatically update the data in cache from those updated files. You can set up either a one-time or a recurring schedule. Additionally, you can check the details of a schedule to see the last run time and the next scheduled run. This process and capability provides a way to update visualizations with the latest data from datasets based on files.

Call to Action

We hope you've enjoyed this overview of creating datasets from files in OCI Object Storage buckets! And we challenge you to start creating datasets from your files stored in buckets and hope that you find them to be both powerful and user-friendly. Keep exploring the powerful world of self-service data modeling and stay tuned for our upcoming blog posts, where we'll share more tips and tricks on both new and existing features of our product.

Source: oracle.com

Continue reading

Announcing New Spatial Machine Learning Algorithms in OML4Py on Autonomous Database Serverless

We are pleased to announce the general availability of a new feature of Oracle Machine Learning for Python (OML4Py), Spatial AI. OML4Py Spatial AI provides specialized machine learning algorithms that incorporate the effects of location. Using machine learning with spatial algorithms can improve model quality and prediction accuracy by accounting for the effects of location. For example, spatial regression algorithms are able to enhance home value predictions by incorporating the influence of neighboring home values. Spatial algorithms also allow you to detect location patterns, like spatial clustering of traffic accidents. As part of OML4Py on Autonomous Database Serverless, Spatial AI provides  a single environment for spatial ML workflows that minimizes data movement to external ML engines, simplifies your architecture, and accelerates time to value.

Sample notebooks are provided to help get you started. From the Oracle Machine Learning user interface, navigate to Templates > Examples and filter for "spatial". Clicking on the title of a sample notebook opens it in read-only mode to review the content. To create your own editable/runnable copy of a sample notebook, first select the sample notebook by clicking on the tile, click "Create Notebook", and then select your project. Begin with the notebook "OML4Py Spatial AI Run Me First" to seed sample data, and then try other notebooks based on your areas of interest.

After the data are seeded, sample notebooks can be run in any order. For example, in the sample notebook OML4Py Spatial AI Agglomerative Clustering and Regionalization, you apply a spatial ML algorithm that combines sets of Census Block Groups into broad regions based on similar demographics. Such regions are useful for regional marketing, where strategies are adjusted based localized demographics and associated buying patterns. The following image shows the results of a non-spatial clustering based solely on similar feature values (left), and results of clustering with a spatial ML algorithm that combines areas into regions based on similarity of both feature values and location.

Source: oracle.com

Continue reading

EMPTY STRING ON NULL for JSON Generation in Oracle Database 23ai

Oracle database 23ai has introduced the ability to convert nulls to empty strings during JSON generation.

◉ Setup

We create and populate a table. Notice each row has a different combination of values with regards to null.

conn testuser1/testuser1@//localhost:1521/freepdb1

drop table if exists t1 purge;

create table t1 (

  col1 VARCHAR2(10),

  col2 VARCHAR2(10)

);

insert into t1 values ('ONE', 'TWO');

insert into t1 values ('ONE', null);

insert into t1 values (null, 'TWO');

insert into t1 values (null, null);

commit;

select rownum, col1, col2 from t1;

 ROWNUM COL1 COL2

----------      --------- ----------

         1          ONE       TWO

         2         ONE

         3                         TWO

         4

SQL>

◉ NULL ON NULL and ABSENT ON NULL

In previous releases our only options for null handling during JSON generation were NULL ON NULL and ABSENT ON NULL. We'll use JSON_OBJECT as an example, where the default behaviour is NULL ON NULL.

When using NULL ON NULL, implicitly or explictly, any null values in the data are generated as "null" in the output.

-- Implicit - NULL ON NULL.

select json_object(*) as data from t1;

DATA

--------------------------------------------------------------------------------

{"COL1":"ONE","COL2":"TWO"}

{"COL1":"ONE","COL2":null}

{"COL1":null,"COL2":"TWO"}

{"COL1":null,"COL2":null}

SQL>

-- Explicit - NULL ON NULL.

select json_object(* null on null) as data from t1;

DATA

--------------------------------------------------------------------------------

{"COL1":"ONE","COL2":"TWO"}

{"COL1":"ONE","COL2":null}

{"COL1":null,"COL2":"TWO"}

{"COL1":null,"COL2":null}

SQL>

When using ABSENT ON NULL any null values in the data result in the corresponding element being removed from the output.

select json_object(* absent on null) as data from t1;

DATA

--------------------------------------------------------------------------------

{"COL1":"ONE","COL2":"TWO"}

{"COL1":"ONE"}

{"COL2":"TWO"}

{}

SQL>

Here's an example with JSON_ARRAY, which uses ABSENT ON NULL by default.

-- Default - ABSENT ON NULL

select json_array(col1) as data from t1;

DATA

--------------------------------------------------------------------------------

["ONE"]

["ONE"]

[]

[]

SQL>

-- Explicit - ABSENT ON NULL

select json_array(col1 absent on null) as data from t1;

DATA

--------------------------------------------------------------------------------

["ONE"]

["ONE"]

[]

[]

SQL>

-- NULL ON NULL

select json_array(col1 null on null) as data from t1;

DATA

--------------------------------------------------------------------------------

["ONE"]

["ONE"]

[null]

[null]

SQL>

◉ EMPTY STRING ON NULL

In Oracle database 23ai we now have the option of using EMPTY STRING ON NULL. In this case null values are presented as empty strings in the output.

Here we see JSON_OBJECT being used with EMPTY STRING ON NULL.

select json_object(* empty string on null) as data from t1;

DATA

--------------------------------------------------------------------------------

{"COL1":"ONE","COL2":"TWO"}

{"COL1":"ONE","COL2":""}

{"COL1":"","COL2":"TWO"}

{"COL1":"","COL2":""}

SQL>

Here is an example of JSON_ARRAY using EMPTY STRING ON NULL.

select json_array(col1 empty string on null) as data from t1;

DATA

--------------------------------------------------------------------------------

["ONE"]

["ONE"]

[""]

[""]

SQL>

◉ Thoughts

In Oracle SQL and PL/SQL an empty string and a null are treated the same. This is not the case in other languages, so it's important to give people an option of presenting nulls as empty strings if required.

The default null handling of each JSON generation function can differ, so it's important to check the documentation, or always be explicit so other developers know your intentions.

Source: oracle-base.com

Continue reading

Announcing Globally Distributed Autonomous Database Support for Oracle Database 23ai Enabling Raft Replication

Over the last four years, the Oracle Database Development team has diligently worked to deliver the next long-term support release: Oracle Database 23ai, with a focus on AI and enhancing developer productivity.

Today, we are thrilled to announce support for Oracle Database 23ai on the Oracle Globally Distributed Autonomous Database!. Oracle Database 23ai, in conjunction with the Oracle Globally Distributed Autonomous Database, is a game-changer for enterprises worldwide seeking to build cloud-scale distributed databases with ultra-high scalability, availability, and advanced AI capabilities.

With the latest release, Oracle Globally Distributed Autonomous Database allows customers to select their preferred replication strategy to ensure high availability and disaster recovery. You can choose between the new built-in Raft replication (introduced in 23ai) and Oracle Data Guard (available in releases 19c and 23ai). The service automatically deploys the specified replication topology to the configured systems and enables seamless data replication. If you are used to NoSQL databases and do not expect to know anything about how replication works, Globally Distributed Autonomous Database built-in Raft replication just works.

Key benefits for Raft replication include:

Fast failover with zero data loss: Raft replication is built into Globally Distributed Autonomous Database to provide a consensus-based, high-performance, low-overhead availability solution, with distributed replicas and fast failover with zero data loss, while automatically maintaining the replication factor if shards fail.

Active-Active symmetric configuration: Each node accepts writes and reads for a subset of data facilitating an Active-Active symmetric distributed database architecture that enhances availability, and optimizes resource utilization globally.

Simplified management: Raft replication is integrated and transparent to applications without requiring configuration of Oracle GoldenGate or Oracle Data Guard. Raft replication automatically reconfigures replication in case of shard host failures or when shards are added or removed from the sharded database.

Automatic data distribution: The service handles workload distribution and balancing upon scaling (addition or removal of shards), including planned or unplanned changes in shard availability. 

By supporting Oracle Database release 23ai, Globally Distributed Autonomous Database offers customers enhanced replication options and advanced AI capabilities including Select AI and AI Vector Search. It provides all of the new benefits on top of 23ai including unparalleled flexibility, improved operational efficiency, enhanced performance, simplified scalability, and compliance for modern applications. Stay tuned for more updates and detailed guides on leveraging these new capabilities to maximize your database’s potential.

Source: oracle.com

Continue reading

Oracle Spatial Studio 24.1 Available Now

We’re pleased to announce the release of Oracle Spatial Studio 24.1, a no-code web tool for working with the spatial features of Oracle Database. In this release we've focused on enchancing existing features and including new support for a popular file type, 3D tiles. This blog post will explore some of the enhancements available in this release.

Enhanced embedding of published Studio projects

Spatial Studio provides functionality to embed published maps into other web applications. New capabilities enable interactive behavior between your application and an embedded map using an enhanced API. For an application built on a platform without native map visualization features, embedding Spatial Studio maps is an easy alternative to custom coding. But even for an application built on a platform having map features, Spatial Studio provides advantages, for example high performance in map rendering of large volumes. The image below is an elementary example of a Spatial Studio map embedded into an Oracle APEX application, allowing the user to select items in the map to highlight corresponding records in the native APEX table, and vice versa.

Enhanced Cesium 3D visualizations to support updated 3D Tiles specification

3D Tiles is a popular file format for streaming massive heterogeneous 3D geospatial datasets, allowing for efficient interaction at runtime. The Open Geospatial Consortium updated the 3D Tiles specification in 2023, and subsequently, Cesium has included support for the features in this standard. Spatial Studio has been enhanced to include these features.

Support for incremental Geocoding

Spatial Studio provides the ability to geocode data, whereby addresses are converted to point locations for mapping and spatial analysis. For scenarios where your data are continually or periodically appended with new addresses, Spatial Studio has added the ability to perform “incremental geocoding”. This means that only new addresses are geocoded, regardless of the volume of previously geocoded addresses. This ensures less rework, saving resources, and improving efficiency.

Source: oracle.com

Continue reading

OCI AI Vision Facial Detection in Oracle Analytics Cloud

This post explores the OCI service AI Vision Facial Detection exposed directly in Oracle Analytics. With prebuilt models available, developers can easily build image recognition and text recognition into their applications without machine learning (ML) expertise.  This post will walk users through registering their AI Vision Face Detection model, uploading images, running their dataflow, and analyzing the output. 

Users can analyze single images or a batch of images directly in OAC using the pre-trained face detection model. OCI AI Vision Facial Detection feature provides the following information:

1. Identifies the existence of faces in each image.

2. The location of faces in each image using bounding box coordinates.

3. Facial landmarks for each detected face, including left eye, right eye, nose tip, and left and right edges of mouth.

4. Visual quality of each face: a higher score indicates that the image of the face is more likely to be suitable for biometrics.

Register your OCI AI Vision Face Detection Model

◉ In the OCI AI Vision service, register your Face Detection Model.

◉ Select the connection to your OCI tenancy.

◉ Choose the desired compartment.

◉ Select the face detection model and provide a staging bucket.

Uploading and preparing your image files

To start, first upload images to an OCI bucket.  Once all images reside in a OCI bucket, there are two options to build the required data input file that will be used in your data flow.

1. The first option is to create a CSV/xls file that points directly to all the images you wish to use with the face detection model (see left side of screenshot).

2. The second option option is to create a CSV/xls that points to the root of the bucket where all images reside.  In this scenario, the model will consume all images within the bucket (see right side of screenshot).

3. Once the input data file is created, you can proceed directly to OAC to build your dataflow and run the face detection model.

Executing the Face Detection model

Dataset Prep

1. To this point, the following steps have been completed: registering the face detection model, uploading images to an OCI bucket, and creating an input data file.

2. With all these components in place, it's time to build the dataflow and execute the model. First, you need to upload the input data file created in the previous step.

3. Follow the screenshot flow below to upload and save the data file in OAC.  This example shows an input file that points directly to each individual image file.  Once the data file is saved, you can build your dataflow.

Build the Dataflow.

3. To build the dataflow, start with your newly created input dataset and add the AI Face Detection model.

4. With the model added to the data flow, you'll see all the output columns that will be produced by the Face Detection Model.

• In the parameters sections, choose the input column (the source column of your images within your data file).
• For the input type selection, specify from the dropdown to indicate whether your input dataset uses images or images sourced directly from a bucket.

5. The last step is to save the model output in a data file.

• Add a step to your dataflow, 'save data'.
• Once you provided a dataset name, click run, and provide a dataflow name.
• As part of the save data process, you'll see an output preview of the Face Detection model. Note that depending on the number of images in your model, execution times will vary (currently OAC supports up to 250 face detections per image).

Visualize in OAC

With the data flow successfully executed, you can now create a new workbook in OAC and visualize the output. Before starting, you need to first download the vision series plugin from the OAC library and upload it into your OAC deployment, as this is the visualization type you'll use with the Face Detection Model.

Build your OAC Workbook

1. Start by adding your model output dataset to the workbook and selecting the Vision Plugin.

2. Then, using facial landmarks, you can now identify the exact coordinates and features of each within your images.

3. In addition, the model provides a quality and confidence score.

4. In this sample, the use of dashboard filters allows for a quick review of all images that were put through the model.

AI Explain

In addition, through the use of AI Explain, you can gain valuable insights into the model where all images are analyzed and reported on together.

Source: oracle.com

Continue reading