Saturday, July 18, 2020

File Handling From PL/SQL

Oracle Database Tutorial and Materials, Oracle Database Study Materials, Oracle Database Exam Prep

Using a Java stored procedure it is possible to manipulate operating system files from PL/SQL.

◉ Create the Java Stored Procedure


First we need to create the Java class to perform all file manipulation using the Java File Class.

CREATE OR REPLACE AND COMPILE JAVA SOURCE NAMED "FileHandler" AS
import java.lang.*;
import java.util.*;
import java.io.*;
import java.sql.Timestamp;

public class FileHandler
{
  private static int SUCCESS = 1;
  private static  int FAILURE = 0;
 
  public static int canRead (String path) {
    File myFile = new File (path);
    if (myFile.canRead()) return SUCCESS; else return FAILURE;
  }

  public static int canWrite (String path) {
    File myFile = new File (path);
    if (myFile.canWrite()) return SUCCESS; else return FAILURE;
  }

  public static int createNewFile (String path) throws IOException {
    File myFile = new File (path);
    if (myFile.createNewFile()) return SUCCESS; else return FAILURE;
  }

  public static int delete (String path) {
    File myFile = new File (path);
    if (myFile.delete()) return SUCCESS; else return FAILURE;
  }

  public static int exists (String path) {
    File myFile = new File (path);
    if (myFile.exists()) return SUCCESS; else return FAILURE;
  }

  public static int isDirectory (String path) {
    File myFile = new File (path);
    if (myFile.isDirectory()) return SUCCESS; else return FAILURE;
  }

  public static int isFile (String path) {
    File myFile = new File (path);
    if (myFile.isFile()) return SUCCESS; else return FAILURE;
  }

  public static int isHidden (String path) {
    File myFile = new File (path);
    if (myFile.isHidden()) return SUCCESS; else return FAILURE;
  }

  public static Timestamp lastModified (String path) {
    File myFile = new File (path);
    return new Timestamp(myFile.lastModified());
  }

  public static long length (String path) {
    File myFile = new File (path);
    return myFile.length();
  }
 
  public static String list (String path) {
    String list = "";
    File myFile = new File (path);
    String[] arrayList = myFile.list();
   
    Arrays.sort(arrayList, String.CASE_INSENSITIVE_ORDER);
   
    for (int i=0; i < arrayList.length; i++) {
      // Prevent directory listing expanding if we will blow VARCHAR2 limit.
      if ((list.length() + arrayList[i].length() + 1) > 32767)
        break;
       
      if (!list.equals(""))
        list += "," + arrayList[i];
      else
        list += arrayList[i];
    }
    return list;
  }

  public static int mkdir (String path) {
    File myFile = new File (path);
    if (myFile.mkdir()) return SUCCESS; else return FAILURE;
  }

  public static int mkdirs (String path) {
    File myFile = new File (path);
    if (myFile.mkdirs()) return SUCCESS; else return FAILURE;
  }

  public static int renameTo (String fromPath, String toPath) {
    File myFromFile = new File (fromPath);
    File myToFile   = new File (toPath);
    if (myFromFile.renameTo(myToFile)) return SUCCESS; else return FAILURE;
  }

  public static int setReadOnly (String path) {
    File myFile = new File (path);
    if (myFile.setReadOnly()) return SUCCESS; else return FAILURE;
  }

  public static int copy (String fromPath, String toPath) {
    try {
      File myFromFile = new File (fromPath);
      File myToFile   = new File (toPath);
 
      InputStream  in  = new FileInputStream(myFromFile);
      OutputStream out = new FileOutputStream(myToFile);
     
      byte[] buf = new byte[1024];
      int len;
      while ((len = in.read(buf)) > 0) {
        out.write(buf, 0, len);
      }
      in.close();
      out.close();
      return SUCCESS;
    }
    catch (Exception ex) {
      return FAILURE;
    }
  }
};
/
show errors java source "FileHandler"

◉ Publish the Java Call Specification


Next we publish the call specification using a PL/SQL "wrapper" package. Notice no package body is required since it only contains references to Java stored procedures.

CREATE OR REPLACE PACKAGE file_api AS

FUNCTION canRead (p_path  IN  VARCHAR2) RETURN NUMBER
AS LANGUAGE JAVA
NAME 'FileHandler.canRead (java.lang.String) return java.lang.int';

FUNCTION canWrite (p_path  IN  VARCHAR2) RETURN NUMBER
AS LANGUAGE JAVA
NAME 'FileHandler.canWrite (java.lang.String) return java.lang.int';

FUNCTION createNewFile (p_path  IN  VARCHAR2) RETURN NUMBER
AS LANGUAGE JAVA
NAME 'FileHandler.createNewFile (java.lang.String) return java.lang.int';

FUNCTION delete (p_path  IN  VARCHAR2) RETURN NUMBER
AS LANGUAGE JAVA
NAME 'FileHandler.delete (java.lang.String) return java.lang.int';

FUNCTION exists (p_path  IN  VARCHAR2) RETURN NUMBER
AS LANGUAGE JAVA
NAME 'FileHandler.exists (java.lang.String) return java.lang.int';

FUNCTION isDirectory (p_path  IN  VARCHAR2) RETURN NUMBER
AS LANGUAGE JAVA
NAME 'FileHandler.isDirectory (java.lang.String) return java.lang.int';

FUNCTION isFile (p_path  IN  VARCHAR2) RETURN NUMBER
AS LANGUAGE JAVA
NAME 'FileHandler.isFile (java.lang.String) return java.lang.int';

FUNCTION isHidden (p_path  IN  VARCHAR2) RETURN NUMBER
AS LANGUAGE JAVA
NAME 'FileHandler.isHidden (java.lang.String) return java.lang.int';

FUNCTION lastModified (p_path  IN  VARCHAR2) RETURN DATE
AS LANGUAGE JAVA
NAME 'FileHandler.lastModified (java.lang.String) return java.sql.Timestamp';

FUNCTION length (p_path  IN  VARCHAR2) RETURN NUMBER
AS LANGUAGE JAVA
NAME 'FileHandler.length (java.lang.String) return java.lang.long';

FUNCTION list (p_path  IN  VARCHAR2) RETURN VARCHAR2
AS LANGUAGE JAVA
NAME 'FileHandler.list (java.lang.String) return java.lang.String';

FUNCTION mkdir (p_path  IN  VARCHAR2) RETURN NUMBER
AS LANGUAGE JAVA
NAME 'FileHandler.mkdir (java.lang.String) return java.lang.int';

FUNCTION mkdirs (p_path  IN  VARCHAR2) RETURN NUMBER
AS LANGUAGE JAVA
NAME 'FileHandler.mkdirs (java.lang.String) return java.lang.int';

FUNCTION renameTo (p_from_path  IN  VARCHAR2,
                   p_to_path    IN  VARCHAR2) RETURN NUMBER
AS LANGUAGE JAVA
NAME 'FileHandler.renameTo (java.lang.String, java.lang.String) return java.lang.int';

FUNCTION setReadOnly (p_path  IN  VARCHAR2) RETURN NUMBER
AS LANGUAGE JAVA
NAME 'FileHandler.setReadOnly (java.lang.String) return java.lang.int';

FUNCTION copy (p_from_path  IN  VARCHAR2,
               p_to_path    IN  VARCHAR2) RETURN NUMBER
AS LANGUAGE JAVA
NAME 'FileHandler.copy (java.lang.String, java.lang.String) return java.lang.int';

END file_api;
/
SHOW ERRORS

◉ Grant Privileges to Give JServer Access to the Filesystem


In this example we are granting access to all directories on the server. That is really dangerous. You need to be more specific about these grants and/or be very careful about who you grant access to this functionality.

The relevant permissions must be granted from SYS for JServer to access the file system.

EXEC DBMS_JAVA.grant_permission('SCHEMA-NAME', 'java.io.FilePermission', '<<ALL FILES>>', 'read ,write, execute, delete');
EXEC DBMS_JAVA.grant_permission('SCHEMA-NAME', 'SYS:java.lang.RuntimePermission', 'writeFileDescriptor', '');
EXEC DBMS_JAVA.grant_permission('SCHEMA-NAME', 'SYS:java.lang.RuntimePermission', 'readFileDescriptor', '');
GRANT JAVAUSERPRIV TO SCHEMA-NAME;

The affects of the grant will not be noticed until the grantee reconnects. It is up to the individual to decide on the level of access that is required.

◉ Test It


Finally we call the FILE_API packaged functions from PL/SQL. An example of every syntax can be seen below.

SET SERVEROUTPUT ON
BEGIN
  DBMS_OUTPUT.PUT_LINE('canRead      : ' ||  FILE_API.canRead ('C:\temp\test1.txt'));
  DBMS_OUTPUT.PUT_LINE('canWrite     : ' ||  FILE_API.canWrite ('C:\temp\test1.txt'));
  DBMS_OUTPUT.PUT_LINE('createNewFile: ' ||  FILE_API.createNewFile ('C:\temp\test1.txt'));
  DBMS_OUTPUT.PUT_LINE('delete       : ' ||  FILE_API.delete ('C:\temp\test2.txt'));
  DBMS_OUTPUT.PUT_LINE('exists       : ' ||  FILE_API.exists ('C:\temp\test2.txt'));
  DBMS_OUTPUT.PUT_LINE('isDirectory  : ' ||  FILE_API.isDirectory ('C:\temp\test1.txt'));
  DBMS_OUTPUT.PUT_LINE('isFile       : ' ||  FILE_API.isFile ('C:\temp\test1.txt'));
  DBMS_OUTPUT.PUT_LINE('isHidden     : ' ||  FILE_API.isHidden ('C:\temp\test1.txt'));
  DBMS_OUTPUT.PUT_LINE('lastModified : ' ||  TO_CHAR(FILE_API.lastModified ('C:\temp\test1.txt'), 'DD-MON-YYYY HH24:MI:SS'));
  DBMS_OUTPUT.PUT_LINE('length       : ' ||  FILE_API.length ('C:\temp\test1.txt'));
  DBMS_OUTPUT.PUT_LINE('mkdir        : ' ||  FILE_API.mkdir ('C:\temp\dir1'));
  DBMS_OUTPUT.PUT_LINE('mkdirs       : ' ||  FILE_API.mkdirs ('C:\temp\dir2\dir3'));
  DBMS_OUTPUT.PUT_LINE('renameTo     : ' ||  FILE_API.renameTo ('C:\temp\test1.txt','C:\temp\test2.txt'));
  DBMS_OUTPUT.PUT_LINE('setReadOnly  : ' ||  FILE_API.setReadOnly ('C:\temp\test1.txt'));
  DBMS_OUTPUT.PUT_LINE('copy         : ' ||  FILE_API.copy ('C:\temp\test2.txt','C:\temp\test1.txt'));
END;
/

◉ List Files in a Directory


We can use the LIST function in the FILE_API package to list files and sub-directories in a directory. Notice the files are presented as a comma-separated list.

SET SERVEROUTPUT ON
BEGIN
  DBMS_OUTPUT.PUT_LINE('Output : ' ||  File_API.list ('/u01/app/oracle'));
END;
/
Output : admin,audit,cfgtoollogs,checkpoints,diag,product


PL/SQL procedure successfully completed.

SQL>

We can split the list into an array. There are a number of ways to do that, but this method uses the APEX_STRING package.

DECLARE
  l_array APEX_APPLICATION_GLOBAL.vc_arr2;
  l_string varchar2(32767);
BEGIN
  l_array:= APEX_STRING.string_to_table(File_API.list ('/u01/app/oracle'), ',');

  FOR i in 1..l_array.count LOOP
    DBMS_OUTPUT.put_line('Array(' || i || ') : ' || l_array(i));
  END LOOP;
END;
/
Array(1) : admin
Array(2) : audit
Array(3) : cfgtoollogs
Array(4) : checkpoints
Array(5) : diag
Array(6) : product


PL/SQL procedure successfully completed.

SQL>

Friday, July 17, 2020

PDB Point-in-Time Recovery and Flashback in Oracle 20c

Oracle Database Tutorial and Material, Database Certification, Database Learning, Oracle 20c

The most significant point about the Oracle 20c database architecture is that non-CDB Oracle Database upgrades to non-CDB architecture are desupported. Meaning you need a container database in 20c and your data will reside within a pluggable database.

But then how about if you need to restore one PDB to any time in the recent past?

In Oracle database 20c, flashback and PITR (=point-in-time recovery) are supported when recovering PDBs to an ancestor or orphan PDB incarnations. These operations were not possible in 19c and below. Just as a reminder, in Oracle 12.1 flashback database operations were possible on root container level and thus affected all PDBs under the root container. Oracle 12.2 started supporting flashback of a PDB.

There is one restriction though in 20c: you cannot perform PDB flashback or PITR operation to a PDB incarnation within an orphan database incarnation. In other words, you can flashback as long as the CDB incarnation does not change. Or restated: you can flashback a PDB to an orphan PDB incarnation that is either within the same CDB incarnation or in an ancestor CDB incarnation. Also, flashback of a PDB to an orphan incarnation is supported only when the database uses local undo.

Now, this might confusing. Let me first shortly explain what is an ancestor incarnation and an orphan incarnation.

Database incarnations have the following relationships to each other:

– The current incarnation is the one in which the database is currently operating
– The incarnation from which the current incarnation originated after an OPEN RESETLOGS operation is the parent incarnation of the current incarnation
– The parent of the parent incarnation is an ancestor incarnation and any parent of an ancestor incarnation is also an ancestor of the current incarnation
– A noncurrent incarnation that is not a direct ancestor of the current incarnation is called orphan incarnation

During the flashback of the PDB, Oracle modifies only the data files for that PDB. The data in the other PDBs is not impacted. The point in time for the flashback can be one of the following:

– System Change Number
– Specific time in the past
– CDB restore point
– PDB restore point
– PDB clean restore point
– PDB guaranteed restore point

Here is an example of how flashback to any time in the recent past works in Oracle 20c.

We have lost at 2:30pm a table called RDBMS_BRANDS and a materialized zone map RDBMS_ZMAP from a pluggable database called NOVOPDB2. We have a restore point called rp1_novo_pdb2 created before the “disaster” at 8am in the morning. So, let us first flashback and verify we get the 2 objects back:

SQL> select systimestamp from dual;

SYSTIMESTAMP
-------------------------------------------------------------
05-JUL-20 02.35.30.569344 PM +00:00

SQL> SELECT table_name FROM DBA_TABLES where TABLE_NAME like '%RDBMS%';

no rows selected

SQL> ALTER PLUGGABLE DATABASE novopdb2 CLOSE;

Pluggable database altered.

SQL> FLASHBACK PLUGGABLE DATABASE novopdb2 TO RESTORE POINT rp1_novo_pdb2;

Flashback complete.

SQL> ALTER PLUGGABLE DATABASE novopdb2 OPEN RESETLOGS;

Pluggable database altered.

SQL> SELECT table_name FROM DBA_TABLES where TABLE_NAME like '%RDBMS%';

TABLE_NAME
----------------------------------------------------------------------
RDBMS_BRANDS
RDBMS_ZMAP
 
Well, unfortunately, now we notice that slightly before noon time data was loaded into a new table called RDBMS_HISTORY which was not at 8am in the PDB. All SCNs between 8am and the current time are now on an orphan PDB incarnation. We will flahsback again using another restore point created at 12 o’clock.

SQL> ALTER PLUGGABLE DATABASE novopdb2 CLOSE;

Pluggable database altered.

SQL> FLASHBACK PLUGGABLE DATABASE novopdb2 TO RESTORE POINT rp2_novo_pdb2;

Flashback complete.

SQL> ALTER PLUGGABLE DATABASE novopdb2 OPEN RESETLOGS;

Pluggable database altered.

SQL> SELECT table_name FROM DBA_TABLES where TABLE_NAME like '%RDBMS%';

TABLE_NAME
--------------------------------------------------------------------------------
RDBMS_BRANDS
RDBMS_ZMAP
RDBMS_HISTORY

SQL> select systimestamp from dual;

SYSTIMESTAMP
---------------------------------------------------------------------------
05-JUL-20 02.48.30.569344 PM +00:00

So, we managed to flashback to a point few hours after we opened with RESETLOGS from the previous flashback. But within the same incarnation of the CDB.

DBAs can follow the process of restore and recovery using the V$SESSION_LONGOPS and V$RECOVERY_PROGRESS views, respectively.

For the restore, the V$SESSION_LONGOPS view’s column OPNAME should be ‘Flashback Database’. Just like this:

SELECT sofar, totalwork, units
FROM v$session_longops
WHERE opname = 'Flashback Database';

The column SOFAR shows the data currently read in megabytes while the column TOTALWORK shows the total number of megabytes of flashback logs that must be read.

Restore points are created with the following command:

SQL> CREATE RESTORE POINT rp17 FOR PLUGGABLE DATABASE novopdb2;

Restore point created.

Wednesday, July 15, 2020

A Look at the Oracle Group-by Bug

Database Tutorial and Material, Database Exam Prep, Database Certification

Oracle introduced a new feature, group by elimination, for queries where the group by column is also the table's unique key. As with many new features this one still has not had all the kinks resolved. The problem arises when key values are manipulated with function calls. The following example will illustrate the issue by using a table with a DATE as the primary key and by extracting the year is extracted using TO_CHAR or EXTRACT.

Read More: Database Certification

A table is created as follows:

create table bug_test_calendar(
        cal_name   char(17),
        bus_dt   date,
        updt_timestamp       timestamp (6) default systimestamp,
        constraint pk_bug_test_calendar
                        primary key (bus_dt)
)
/

insert into bug_test_calendar (bus_dt)
select
        sysdate + 10 * rownum
from
        all_objects
where
        rownum <= 40
/

commit;

When the query shown below is executed, it produces the following results:

select
        to_char(bus_dt,'YYYY') bus_dt, count(*) ct
from
       bug_test_calendar
group by
        to_char(bus_dt,'YYYY')
order by
        to_char(bus_dt,'YYYY')
/

BUS_DF   CT
-------  --
2020      1
2020      1
...
2020      1

40 rows returned

Database Tutorial and Material, Database Exam Prep, Database Certification
Oracle doesn't 'know' that the key values have been manipulated so that they are no longer unique, thus the optimizer applies the unique-key-based group-by elimination with less than stellar results,

EXTRACT fares no better, returning the same results. This behavior is controlled by the "_optimizer_aggr_groupby_elim" parameter, which is set to true by default. As it's a hidden parameter, its setting is not reported by Oracle in either of the V$PARAMEter or V$SPPARAMETER views. The work-around is to simply set this parameter to false. However, having it active might help other group-by queries where the unique key values are not manipulated.

Enter Oracle 19c, where this functionality is partially fixed:

select
        to_char(bus_dt,'YYYY') bus_dt, count(*) ct
from
       bug_test_calendar
group by
        to_char(bus_dt,'YYYY')
order by
        to_char(bus_dt,'YYYY')
/

BUS_DF   CT
-------  --
2020     40

Unfortunately EXTRACT is still broken in 19c:

select
        to_char(bus_dt,'YYYY') bus_dt, count(*) ct
from
       bug_test_calendar
group by
        extract(year deom bus_dt)
order by
        extract(year deom bus_dt)
/

BUS_DF   CT
-------  ==
2020      1
2020      1
...
2020      1

40 rows returned

Obviously given truly unique key values a group-by query would produce a count of 1 for each key. And, just as obvious, Oracle should be able to recognize when values are no longer unique and invoke the proper group-by mechanism. It remains to be seen if versions after 19c will fix the second condition and thus return correct results without having to turn off this feature.

This may not affect every installation of Oracle newer than 12.1, but it is worth knowing about should wrong results start appearing in selected group by queries.

Monday, July 13, 2020

Oracle Database 20c Automatic In-Memory Enhancements

In Oracle Database 20c the Database In-Memory feature Automatic In-Memory (AIM) has been significantly enhanced. I wrote about AIM when it first came out in Oracle Database 18c here. Oracle Database 20c adds a new HIGH option to the INMEMORY_AUTOMATIC_LEVEL initialization parameter. With this setting all objects that do not have a pre-existing INMEMORY setting are automatically set to INMEMORY MEMCOMPRESS AUTO by default. AIM then automatically manages objects populated into the In-Memory (IM) column store using access tracking and column statistics. This is a big change in behavior and addresses one of the most frequent questions that customers have had, which is "How do I determine which objects to populate into the IM column store?"

Oracle Database 20c, Oracle Database Certifications, DB Exam Prep, Database Learning

The previous parameter options of LOW and MEDIUM still exist and function the same as they did when introduced in Oracle Database 18c. However, with the new HIGH option in Oracle Database 20c the database automatically manages the contents of the IM column store. It monitors segment activity using an access tracking and column statistic infrastructure similar to Heat Map data which was introduced as part of Automatic Data Optimization (ADO). With AIM set to HIGH segments are automatically evicted and populated based on usage. You do not have to pick and choose which objects to enable for in-memory. In addition, individual columns may be automatically compressed as well by AIM. All of this has been done to make the most optimal use of the IM column store and provide the best performance possible, automatically!

When the INMEMORY_AUTOMATIC_LEVEL initialization parameter has been set to HIGH all objects that do not have a pre-existing INMEMORY setting are automatically set to INMEMORY MEMCOMPRESS AUTO. This is a new option of the INMEMORY MEMCOMPRESS subclause in 20c that is part of the AIM feature. If you do not want specific objects to be populated in the IM column store then you can still manually set them to NO INMEMORY. Also, segments marked with an INMEMORY PRIORITY setting other than NONE are excluded from automatic eviction.

If you decide that you no longer want to use the HIGH setting of the INMEMORY_AUTOMATIC_LEVEL parameter then when you change the parameter value or unset it all segments with MEMCOMPRESS AUTO will be set to NO INMEMORY.

One other interesting feature of AIM is that you can adjust the window that AIM considers for segment usage. In other words, AIM can be adjusted to only consider the object usage statistics that coincide with your active workload window. That way those statistics won't be skewed by periods of inactivity, other application usage or perhaps even maintenance activities.

How do you tell what AIM has done? AIM runs tasks, similar to ADO, and those tasks are exposed in two data dictionary views. The views DBA_INMEMORY_AIMTASKS and DBA_INMEMORY_AIMTASKDETAILS can be queried to see what actions AIM has performed. Of course you can still query the contents of the IM column store using the view v$im_segments and see how much memory has been used using the view v$inmemory_area.

Saturday, July 11, 2020

What is a Converged Database?

Oracle Database Tutorial and Material, Oracle Database Learning, Oracle Database Certification, Database Exam Prep

As a database administrator or manager, you may have had one or more of these conversations with your application teams?

We need to build a new mobile app so customers can submit and retrieve documents, so I’m going to need a specialized database to store the documents, right?

Oh, wait, next week this other project requires we provide a new payment system that has user fraud protection built in to meet compliance and I am going to store relational data as well.  That requires a specialized Blockchain database and a relational database, right?

These conversations can leave you feeling concerned and frustrated, wondering how you are going to allocate your resources among all of these specialized data stores.

But perhaps the solution to your problem is not more resources but a converged database.

A converged database is a database that has native support for all modern data types and the latest development paradigms built into one product.

Oracle Database Tutorial and Material, Oracle Database Learning, Oracle Database Certification, Database Exam Prep
Converged databases support Spatial data for location awareness, JSON for document stores, IoT for device integration, in-memory technologies for real-time analytics, and of course, traditional relational data. By providing support for all of these data types, a Converged Database can run all sorts of workloads from IoT to Blockchain to Analytics and Machine Learning. It can also handle any development paradigm, including Microservices, Events, REST, SaaS, and CI/CD, to name a few.

Traditionally when new data management technologies first come out, they are implemented as separate products. For example, when Blockchain first came out, it was a separate stand-alone system that required you to use an entirely different, proprietary way to store and access data.

By integrating new data types, workloads, and paradigms as features within a converged database, you can support mixed workloads and data types in a much simpler way. You don't need to manage and maintain multiple systems or worry about having to provide unified security across them.

You also get synergy across these capabilities. For example, by having support for Machine Learning algorithms and Spatial data in the same database, you can easily do predictive analytics on Spatial data. Making it dramatically easier and faster to develop data-driven apps.

Oracle Database Tutorial and Material, Oracle Database Learning, Oracle Database Certification, Database Exam Prep
A good analogy for a converged database is a smartphone. In the past, if you wanted to make phone calls, you would use a phone, and if you wanted to take a picture or video, you would use a camera. If you wanted to navigate somewhere, you would need a map or a navigation system. If you wanted to listen to music, you needed an iPod or other similar device.

But with a smartphone, all of these products have been converted or converged into one. Each of these original products is now a feature of the smartphone. Having all of these features converged into a single product inherently makes your life easier, as you can stream music over the phone's data plan or upload pictures or videos directly to social media sites.

The same ease of use and convenience you get from a smartphone also hold for a converged database.

Oracle Database is an excellent example of a converged database, as it provides support for Machine Learning, Blockchain, Graph, Spatial, JSON, REST, Events, Editions, and IoT Streaming as part of the core database at no additional cost. It allows you to support many diverse projects using a single platform, significantly reducing complexity and management overhead, while minimizing risk.

Friday, July 10, 2020

Oracle Database 20c Preview

Innovations in Oracle Database


In this blog entry we'll take a look at some of the new features inside Oracle Database 20c which has just been refreshed to version 20.3 on the Oracle Cloud's DBCS platform.

Oracle Database 20c is the next release of Oracle's multi-model database. Like the versions that have preceded it, Oracle Database 20c provides industry leading scalability, availability and security for both OLTP and analytical workloads. It supports relational, JSON, XML, spatial, graph, columnar, OLAP and unstructured data, enabling you to focus on building applications without having to worry about how to persist such data.

Support and Previous Releases


You can find details on the significant features in our previous yearly release in the following posts:

Oracle Database 18c
Oracle Database 19c

Oracle Database 20c is a yearly short term support release allowing users to try out new functionality, or take advantage of cutting-edge features for applications that could benefit from it. While we are very excited about this latest release, Oracle recommends that most users should consider upgrading to Oracle Database 19c, since it provides long term support all the way through to April 2026.

Converged Database


Since the initial releases of Oracle Database, Oracle has taken the approach that storing and managing data in a single database platform makes more sense than breaking it up and storing it in single use engines. Using multiple independent engines inevitably results in issues with data integrity, consistency and security. By using a single engine that provides the best of breed support for all of the major data types and development paradigms, users can benefit from all of Oracle Database's key capabilities such as, ACID transactions, read consistency, centralised security model, parallel scans and DML, online backups, point in time recovery etc. - regardless of the approach you take to storing the data.

The decision to centralize your data inside Oracle Database doesn't mean sacrificing the ability to build applications using whatever design approach you think is appropriate. Oracle supports the creation of single database application as well as those adopting event-driven or microservice paradigms. Key to this approach is the use of Oracle's Multitenant Architecture to provide each service with its own virtual database (PDB). This still allows you to manage many PDBs as one, and it simplifies the federation of the database via inter PDB SQL operations.

And, if you need to support millions of concurrent users or geographically distribute your database because of regulatory requirements, Oracle Database Sharding makes it simple to do this while still providing a converged data model.

Oracle Database 20c New Features


Oracle Database 20c introduces several features, far more than is covered in this short blog posting.

Let's go through some of the significant enhancements in Oracle Database 20c Preview release.

Blockchain Tables


Blockchain as a technology has promised much in terms of solving many of the problems associated with the verification of transactions. While considerable progress has been made in bringing this technology to the enterprise, a number of problems exist, with arguably the largest being the complex nature of building applications that can support a distributed ledger.

To simplify the introduction of this exciting technology in Oracle Database 20c we're introducing Blockchain Tables. These tables operate like any normal heap table, but with a number of important differences. The most notable of these being that rows are cryptographically hashed as they are inserted into the table, ensuring that the row can no longer be changed at a later date.

Oracle Database 20c, Oracle Database Exam Prep, Oracle Database Certification, DB Learning

This essentially creates an insert only table. Blockchain Tables don't allow users to update or delete rows. Users are also prevented from truncating the data, dropping partitions or dropping the table within certain time limits.

This important capability means that other users can trust that the data held in the blockchain table is an accurate record of events. Oracle Database 20c enables you to run a process that will verify all of the records are consistent with their hash signature.

SQL Macros


It is not unusual for a SQL statement to grow in complexity as the number of joins increase, or the operations performed on the retrieved data becomes more involved. It is also not uncommon for developers to try and solve this problem by using stored procedures and table functions to simplify these commonly used operations. This works extremely well to simplify code, but can potentially sacrifice some performance as the SQL engine switches context with the PL/SQL Engine. In Oracle Database 20c, SQL Macros solve this problem by allowing SQL expressions and table functions to be replaced by calls to stored procedures which return a string literal to be inserted in the SQL we want to execute. It's an incredibly simple concept and one that C and Rust programmers will be familiar with. The following trivial example shows it in action.

First, let's create a tables and insert a few rows.

CREATE TABLE(id integer, name varchar2(30), item_type varchar2(30), price float );
insert into line_items values (1, 'Red Red Wine', 'ALCOHOL', 15.6)
insert into line_items values (2, 'Its Cold Out There Heater', 'RADIATOR', 200.49);
insert into line_items values (3, 'How Sweet It Is Cake', 'FOOD', 4.56);

The SQL below calculates the value added tax on rows in our LINE_ITEMS table

select id,
  case
    when item_type = 'ALCOHOL' then round(1.2 * price, 2)
    when item_type = 'SOLAR PANEL' then round(1.05 * price, 2)
    when item_type = 'RADIATOR' then round(1.05 * price, 2)
    else price end as total_price_with_tax
from line_items; 

However in Oracle Database 20c we can simplify it by creating a function with the new SQL_MACRO keyword and returning a string.

create or replace function total_price_with_tax(the_price float, the_item_type varchar)
  return varchar2 SQL_MACRO(SCALAR) is
begin
  return q'[case
    when item_type = 'ALCOHOL' then round(1.2 * price, 2)
    when item_type = 'SOLAR PANEL' then round(1.05 * price, 2)
    when item_type = 'RADIATOR' then round(1.05 * price, 2)
    else price end as total_price_with_tax]';
end; 

We can then simply reference the SQL Macro inside of a select statement. The SQL that's executed is exactly the same as the original SQL Statement without the overhead of a context switch each time the row is fetched to execute our function.

SQL > select id, total_price_with_tax(price, item_type) from line_items;

  ID      TOTAL_PRICE_WITH_TAX(PRICE,ITEM_TYPE) 
  ------------------------------------------
   1                                   18.72 
   2                                   210.5145 
   3                                   4.56 

The same approach can be used in parametrised views and Polymorphic tables.

Oracle Database In-Memory Enhancements


Analysing data using a columnar model can result in massive performance improvements when compared to doing the same operations using a row-based model. However, updating data is significantly faster when using data held in rows. Oracle Database In-Memory is unique in that it allows you to benefit from both approaches. With this capability you can run your relational or JSON application unchanged and Oracle Database will maintain a columnar store supporting blazingly fast real-time analytical queries.

Oracle Database 20c introduces three major improvements to enhance performance and ease of use when using Oracle Database In-Memory functionality.

◉ Database In-Memory Vector Joins : Through the use of its newly enhanced Deep Vectorization SIMD Framework, Oracle Database In-Memory can accelerate operations like hash joins on columns held inside of the In-Memory column store. In the case of a hash join, the join is broken down into smaller operations that can be passed to the vector processor. The key-value table used is SIMD optimized and used to match rows on the left and right-hand sides of the join. This approach can result in join performance improvements of up to 10 times over traditional methods.

◉ Self Managing In-Memory Column Store : When Oracle Database In-Memory was first released, you had to explicitly declare which columns were to be populated into the In-Memory Column Store. This gave you a high degree of control if memory was tight. In Oracle Database 18c, we introduced functionality that would automatically place objects in the Column Store if they are actively used and removed objects that weren't. However, you still had to indicate the objects you wanted considered. In Oracle Database 20c when you set  INMEMORY_AUTOMATIC_LEVEL to HIGH, all objects are considered. This automatic memory management significantly simplifies the job of managing the column store.

◉ In-Memory Hybrid Columnar Scans : It is often not possible to have every column of every table populated in the Column Store because memory is limited. In many instances, this isn't an issue but every once in a while you may encounter  a query which needs some of the data(columns) from the Column Store and some data that's only available in the row store. In previous versions of Oracle Database In-Memory, such querys would simply run against the row store. In Oracle Database 20c we can now use both. The optimizer can now elect to scan the Column Store and fetch projected column values from the row store if needed. This can result in a significant improvements in performance.

Oracle Database 20c, Oracle Database Exam Prep, Oracle Database Certification, DB Learning

Hybrid Columnar Scan

Native JSON Datatype


We introduced support for JSON in Oracle Database 12c (12.1.0.2). It allowed JSON to be stored in the database inside of a varchar2 or a LOB (CLOB or BLOB). This meant it was possible to build applications with the flexibility offered by a schemaless design model but benefiting from the power of the Oracle Database. You could query the JSON documents using standard SQL, take advantage of advanced analytics, index individual attributes or whole documents and process billions of JSON documents in parallel. We also provided tools to discover what attributes made up the JSON documents and trivially create relational views on top of the collections. It was also possible for developers to treat the Oracle Database as if it were a NoSQL Database by accessing it with the SODA (Simple Object Data API) APIs available for Java, Node.js, Python, C and REST.

In Oracle Database 20c we are improving our JSON support by offering a Native data type, "JSON". This means that instead of having to parse JSON on read or update operations, the parse only happens on an insert and the JSON is then held in an internal binary format which makes access much faster. This can result in read and update operations being 4 or 5 times faster and updates to very large JSON documents being 20 to 30 times faster.

CREATE TABLE j_order (
   id     INTEGER PRIMARY KEY,
  po_doc JSON );

The new data type wasn't the only change that got introduced for JSON in Oracle Database 20c Oracle also added a new JSON function JSON_TRANSFORM which makes it much simpler to update and remove multiple attributes in a document in a single operation.

UPDATE j_order SET po_doc = JSON_TRANSFORM( po_doc,
                                            SET '$.address.city' = 'Santa Cruz’,
                                            REMOVE'$.phones[*]?(@.type = "office")’ )
 WHERE id = 555;

And of course, we also added compatibility for the new JSON datatype to our drivers and utilities like Datapump and GoldenGate.

Machine Learning for Python and AutoML


Machine Learning has been built into Oracle Database since the release of 8i. It doesn't require analysts or data scientists to extract the data onto a file system or specialist database, but rather allows you to leverage the power of Oracle Database and build models with over 30 Machine Learning algorithms running directly on data held in your tables. This approach of moving the algorithms to the data minimizes or eliminates data movement, achieves scalability, preserves data security, and accelerates time-to-model deployment.

In Oracle Database 20c we are introducing functionality to make it even simpler for users to take advantage of this functionality by providing Python Machine Learning interfaces to Oracle Database. This new client compliments the R and SQL interfaces already available. Data Scientists can now work in an environment they feel comfortable with, and simply treat the Oracle Database as a high performance compute engine. You can use programmatic structures similar to those in the Scikit-learn, and Panda frameworks which become simple proxies for tables in the database and then call the Oracle Database ML algorithms using Python function calls rather than SQL.

Oracle Machine Learning for Python also looks to simplify the process of selecting and tuning the right model which can be a complicated and time-consuming exercise. New AutoML functionality can select the model, the relevant attributes and then tune the hyper parameters. This dramatically reduces the time required to create a model that accurately creates predictions or classifies your data.

Oracle Database 20c, Oracle Database Exam Prep, Oracle Database Certification, DB Learning

In Oracle Database 20c Machine Learning we are also adding support for the MSET-SPRT and XGBoost algorithms, and the Adam Optimization solver for the Neural Network Algorithm.

Other Notable Enhancements


There were over 100+ enhancements to Oracle Database 20c. And whilst I'd love to cover them all in detail I'd simply be repeating a lot of content from Oracle Database New Features Guide. However there are a few more features that I think are of particular note.

Expression based init.ora parameters : It's now possible to base database parameters (init.ora) on calculations made on the configuration of the system, i.e. set the database parameter CPU_COUNT on half the number of CPUs available to the operating system.

Automatic Zone Maps : Exadata can now automatically create Zone Maps based on the predicates used in queries. Previously this was a manual operation requiring you to understand how the data would be accessed. This can dramatically reduce the number of blocks that need to be scanned.

Optimised Graph Models : Graphs can consist of millions or even billions of edges and vertices and so the storage optimisations we've made to the graph capabilities in Oracle Database 20c preview release can result in big space and performance improvements for your models.

Sharding Enhancements : To make it easier to develop Java applications against Oracle Sharding we've introduced a new Java Data Source that makes it simple to obtain connections without having to define the shard key or manage the connection key explicitly. We have also made sharding more fault-tolerant by automatically looking for alternates if the shard you are working on fails during execution.

Persistent Memory Support : In Oracle Database 20c we provide support for Databases running on top of Persistent Memory File Systems. PMEM File systems can offer significant latency and bandwidth improvements over traditional file systems using SSD or mechanical disks. However, the applications using them need to understand how to safely write to them and the most efficient way to use them in conjunction with other OS resources. Oracle Database 20c's implementation provides atomic writes, safe guarding against partial writes during unexpected power outages. It also offers Fast I/O operations using memory copy. In addition it efficiently uses database buffer cache by bypassing and reading directly from PMEM storage.

Source: oracle.com