RDBMS vs NOSQL vs HADOOP

 As the world becomes more information-driven than ever before, a major challenge has become how to deal with the explosion of data. The most common description of Data talks about the four V’s: Volume, Velocity, Variety, and Veracity.

                Volume - Represents the size of data.

                Velocity - refers to the speed with which data is generated.

                Variety - About data in many forms:  structured, semi-structured, and unstructured data.

                Veracity - How accurate or truthful data. Reduce inconsistencies, incompleteness, and ambiguities.

Here is an overview of important technologies to know about how the pieces of information are stored and retrieved.

Traditional RDBMS

                The Relational Databases (RDBMS) (SQL) have dominated the Database market, and they have done a lot of good. So what changed? Web technology started the revolution. Today, many people shop on Amazon. RDBMS was not designed to handle the number of transactions that take place on Amazon every second. The primary constraining factor was the RDBMS’ schema.

NoSQL Database Systems

                NoSQL (commonly referred to as "Not Only SQL") represents a completely different framework of databases that allows for high-performance, agile processing of information at a massive scale. Also offered an alternative by eliminating schemas at the expense of relaxing ACID principles.

Hadoop

                Hadoop is a file system and not a database. Although Hadoop and associates (Hbase, Mapreduce, Hive, Pig, Zookeeper) have turned it into a mighty database, Hadoop is a scalable, inexpensive distributed filesystem with fault tolerance.

Drop Table - If foreign key exists

 DROP TABLE if exists abc cascade;

JAVA SCRIPT - TO GET THE CHARACTER POSITION & SLICE(STRING CUT)

var out_filename = short_filename.slice(0, short_filename.indexOf("."))

CRON Schedule

0 5 * * * /home/pentaho/pentaho-pdi/data-integration/pan.sh -file="/home/pentaho/test/tr_test.ktr" –level: Basic –logfile="/home/pentaho/test/test.log"

List file names in a directory by file explorer

• Go to the specific directory where files are stored.
• Select all files by CNTRL+A.
• Press Shift and Right click on files
• Select “copy as path” from right click menu
• Paste the result in file editor

VLOOKUP - Data Cleansing - MSExcel

 =IF(ISNA(VLOOKUP($B3,G$3:G$26,1,FALSE)), "Not Present", "Present")

=IF(ISNA(VLOOKUP($D3,B$3:B$27,1,FALSE)), "Not Used", "Used")

Create Sequence in tables

CREATE SEQUENCE public.seq_abcded_id

    INCREMENT 1

    START 1

    MINVALUE 1

    MAXVALUE 9223372036854775807

    CACHE 1;

ALTER SEQUENCE public.seq_abcded_id

    OWNER TO postgres;

CREATE TABLE public.abcde

(

    id integer NOT NULL DEFAULT nextval('seq_abcded_id'::regclass),

    name character varying(100),

    PRIMARY KEY (id)

)

WITH (

    OIDS = FALSE

);

ALTER TABLE public.abcde

    OWNER to postgres;