InfoSphere DataStage

Integrate all types of data on distributed and mainframe platforms

IBM® InfoSphere® DataStage® integrates data across multiple systems using a high performance parallel framework, and it supports extended metadata management and enterprise connectivity. The scalable platform provides more flexible integration of all types of data, including big data at rest (Hadoop-based) or in motion (stream-based), on distributed and mainframe platforms.

InfoSphere DataStage provides these features and benefits:

·         Powerful, scalable ETL platform—supports the collection, integration and transformation of large volumes of data, with data structures ranging from simple to complex.

·         Support for big data and Hadoop—enables you to directly access big data on a distributed file system, and helps clients more efficiently leverage new data sources by providing JSON support and a new JDBC connector.

·         Near real-time data integration—as well as connectivity between data sources and applications.

·         Workload and business rules management—helps you optimize hardware utilization and prioritize mission-critical tasks.

·         Ease of use—helps improve speed, flexibility and effectiveness to build, deploy, update and manage your data integration infrastructure.

·         Rich support for DB2Z and DB2 for z/OS—including data load optimization for DB2Z and balanced optimization for DB2 on z/OS

Diff b/w Rest and Soap

• SOAP stands for Simple Object Access Protocol. REST stands for REpresentational State Transfer.
• SOAP is a XML based messaging protocol and REST is not a protocol but an architectural style.
• SOAP has a standard specification but there is none for REST.
• Whole of the web works based on REST style architecture. Consider a shared resource repository and consumers access the resources.
• Even SOAP based Web service can be implemented in RESTful style. REST is a concept that does not tie with any protocols.
• SOAP is distributed computing style and REST is web style (web is also a distributed computing model).
• REST messages should be self-contained and should help consumer in controlling the interaction between provider and consumer(example, links in message to decide the next course of action). But SOAP doesn’t has any such requirements.

• REST does not enforces message format as XML or JSON or etc. But SOAP is XML based message protocol.
• REST follows stateless model. SOAP has specifications for stateful implementation as well.
• SOAP is strongly typed, has strict specification for every part of implementation. But REST gives the concept and less restrictive about the implementation.
• Therefore REST based implementation is simple compared to SOAP and consumer understanding.
• SOAP uses interfaces and named operations to expose business logic. REST uses (generally) URI and methods like (GET, PUT, POST, DELETE) to expose resources.
• SOAP has a set of standard specifications. WS-Security is the specification for security in the implementation. It is a detailed standard providing rules for security in application implementation. Like this we have separate specifications for messaging, transactions, etc. Unlike SOAP, REST does not has dedicated concepts for each of these. REST predominantly relies on HTTPS.
• Above all both SOAP and REST depends on design and implementation of the application.

Web Service : Rest vs Soap (Detailed)

Web services overview

A Web service, in very broad terms, is a method of communication between two applications or electronic devices over the World Wide Web (WWW). Web services are of two kinds: Simple Object Access Protocol (SOAP) and Representational State Transfer (REST).

SOAP defines a standard communication protocol (set of rules) specification for XML-based message exchange. SOAP uses different transport protocols, such as HTTP, JMS and SMTP. The standard protocol HTTP makes it easier for SOAP model to tunnel across firewalls and proxies without any modifications to the SOAP protocol. SOAP can sometimes be slower than middleware technologies like CORBA or ICE due to its verbose XML format.

REST describes a set of architectural principles by which data can be transmitted over a standardized interface (such as HTTP). REST does not contain an additional messaging layer and focuses on design rules for creating stateless services. A client can access the resource using the unique URI and a representation of the resource is returned. With each new resource representation, the client is said to transfer state. While accessing RESTful resources with HTTP protocol, the URL of the resource serves as the resource identifier and GET, PUT, DELETE, POST and HEAD are the standard HTTP operations to be performed on that resource.

REST vs. SOAP

Multiple factors need to be considered when choosing a particular type of Web service, that is between REST and SOAP. The table below breaks down the features of each Web service based on personal experience.

REST

The RESTful Web services are completely stateless. This can be tested by restarting the server and checking if the interactions are able to survive.
Restful services provide a good caching infrastructure over HTTP GET method (for most servers). This can improve the performance, if the data the Web service returns is not altered frequently and not dynamic in nature.
The service producer and service consumer need to have a common understanding of the context as well as the content being passed along as there is no standard set of rules to describe the REST Web services interface.
REST is particularly useful for restricted-profile devices such as mobile and PDAs for which the overhead of additional parameters like headers and other SOAP elements are less.
REST services are easy to integrate with the existing websites and are exposed with XML so the HTML pages can consume the same with ease. There is hardly any need to refactor the existing website architecture. This makes developers more productive and comfortable as they will not have to rewrite everything from scratch and just need to add on the existing functionality.
REST-based implementation is simple compared to SOAP.


SOAP

The Web Services Description Language (WSDL) contains and describes the common set of rules to define the messages, bindings, operations and location of the Web service. WSDL is a sort of formal contract to define the interface that the Web service offers.
SOAP requires less plumbing code than REST services design, (i.e., transactions, security, coordination, addressing, trust, etc.) Most real-world applications are not simple and support complex operations, which require conversational state and contextual information to be maintained. With the SOAP approach, developers need not worry about writing this plumbing code into the application layer themselves.
SOAP Web services (such as JAX-WS) are useful in handling asynchronous processing and invocation.
SOAP supports several protocols and technologies, including WSDL, XSDs, SOAP, WS-Addressing


In a nutshell, when you're publishing a complex application program interface (API) to the outside world, SOAP will be more useful. But when something with a lower learning curve, and with lightweight and faster results and simple transactions (i.e., CRUD operations) is needed, goes to REST.


Foe example web service creation:
http://javapapers.com/web-service/soap-web-service-introduction/

Web Service : Rest vs Soap

A Web service is a method of communication between two electronic devices over World Wide Web.

Web services are of two kinds: Simple Object Access Protocol (SOAP) and Representational State Transfer (REST).


REST is almost always going to be faster. The main advantage of SOAP is that it provides a mechanism for services to describe themselves to clients, and to advertise their existence.

REST is much more lightweight and can be implemented using almost any tool, leading to lower bandwidth and shorter learning curve. However, the clients have to know what to send and what to expect.

In general, When you're publishing an API to the outside world that is either complex or likely to change, SOAP will be more useful. Other than that, REST is usually the better option.

Storage Tools Typically used in Big- Data

What is Big data!! It’s nothing but a collection of complex data which is difficult to process with the existing tools. The size of data ranges forms few dozen terabytes to many petabytes of data in a single data set. This data can be posts to social media sites, digital pictures and videos or any other information

Apache Hadoop is a very popular solution Big Data .For the storage of Big data we use different kind of storage like S3, Hadoop Distributed File System (HDFS)

Amazon S3 filesystem.

Amazon S3 filesystem

What is S3??

Amazon S3 (Simple Storage Service) is an online file storage web service (Internet hosting service specifically designed to host user files.) offered by Amazon Web Services. Apache Hadoop file systems can be hosted on S3, alsoTumblr, Formspring, Pinterest, and Posterous images are hosted on the S3 servers.

S3 stores arbitrary objects (computer files,) up to 5 terabytes in size. This are stored in the form of buckets. It can store data from web applications to media files and we can retrieve from anywhere in Web.

Hadoop Distributed File System

The Hadoop Distributed File System (HDFS) is a portable file system built for the Hadoop framework.

HDFS is to store very large amount of data by sharing the storage and computation across many servers. HDFS stores large files with ideal file size is a multiple of 64 MB

Below are some of the organizations that are using Hadoop.

Facebook

Yahoo

Amazon.com

LinkedIn

StumbleUpon

Twitter

Google and many more companies…

Languages used..

Oracle: Cursors

A cursor is a mechanism by which you can assign a name to a "select statement" and manipulate the information within that SQL statement.

Oracle/PLSQL: Declare a Cursor

A cursor is a SELECT statement that is defined within the declaration section of your PLSQL code. We'll take a look at three different syntaxes for cursors.

Cursor without parameters (simplest)

The basic syntax for a cursor without parameters is:

CURSOR cursor_name
IS
  SELECT_statement;

Cursor with parameters

The basic syntax for a cursor with parameters is:

CURSOR cursor_name (parameter_list)
IS
  SELECT_statement;

Oracle/PLSQL: OPEN Statement

Once you've declared your cursor, the next step is to open the cursor.

The basic syntax to OPEN the cursor is:

OPEN cursor_name;

Oracle/PLSQL: FETCH Statement

The purpose of using a cursor, in most cases, is to retrieve the rows from your cursor so that some type of operation can be performed on the data. After declaring and opening your cursor, the next step is to FETCH the rows from your cursor.

The basic syntax for a FETCH statement is:

FETCH cursor_name INTO <list of variables>;

Oracle/PLSQL: CLOSE Statement

The final step of working with cursors is to close the cursor once you have finished using it.

The basic syntax to CLOSE the cursor is:

CLOSE cursor_name;