Big Data - Overview

Overview

Due to the advent of new technologies, devices, and communication means like social networking sites, the amount of data produced by mankind is growing rapidly every year. The amount of data produced by us from the beginning of time till 2003 was 5 billion gigabytes. If you pile up the data in the form of disks it may fill an entire football field. The same amount was created in every two days in 2011, and in every ten minutes in 2013. This rate is still growing enormously. Though all this information produced is meaningful and can be useful when processed, it is being neglected.

90% of the world’s data was generated in the last few years.

What is Big Data?

Big data means really a big data, it is a collection of large datasets that cannot be processed using traditional computing techniques. Big data is not merely a data, rather it has become a complete subject, which involves various tools, technqiues and frameworks.

What Comes Under Big Data?

Big data involves the data produced by different devices and applications. Given below are some of the fields that come under the umbrella of Big Data.

• Black Box Data : It is a component of helicopter, airplanes, and jets, etc. It captures voices of the flight crew, recordings of microphones and earphones, and the performance information of the aircraft.
• Social Media Data : Social media such as Facebook and Twitter hold information and the views posted by millions of people across the globe.
• Stock Exchange Data : The stock exchange data holds information about the ‘buy’ and ‘sell’ decisions made on a share of different companies made by the customers.
• Power Grid Data : The power grid data holds information consumed by a particular node with respect to a base station.
• Transport Data : Transport data includes model, capacity, distance and availability of a vehicle.
• Search Engine Data : Search engines retrieve lots of data from different databases.

DELL BOOMI ATOMSPHERE - Core Terms and Concepts

There are several important terms and concepts to understand when working with Dell Boomi AtomSphere:

Atom

Atoms are the lightweight run-time engines that contain process configuration information and actually do the work.

Atoms can be downloaded and installed to a machine behind your firewall to access local resources such as on-premise applications or databases, or deployed to the Dell Boomi data center if only web accessible applications or data sources need to be integrated. You will typically deploy multiple integration processes to a single Atom.

Component

Components are the reusable configuration objects used in processes.

Components are reusable and include certificates, connections, connector operations, cross reference tables, document caches, maps, map functions, processes, process properties, profiles, queues, trading partners and web services. 

Connector

Connectors get data into and send data out of processes.

Connectors abstract the technical details of communicating with various applications, data sources and communication protocols. They are actually comprised of two components: a connection and an operation. The connection represents the "endpoint" and contains the physical connection details, such as an FTP or database host, a web services URL and/or login credentials. The operation represents a specific action to perform against that connection, such as a database SELECT query, an FTP PUT or a specific web service call. You can think of the connection as the "where" and the operation as the "how". For example, when extracting customer records from Salesforce, the connection represents your Salesforce organization account user name and password and the operation represents the "query customer" action.

Execution

An execution is the result of running a process.

Map

Maps allow data to be transformed from one format to another — or specifically, from one profile to another.

Fields or elements are mapped by dragging and dropping a source element to a destination element. Map functions can be used to perform simple or complex field-level manipulations as values are mapped.

Process

Processes represent a single type of data to integrate between two or more systems. They are comprised of a series of steps (represented in the UI by shapes) that specify the business logic and transformation requirements.

Processes are the central components in AtomSphere. Process steps transform, route and otherwise manipulate data to attain the desired end result. Every process begins with a Start shape that gets data from some source and usually ends with one or more connectors to send the data to the destination(s).

Profile

Profiles define the structure or layout of the data to be processed. They are used when reading from or writing to a given format. Several types of profiles are available:

Database — When used for reading, the database profile represents the fields in the result set(s) of SELECT statements. When used for writing, the profile represents the values used in an UPDATE, INSERT or DELETE statement. Database profiles can also contain elements to be used as parameters in dynamic statements.

EDI — Electronic Data Interchange (EDI) is defined as the exchange of business data from one computer to another computer using a public standard format. EDI replaces the traditional processes of preparing data in paper form and sending it by mail or by facsimile. Business communications with EDI is not restricted by software, equipment or computer. EDI profiles represent the structure of the various EDI documents sent through AtomSphere processes, including all the segments, data elements, looping and other configuration information. Similar to the XML profile, the EDI profile allows you to configure repeating data sets (loops). You can also organize segments in Header, Detail and Summary loop sections.

Flat File — Flat files are common, simple formats that typically contain one record per line, like a comma separated value (CSV) file. Each record contains a number of fields or elements that are either delimited or data positioned (fixed width). Flat file profiles can represent simple structures with a single record type (e.g., CSV file) to moderately complex structures with multiple record types to accommodate parent-child relationships or other repeating record types within the same file (e.g., header and detail records).

XML — XML is a common hierarchical format used in much of the Internet communication today characterized by the use of tags that describe data elements. Tags can be nested within one another to establish parent-child relationships. XML profiles are used by web services and the majority of AtomSphere application connectors.

JSON (JavaScript Object Notation) is a text-based data interchange format. It is easy for humans to read and write and for machines to parse and generate. The JSON format is often used to transmit structured data over a network connection. It is similar to XML and is often used as an alternative to XML.

Shape

Process shapes are the steps linked together to form the business logic for a process.

Shapes can perform execution tasks like document splitting and data transformation, or logic tasks like decision and routing. Each shape has its own configuration settings. Shapes often reference components such as profiles, connections, operations, and maps.

DELL BOOMI ATOMSPHERE

The main component in Dell Boomi AtomSphere is the process. A process represents a business process- or transaction-level interface between two systems. Examples of a process might be "Salesforce Account Synchronization to a legacy accounting system" or "Sales Orders from Company ABC to QuickBooks." Processes contain a left-to-right series of shapes connected together like a flow chart to illustrate the steps required to transform, route and otherwise manipulate the data from source to destination.

Processes use connectors to get and send data. Every process begins with the Start shape (containing the inbound connector) on the left. A process can have any number of execution and logic shapes within it. A process typically ends with one or more outbound connectors on the right.

Profiles represent the structure and format of the source and destination data. The following types of profiles are available: Database, EDI, Flat File, XML and JSON. A process typically contains one or more maps along the way that transform the source profile into the destination profile.

When a process is executed, the inbound connector in the Start shape retrieves its data (for example: performs an FTP GET, executes a database SELECT query or invokes a web service call) and passes the resulting data into the process as one or more documents. Documents move through the process steps sequentially, where each shape performs a specific task on the data and then passes it onto the next shape.

What Is Integration?

Integration is the combining of software or hardware components or both into an overall system. Generally speaking, there are two types of systems integration:

B2B Integration — Integrating data or systems across two or more organizations.

Application Integration — Integrating data or systems within a single organization.

What is B2B Integration?

Business-to-Business integration is not a new concept and in fact many Information technology organisations have been running B2B Integration projects since the late 1960’s. Simply put B2B Integration means the integration, automation and optimisation of key business processes that extend outside the four walls of a companies organisation.

While these critical processes vary by vertical, geography and company size one point remains consistent, the automation of key external business processes, those that touch your customers and your suppliers delivers sustainable competitive advantage.

For example receiving purchase orders from your customers electronically, you can process order information faster and more accurately. Processing these orders in real time allows companies to be more responsive to their customers, improve customer service and increase sales. Similarly, by connecting to external suppliers electronically, companies can achieve real time views into the visibility of global shipments, automating the warehouse or distribution centres and optimising inventory or stock control – ultimately increasing working capital and lowering costs.

B2B Integration began with large companies mandating methods of receiving business information technology. It evolved through the widespread adoption of Electronic Data Interchange (EDI) and in recent years has benefited from technology innovations e.g. the advent of the Internet, XML, web services and SOA, Business Process Management and SaaS. These innovations have led to increased benefits being made available to companies of every size. As we explore in this Microsite there are a number of ways to implement B2B Integration solutions. We discuss that the solution approach should be driven by a company’s business needs and objectives, rather than a particular implementation or technology set.

https://en.wikipedia.org/wiki/Business-to-business

iPaaS, Native Cloud, Multi-Tenancy

Integration Platform as a Service (iPaaS) is a suite of cloud services enabling development, execution and governance of integration flows connecting any combination of on premises and cloud-based processes, services, applications and data within individual or across multiple organizations.



Native Cloud Application (NCA) refers to a type of computer software that natively utilizes services and infrastructure provided by cloud computing providers



Multi-tenancy is an architecture in which a single instance of a software application serves multiple customers. Each customer is called a tenant. Tenants may be given the ability to customize some parts of the application, such as color of the user interface (UI) or business rules, but they cannot customize the application's code.

Multi-tenancy can be economical because software development and maintenance costs are shared. It can be contrasted with single-tenancy, an architecture in which each customer has their own software instance and may be given access to code. With a multi-tenancy architecture, the provider only has to make updates once. With a single-tenancy architecture, the provider has to touch multiple instances of the software in order to make updates.

In cloud computing, the meaning of multi-tenancy architecture has broadened because of new service models that take advantage of virtualization and remote access. A software-as-a-service (SaaS) provider, for example, can run one instance of its application on one instance of a database and provide web access to multiple customers. In such a scenario, each tenant's data is isolated and remains invisible to other tenants.

DELL BOOMI

The main component in Dell Boomi AtomSphere is the process. A process represents a business process- or transaction-level interface between two systems.

Processes contain a left-to-right series of shapes connected together like a flow chart to illustrate the steps required to transform, route and otherwise manipulate the data from source to destination.

Processes use connectors to get and send data. Every process begins with the Start shape on the left. A process can have any number of execution and logic shapes within it. A process typically ends with one or more outbound connectors on the right.

Profiles represent the structure and format of the source and destination data. The following types of profiles are available: Database, EDI, Flat File, XML and JSON. A process typically contains one or more maps along the way that transform the source profile into the destination profile.

When a process is executed, the inbound connector in the Start shape retrieves its data (for example: performs an FTP GET, executes a database SELECT query or invokes a web service call) and passes the resulting data into the process as one or more documents. Documents move through the process steps sequentially, where each shape performs a specific task on the data and then passes it onto the next shape.

AtomSphere Editions

You can get a 30–day free trial account at any time. It is a separate edition of Dell Boomi AtomSphere called “Personal Edition”. Once your trial period has expired you have the option to fully subscribe to AtomSphere. Then you can deploy the integration processes you have built during your trial, subject to the same terms.

When you purchase AtomSphere you have the choice of four editions:

• Base Edition
• Professional Edition
• Professional Plus Edition
• Enterprise Edition
• Enterprise Plus Edition