Tuesday, 30 September 2014

Informatica Tips

Informatica Performance Improvement Tips

Following standards/guidelines can improve the overall performance:
  • Use Source Qualifier if the Source tables reside in the same schema.
  • Make use of Source Qualifer  "Filter" Properties if the Source type is Relational.
  • If the subsequent sessions are doing lookup on the same table, use persistent cache in the first session. Data remains in the Cache and available for the subsequent session for usage.
  • Use flags as integer, as the integer comparison is faster than the string comparison.
  • Use tables with lesser number of records as master table for joins.
  • While reading from Flat files, define the appropriate data type instead of reading as String and converting.
  • Have all Ports that are required connected to Subsequent Transformations else check whether we can remove these ports.
  • Suppress ORDER BY using the '--' at the end of the query in Lookup Transformations.
  • Minimize the number of Update strategies.
  • Group by simple columns in transformations like Aggregate, Source Qualifier.
  • Use Router transformation in place of multiple Filter transformations.
  • Turn off the Verbose Logging while moving the workflows to Production environment.
  • For large volume of data drop index before loading and recreate indexes after load.
  • For large of volume of records Use Bulk load Increase the commit interval to a higher value large volume of data.
  • Set 'Commit on Target' in the sessions.

Thursday, 21 August 2014

UNIX COMMANDS 1

FILE COMMANDS

touch - Create a new file.

  • Usage: touch <filename>


cp - Copy files.


  • Usage: cp [options] <source-filename> <destination-filename>
  • cp [options] <source-filepath > <destination filepath>
  • Options: -r recursively copy directory structures.

mv - Move or Rename files or directories.

  • Usage: mv [options] <old-filepath> <new-filepath>
  • mv [options] <old-filename> <new-filename>
  • Options: -i query user for confirmation.

rm - Remove files.


  • Usage: rm [options] <filname>
  • Options: -r recursively remove directory structures. 
  • -i query user for confirmation. 

cat - View complete file content.



  • cat <filename>

more - View file contents in sections determined by the size of the terminal.



  • Usage: more <filename>


less - View file contents in sections determined by the size of the terminal.


  • Has more options and search features than more.
  • Usage: less [options] <filename>


compress - Reduces the size of the file. and adds the extension .Z



  • Usage: compress <filename>.


uncompress


  • Restores a compressed file.
  • Usage: uncompress <filename>
  • Usage: zcat <filename>

Wednesday, 9 July 2014

Project Explanation


Project Explanation:

First u have to start with

1) You have to first explain about objective of the project and what is client expectations
2) You have to start where your involvement and responsibility of your job and limitations of job.

Add some points from post Project Architecture reply like offshore and onsite model and team structure. etc.,

Main objective of this project is we are providing a system with all the information regarding Sales / Transactions (sales if sales domain / transactions if bank domain or insurance domain) of entire organizations all over the country US / UK ( based on the client location US/UK/….). we will get the daily transaction data from all branches at the end of the day. We have to validate the transactions and implement the business logic based on the transactions type or transaction code. We have to load all historical data into data warehouse and once finished historical data. We have to load Delta Loads. Delta load means last 24 hrs transactions captured from the source system. In other words u can call it as Change Data Capture (CDC). This Delta loads are scheduled daily basis. Pick some points from What is Target Staging Area Post. Source to Staging mappings, staging to warehousing. based on your comfort level.

Each transaction contains Transaction code. based on the transaction code u can identify whether that transaction belongs to sales, purchase / car insurance, health insurance, /deposit , loan, payment ( u have to change the words based on the project.) etc., based on that code business logic will be change. we validate and calculate the measure and load to database.

Mapping explanation :

In Informatica mapping, we first look up all the transaction codes with code master table to identify the transaction type to implement the correct logic and filter the unnecessary transactions. because in an organization there are lot of transactions will be there but you have to consider only required transactions for your project. the transaction code exists in the code master table are only transactions u have to consider and other transactions load into one table called Wrap table and invalid records( transaction code missing, null,spaces) to error table. For each dimension table we are creating surrogate key and load into data-warehouse tables.

SCD2 Mapping:We are implementing SCD2 mapping for customer dimension or account dimension to keep history of the accounts or customers. We are using SCD2 Date method. before telling this you should know it clearly about this SCD2 method.

Wednesday, 25 June 2014

My 100 th post

Boom, it's my 100 th post. I just wanna share my happiness at this time. I started this blog two year ago with no idea where to start and where to end. Later I came to know this will be my never ending one till my life end.

I began to love this blog. It has a peaceful place place for me to learn.

Sharing these things with you has done so much for me. I hope whoever reads this blog, learn it, feels encouraged because of what I write here.

'I hope it brings some good to the world'.


Application programming interface 1

What is an API?

An application-programming interface (API) is a set of programming instructions and standards for accessing a Web-based software application or Web tool. A software company releases its API to the public so that other software developers can design products that are powered by its service.
For example, Amazon.com released its API so that Web site developers could more easily access Amazon's product information. Using the Amazon API, a third party Web site can post direct links to Amazon products with updated prices and an option to "buy now."
An API is a software-to-software interface, not a user interface. With APIs, applications talk to each other without any user knowledge or intervention. When you buy movie tickets online and enter your credit card information, the movie ticket Web site uses an API to send your credit card information to a remote application that verifies whether your information is correct. Once payment is confirmed, the remote application sends a response back to the movie ticket Web site saying it's OK to issue the tickets.
As a user, you only see one interface -- the movie ticket Web site -- but behind the scenes, many applications are working together using APIs. This type of integration is called seamless, since the user never notices when software functions are handed from one application to another [source: TConsult, Inc.]
An API resembles Software as a Service (SaaS), since software developers don't have to start from scratch every time they write a program. Instead of building one core application that tries to do everything -- e-mail, billing, tracking, etcetera -- the same application can contract out certain responsibilities to remote software that does it better.­
Let's use the same example of Web conferencing from before. Web conferencing is SaaS since it can be accessed on-demand using nothing but a Web site. With a conferencing API, that same on-demand service can be integrated into another Web-based software application, like an instant messaging program or a Web calendar.
The user can schedule a Web conference in his Web calendar program and then click a link within the same program to launch the conference. The calendar program doesn't host or run the conference itself. It uses a conferencing API to communicate behind the scenes with the remote Web conferencing service and seamlessly delivers that functionality to the user.

Monday, 23 June 2014

Mapplets

A mapplet is a reusable object that you create in the Mapplet Designer. It contains a set of transformations (more than 1 transformation) and lets you reuse the transformation logic in multiple mappings.

For example, you a similar logic need to add more than 1 mapping. Like concatenate First_Name & Last_Name, Address_Street_1 & Address_Street_2 etc and this logic is need for Employee_Table, Student_Table, Patient_Table 3 mappings.

In this scenario instead of repeating the logic, we can have a mapplet for concatenate (Merge) columns and use this mapplet instance for the above mentioned 3 mappings.

When you use a mapplet in a mapping, you use an instance of the mapplet. Like a reusable transformation, any change made to the mapplet is inherited by all instances of the mapplet.

Mapplets help simplify mappings in the following ways:

Include source definitions. Use multiple source definitions and source qualifiers to provide source data for a mapping.
Accept data from sources in a mapping. If you want the mapplet to receive data from the mapping, use an Input transformation to receive source data.
Include multiple transformations. A mapplet can contain as many transformations as you need.
Pass data to multiple transformations. You can create a mapplet to feed data to multiple transformations. Each Output transformation in a mapplet represents one output group in a mapplet.
Contain unused ports. You do not have to connect all mapplet input and output ports in a mapping.

To use a mapplet in a mapping, first we need to configure it for input and output. In addition to transformation logic that you configure, a mapplet has the following components:


  1. Mapplet input
  2. Mapplet output
  3. Mapplet ports



Mapplet Input

Mapplet input can originate from a source definition and/or from an Input transformation in the mapplet. You can create multiple pipelines in a mapplet. Use multiple source definitions and source qualifiers or Input transformations. You can also use a combination of source definitions and Input transformations.

Mapplet Output

Mapplet Output transformation in a mapplet to pass data through the mapplet into a mapping. A mapplet must contain at least one Output transformation with at least one connected port in the mapplet. Each connected port in an Output transformation displays as a mapplet output port in a mapping. Each Output transformation in a mapplet displays as an output group in a mapping. An output group can pass data to multiple pipelines in a mapping.

Use the following rules and guidelines when you add transformations to a mapplet:
If you use a Sequence Generator transformation, you must use a reusable Sequence Generator transformation.

If you use a Stored Procedure transformation, you must configure the Stored Procedure Type to be Normal.
You cannot include PowerMart 3.5-style LOOKUP functions in a mapplet
You cannot include the following objects in a mapplet:

Normalizer transformations
COBOL sources
XML Source Qualifier transformations
XML sources
Target definitions
Other mapplets

Although reusable transformations and shortcuts in a mapplet can be used, to protect the validity of the mapplet, use a copy of a transformation instead. Reusable transformations and shortcuts inherit changes to their original transformations. This might invalidate the mapplet and the mappings that use the mapplet.

Note:


  • Use the following rules and guidelines when you edit a mapplet that is used by mappings:
  • Do not delete a port from the mapplet. The Designer deletes mapplet ports in the mapping when you delete links to an Input or Output transformation or when you delete ports connected to an Input or Output transformation.
  • Do not change the datatype, precision, or scale of a mapplet port. The datatype, precision, and scale of a mapplet port is defined by the transformation port to which it is connected in the mapplet. Therefore, if you edit a mapplet to change the datatype, precision, or scale of a port connected to a port in an Input or Output transformation, you change the mapplet port.
  • Do not change the mapplet type. If you remove all active transformations from an active mapplet, the mapplet becomes passive. If you add an active transformation to a passive mapplet, the mapplet becomes active. 

Application programming interface

In computer programming, an application programming interface (API) specifies how some software components should interact with each other.

In addition to accessing databases or computer hardware, such as hard disk drives or video cards, an API can be used to ease the work of programming graphical user interface components. In practice, many times an API comes in the form of a library that includes specifications for routines, data structures, object classes, and variables. In some other cases, notably for SOAP and REST services, an API comes as just a specification of remote calls exposed to the API consumers

Another one.,

API, an abbreviation of application program interface, is a set of routines, protocols, and tools for building software applications. The API specifies how software components should interact and are used when programming graphical user interface (GUI) components.  A good API makes it easier to develop a program by providing all the building blocks. A programmer then puts the blocks together.

Popular API Examples

Programmable Web, a site that tracks more than 9,000 APIs, lists Google Maps, Twitter, YouTube, Flickr and Amazon Product Advertising as the most popular APIs (view all sorted by popularity).

1. Google Maps API: Google Maps APIs lets developers embed Google Maps on webpages using a JavaScript or Flash interface. The Google Maps API is designed to work on mobile devices and desktop browsers.

Example API Use : Charting Your Course Using the Google Maps API.

2. YouTube APIs: YouTube API: Google's APIs lets developers integrate YouTube videos and functionality into websites or applications. YouTube APIs include the YouTube Analytics API, YouTube Data API, YouTube Live Streaming API, YouTube Player APIs and others.

3. Flickr API: The Flickr API is used by developers to access the Flick photo sharing community data. The Flickr API consists of a set of callable methods, and some API endpoints.

4. Twitter APIs: Twitter offers two APIs. The REST API allows developers to access core Twitter data and the Search API provides methods for developers to interact with Twitter Search and trends data.
Example API Use : Using the Twitter API to Create an Early Alert System.

5. Amazon Product Advertising API: Amazon's Product Advertising API gives developers access to Amazon's product selection and discovery functionality to advertise Amazon products to monetize a website.

Kiro - Core Features

What is Kiro Kiro is an innovative AI-powered IDE that revolutionizes software development through intelligent assistance and structured wor...