(Adopted from Business Times - my own personal archive)

Data is becoming a commodity. And just like all other commodities, data is beginning to lose some of its value as there is really too much raw data around. The situation was different 50 years ago. Investors interested in trading in a futures market such as rubber would actually send informants into the rubber estates to get information on rubber production. The reward for that research could be the investor becoming an overnight millionaire in the futures market.

Today, investors are inundated with higher quality raw data, including satellite pictures if you wish, over the Internet, for less than 10 dollars a month. More data can be obtained in 15 minutes, from a broader and richer set of sources than any investor 50 years ago could get with months of effort. The staggering quantities of data available today make it difficult to deal with. When data was scarce, the amount available was consumed by users with fairly primitive tools.

Investors now are faced with the problem of separating the significant facts from the rest. But getting the right data is still important. It is still the key to making money and being successful in business.

Good business practice can generally be described as follows:

• Acquiring quality raw data;

• Combining and integrating the data;

• Storing the data in a way that it delivers useful information; and

• Analysing the information and making good decisions.

To acquire quality raw data and to combine and integrate data are constant challenges in today's business environment. There is always that need to ensure that a company balances today's business needs vis-a-vis preparing itself for tomorrow's challenges. This balance, usually, can only be attained with a careful strategy and the right technology.

When acquiring quality raw data, there is a need to transform it -- which can be a a tedious and lengthy process that requires finding the right data, getting the syntax correct for each database type, and implementing the solution before the business needs change.

Problems in programming increase as complexity increases with the multiple number of data sources and data views. For example, Platinum Technology Asia Australasia can be input into the system as Platinum Technology, inc., for marketing; Platinum Tech for delivery and even Platinum Technology Pte Ltd for accounts.

Although all these names refer to the same company, the system will not recognise them as a unique company. You must have a solution to reconcile these differences.

But most tools for creating transformations are limited in scope and capability. These products are useful for pilot projects, but they fall short when the needs of organisations grow. This may result in having to resort to manually coded transformations that cannot be re-used or easily managed. Such incomplete solutions are unacceptable.

These products should allow organisations to quickly build data marts and warehouses without compromising ease-of-use, complex transformation capabilities or robust metadata management.

Organisations looking for a metadata-driven transformation solution that provides point-and-click source should ensure that the product performs:

- code generation,

- selection and conversion of new targets, and

- multiple source combination automatically.

The tool should also be able to handle sophisticated transformations to ensure data consistency across multiple data marts and warehouses.